Daniel Reetz, the founder of the DIY Book Scanner community, has recently started making videos of prototyping and shop tips. If you are tinkering with a book scanner (or any other project) in your home shop, these tips will come in handy. https://www.youtube.com/channel/UCn0gq8 ... g_8K1nfInQ

Scantailor preprocess script

General discussion about software packages and releases, new software you've found, and threads by programmers and script writers.
abmartin
Posts: 79
Joined: 15 Sep 2010, 15:33
Number of books owned: 2000
Country: USA
Location: Ohio

Re: Scantailor preprocess script

Post by abmartin » 11 Jun 2013, 20:22

There's no problem with noob questions that are asked clearly like you did!

Have you compiled ppmunwarp yet? The cpp file is just the source. Compile it on Linux using: g++ -o ppmunwarp ppmunwarp.cpp

That will produce a file called ppmunwarp (no extension) Put it wherever you want. There are three ways of doing it that I see:
1. Put it in a folder that the system looks too when calling a binary, like /usr/bin. Nothing needs added to the script for that.
2. A folder of your choice, such as a subdirectory called software in your home directory, add ~/software/ to the script whenever ppmunwarp is called. (It's in there four times, I think. Lines 109, 118, 129, 152.)
3. Place the ppmunwarp binary in the working directory. Nothing needs changed there either.

I think it's just easier to put it in a binary folder. (I use ~/bin/)

Maybe it would be worth adding a configuration option for a path to ppmunwarp. (As in option 2) I never really thought about it! Thanks for pointing that out to me.

lab_rat
Posts: 6
Joined: 04 Jan 2013, 23:55
E-book readers owned: kindle
Number of books owned: 0
Country: USA

Re: Scantailor preprocess script

Post by lab_rat » 11 Jun 2013, 23:19

note to self. install GNU C++ compiler... SMH. This is what happens when you have a Windows user dual booting for the first time.

Thank you for your help!

lab_rat
Posts: 6
Joined: 04 Jan 2013, 23:55
E-book readers owned: kindle
Number of books owned: 0
Country: USA

Re: Scantailor preprocess script

Post by lab_rat » 13 Jun 2013, 00:35

I have dpi set to automatic.

Code: Select all

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
Calculated PPI is: 

ImageMagick will now prepare images for Scantailor...
mogrify.im6: invalid argument for option `-units': -density @ error/mogrify.c/MogrifyImageCommand/4325.
done
when I set it to manual, and use gimp to input dpi

Code: Select all

calibration_corrected.ppm is being opened in GIMP
Manually determine the PixelsPerInch

PixelsPerInch?: 
PPI is set to: 127

ImageMagick will now prepare images for Scantailor...
mogrify.im6: improper image header `01_L_corrected.ppm' @ error/pnm.c/ReadPNMImage/293.
what am I doing wrong?

abmartin
Posts: 79
Joined: 15 Sep 2010, 15:33
Number of books owned: 2000
Country: USA
Location: Ohio

Re: Scantailor preprocess script

Post by abmartin » 15 Jun 2013, 20:48

Could I see the entire log file? I don't think I have enough information to answer properly. I'll speculate though, because that's what men do even when we don't have enough data. (I'm suddenly picturing the guys on King of the Hill all giving different reasons why a truck won't start) While you are at it, I guess I can check your config section too from the main script.

The first problem:
The first question I have is if it is unwarping things correctly up to the DPI stage.

Did you change anything in the script? I'm not seeing the output from ppmunwarp that I get with the commands I run.

My log gives:

Code: Select all

Calculating DPI from image: calibration_corrected.ppm
Number of detected points: 1938
Average: 64.579329, calculated from 1900 of 1900 data points. PPI: 328
Deskewed picture size: 3355 x 2516   (99.91% x 99.93%)
Calculated PPI is: 328
In order to have it give the DPI, you need to have the correct -mul option set in the UNWARP_OPT section of the config. (If using the half centimeter sheet, it should be UNWARP_OPT="-mul 5.08")

The imagemagick error there is because there is no argument being given to density option. (because the variable isn't being read)

The second error:
That looks like the script is working properly, but imagemagick is choking somewhere. It looks to me like a memory leak of some sort. In your package manager, you may be able to try installing a different version of imagemagick to see if it solves it. (I don't know enough about imagemagick or c to really understand that error message. It comes from here somewhere: http://trac.imagemagick.org/browser/Ima ... ders/pnm.c )


I'm also curious if things work properly if you just leave the DPI config option blank. In that instance, it will just try to convert the unwarped images into tif files without inserting DPI. That would isolate whether it is an imagemagick problem or a script problem. Also, the DPI value you entered seems pretty low. (Scantailor won't even work with files below 150 dpi) I have no idea why that could throw imagemagick off, though.

lab_rat
Posts: 6
Joined: 04 Jan 2013, 23:55
E-book readers owned: kindle
Number of books owned: 0
Country: USA

Re: Scantailor preprocess script

Post by lab_rat » 17 Jun 2013, 18:21

Currently running Ubuntu 13.04 in virtual machine. I am using the red dot (cm) calibration page.

log (ran with fewer images so I could post the entire log)

Code: Select all

Your camera images are now being prepared for Scantailor

Determining RGB values of color.CR2...
Detected RGB values are 43, 26, 12
RGB values to be adjusted by 3.7674418604, 6.2307692307, 13.3333333333

Running Imagemagick to adjust colors and convert CR2 to ppm...
done

Calculating geometry calibration data from image: calibration.ppm
Usage: ppmunwarp [options] [--] [inpname or stdin]

Options:
  --version     Print program version.
  -h            Print program usage.
  -q            Suppress normal program messages.
  -c            Enforce calibration mode.
  -m            Set output PPM picture with checked calibration points.
  -cc (inpname) Set input PPM picture with calibration points.
  -cp           Set file name for textual calibration point list.
  -d  (stdout)  Set file name for binary deformation grid.
  -i  (inpname) Set input file name.
  -o  (stdout)  Set ouput file name.
  -gm (1)       Set additional margin around calibration points.
  -gs (2)       Set scale factor for deformation grid.
  -pc (ff0000)  Point color for calibration point detection.
  -ph (85)      Range [%] around point color for detection.
  -ps (50)      Saturation [%] for point detection.
  -pv (50)      Intensity volume threshold [%] for point detection.

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
done

Correcting geometry.  This will take some time...
Usage: ppmunwarp [options] [--] [inpname or stdin]

Options:
  --version     Print program version.
  -h            Print program usage.
  -q            Suppress normal program messages.
  -c            Enforce calibration mode.
  -m            Set output PPM picture with checked calibration points.
  -cc (inpname) Set input PPM picture with calibration points.
  -cp           Set file name for textual calibration point list.
  -d  (stdout)  Set file name for binary deformation grid.
  -i  (inpname) Set input file name.
  -o  (stdout)  Set ouput file name.
  -gm (1)       Set additional margin around calibration points.
  -gs (2)       Set scale factor for deformation grid.
  -pc (ff0000)  Point color for calibration point detection.
  -ph (85)      Range [%] around point color for detection.
  -ps (50)      Saturation [%] for point detection.
  -pv (50)      Intensity volume threshold [%] for point detection.

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
Usage: ppmunwarp [options] [--] [inpname or stdin]

Options:
  --version     Print program version.
  -h            Print program usage.
  -q            Suppress normal program messages.
  -c            Enforce calibration mode.
  -m            Set output PPM picture with checked calibration points.
  -cc (inpname) Set input PPM picture with calibration points.
  -cp           Set file name for textual calibration point list.
  -d  (stdout)  Set file name for binary deformation grid.
  -i  (inpname) Set input file name.
  -o  (stdout)  Set ouput file name.
  -gm (1)       Set additional margin around calibration points.
  -gs (2)       Set scale factor for deformation grid.
  -pc (ff0000)  Point color for calibration point detection.
  -ph (85)      Range [%] around point color for detection.
  -ps (50)      Saturation [%] for point detection.
  -pv (50)      Intensity volume threshold [%] for point detection.

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
Usage: ppmunwarp [options] [--] [inpname or stdin]

Options:
  --version     Print program version.
  -h            Print program usage.
  -q            Suppress normal program messages.
  -c            Enforce calibration mode.
  -m            Set output PPM picture with checked calibration points.
  -cc (inpname) Set input PPM picture with calibration points.
  -cp           Set file name for textual calibration point list.
  -d  (stdout)  Set file name for binary deformation grid.
  -i  (inpname) Set input file name.
  -o  (stdout)  Set ouput file name.
  -gm (1)       Set additional margin around calibration points.
  -gs (2)       Set scale factor for deformation grid.
  -pc (ff0000)  Point color for calibration point detection.
  -ph (85)      Range [%] around point color for detection.
  -ps (50)      Saturation [%] for point detection.
  -pv (50)      Intensity volume threshold [%] for point detection.

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
Usage: ppmunwarp [options] [--] [inpname or stdin]

Options:
  --version     Print program version.
  -h            Print program usage.
  -q            Suppress normal program messages.
  -c            Enforce calibration mode.
  -m            Set output PPM picture with checked calibration points.
  -cc (inpname) Set input PPM picture with calibration points.
  -cp           Set file name for textual calibration point list.
  -d  (stdout)  Set file name for binary deformation grid.
  -i  (inpname) Set input file name.
  -o  (stdout)  Set ouput file name.
  -gm (1)       Set additional margin around calibration points.
  -gs (2)       Set scale factor for deformation grid.
  -pc (ff0000)  Point color for calibration point detection.
  -ph (85)      Range [%] around point color for detection.
  -ps (50)      Saturation [%] for point detection.
  -pv (50)      Intensity volume threshold [%] for point detection.

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
Usage: ppmunwarp [options] [--] [inpname or stdin]

Options:
  --version     Print program version.
  -h            Print program usage.
  -q            Suppress normal program messages.
  -c            Enforce calibration mode.
  -m            Set output PPM picture with checked calibration points.
  -cc (inpname) Set input PPM picture with calibration points.
  -cp           Set file name for textual calibration point list.
  -d  (stdout)  Set file name for binary deformation grid.
  -i  (inpname) Set input file name.
  -o  (stdout)  Set ouput file name.
  -gm (1)       Set additional margin around calibration points.
  -gs (2)       Set scale factor for deformation grid.
  -pc (ff0000)  Point color for calibration point detection.
  -ph (85)      Range [%] around point color for detection.
  -ps (50)      Saturation [%] for point detection.
  -pv (50)      Intensity volume threshold [%] for point detection.

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
Usage: ppmunwarp [options] [--] [inpname or stdin]

Options:
  --version     Print program version.
  -h            Print program usage.
  -q            Suppress normal program messages.
  -c            Enforce calibration mode.
  -m            Set output PPM picture with checked calibration points.
  -cc (inpname) Set input PPM picture with calibration points.
  -cp           Set file name for textual calibration point list.
  -d  (stdout)  Set file name for binary deformation grid.
  -i  (inpname) Set input file name.
  -o  (stdout)  Set ouput file name.
  -gm (1)       Set additional margin around calibration points.
  -gs (2)       Set scale factor for deformation grid.
  -pc (ff0000)  Point color for calibration point detection.
  -ph (85)      Range [%] around point color for detection.
  -ps (50)      Saturation [%] for point detection.
  -pv (50)      Intensity volume threshold [%] for point detection.

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
done

Calculating DPI from image: calibration_corrected.ppm
Usage: ppmunwarp [options] [--] [inpname or stdin]

Options:
  --version     Print program version.
  -h            Print program usage.
  -q            Suppress normal program messages.
  -c            Enforce calibration mode.
  -m            Set output PPM picture with checked calibration points.
  -cc (inpname) Set input PPM picture with calibration points.
  -cp           Set file name for textual calibration point list.
  -d  (stdout)  Set file name for binary deformation grid.
  -i  (inpname) Set input file name.
  -o  (stdout)  Set ouput file name.
  -gm (1)       Set additional margin around calibration points.
  -gs (2)       Set scale factor for deformation grid.
  -pc (ff0000)  Point color for calibration point detection.
  -ph (85)      Range [%] around point color for detection.
  -ps (50)      Saturation [%] for point detection.
  -pv (50)      Intensity volume threshold [%] for point detection.

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
Calculated PPI is: 

ImageMagick will now prepare images for Scantailor...
mogrify.im6: invalid argument for option `-units': -density @ error/mogrify.c/MogrifyImageCommand/4325.
done

Deleting temporary files...
done

Your images are ready for Scantailor
preprocess: I did change file extension to CR2

Code: Select all

#!/bin/bash

(

###Scantailor Preprocessor###
#Fixes color with a gray card and geometry with ppmunwarp and a calibration grid
#Dependencies: Imagemagick, ppmunwarp as modified by royeven, Ufraw (only needed for RAW), Ufraw-batch (only needed for RAW), GIMP (if manual DPI calculation is desired)
#Written by abmartin and royeven at www.diybookscanner.org

###Configuration###

# Global "variables"
INPUT_FORMAT="CR2" #Change as appropriate
OUTPUT_FORMAT="ppm" #Do not change
COLOR_IMAGE="color" # The file name (without extension) of the gray card photo
CALIBRATION_IMAGE="calibration" # The file name (without extension) of the geometry calibration file
CONTROL_IMAGE="check" # The name of the deskew control file created with the -m attribute of ppmunwarp
CONTROL_PPI="check_ppi" # The name of the ppi control file created with the -m attribute of ppmunwarp
CALIBRATION_FILE="calibration.bin" # The full file name of the binary calibration data
UNWARP_OPT="-mul 5.08" # Options to ppmunwarp. Change according to your setup. Note: the mul attribute is a multiplier to the pixel-count. Since there are 5.08 calibration points pr. inch (i.e. 2*2.54), we must multiply the pixel count between two adjacent calibration dots with 5.08 to get the PPI
CORRECTED_POSTFIX="_corrected" # the postfix of corrected images. E.g. the file "PAGE01_R.PPM" becomes "PAGE_01_POSTFIX.PPM"
COLOR_CORRECTION_METHOD="imagemagick" #options for imagemagick and ufraw - leave blank if no color correction is desired
DPI_CALCULATION_METHOD="automatic" #options are automatic, manual, and autocheck
IM_VERBOSE_OPTION="" #use "-verbose" if imagemagick details are desired, otherwise leave blank
IM_FINAL_OUTPUT_OPTION="-format tif -compress lzw" #set desired output format and compression options -- for tif with lzw compression, use "-format tif -compress lzw"

#Imagemagick Color Correction Options
#True RGB values of Gray Card
TRUE_RED=162
TRUE_GREEN=162
TRUE_BLUE=160
COLOR_CROP_AREA="500x500" #crop area for the gray card in the center in order to find the average color of the gray card. If the crop area extends beyond the gray card, color correction will be incorrect.

# Include current folder in search path for program ppmunwarp
PATH=$PATH:`pwd`

echo -e "\nYour camera images are now being prepared for Scantailor"

###Color Correction###

##Using Imagemagick to Fix Colors

if [ "$COLOR_CORRECTION_METHOD" == "imagemagick" ];
	then
	
	echo -e "\nDetermining RGB values of $COLOR_IMAGE.$INPUT_FORMAT..."
	#Convert RAW file extension and crop an area in the center of calibration image
	convert $COLOR_IMAGE.$INPUT_FORMAT -gravity center -crop "$COLOR_CROP_AREA"+0+0 color-test.png

	#Determine average colors in cropped area
	SOURCE_RED=$(convert color-test.png -resize 1x1 -format "%[fx:int(255*p{10,10}.r)]" info:)
	SOURCE_GREEN=$(convert color-test.png -resize 1x1 -format "%[fx:int(255*p{10,10}.g)]" info:)
	SOURCE_BLUE=$(convert color-test.png -resize 1x1 -format "%[fx:int(255*p{10,10}.b)]" info:)
	echo "Detected RGB values are $SOURCE_RED, $SOURCE_GREEN, $SOURCE_BLUE"
	
	#Calculate necessary adjustments
	RED_ADJUST="$(echo "scale=10; $TRUE_RED/$SOURCE_RED" | bc)"
	GREEN_ADJUST="$(echo "scale=10; $TRUE_GREEN/$SOURCE_GREEN" | bc)"
	BLUE_ADJUST="$(echo "scale=10; $TRUE_BLUE/$SOURCE_BLUE" | bc)"
	echo "RGB values to be adjusted by $RED_ADJUST, $GREEN_ADJUST, $BLUE_ADJUST"

	#Adjust Colors and Output Raw images as ppm files
	echo -e "\nRunning Imagemagick to adjust colors and convert $INPUT_FORMAT to $OUTPUT_FORMAT..."
	mogrify -format ppm $IM_VERBOSE_OPTION  -color-matrix "$RED_ADJUST 0 0 0 $GREEN_ADJUST 0 0 0 $BLUE_ADJUST" *.$INPUT_FORMAT
	
	#Remove temporary file
	rm color-test.png
	echo "done"
	
##Using UFRaw to Fix Colors

elif [ "$COLOR_CORRECTION_METHOD" == "ufraw" ];

	#1. select an area on your gray card,
	#2. click the eyedropper, which equalizes RGB values
	#3. adjust the exposure control to get RGB values to their ultimate goal.
	#4. Input values in the script

	then
	#Interactive variables
	echo -e "\nThe color calibration image is being loaded in UFRaw. Enter the following values for color correction"
	ufraw $COLORIMAGE.$INPUT_FORMAT &
	echo "Color temperature?: "
	read UFRAW_TEMP
	echo "Green Value?: "
	read UFRAW_GREEN
	echo "Exposure change?: "
	read UFRAW_EXPOSURE
	
	##Convert all files of INPUT_FORMAT to OUTPUT_FORMAT
	echo -e "\nRunning ufraw-batch, this will take a while..."
	ufraw-batch --out-type=$OUTPUT_FORMAT *.$INPUT_FORMAT #--temperature=$UFRAW_TEMP --green=$UFRAW_GREEN --exposure=$UFRAW_EXPOSURE 
	echo "done"

##No Color Correction

else
	echo -e "\nNo color correction will be done"
	echo "Preparing images for ppmunwarp by converting $INPUT_FORMAT to $OUTPUT_FORMAT"
	mogrify -format ppm $IM_VERBOSE_OPTION *.$INPUT_FORMAT
	echo "done"	
fi

###Geometry Correction###

##Calibration

echo -e "\nCalculating geometry calibration data from image: $CALIBRATION_IMAGE.$OUTPUT_FORMAT"
ppmunwarp $UNWARP_OPT -m "$CONTROL_IMAGE.$OUTPUT_FORMAT" "$CALIBRATION_IMAGE.$OUTPUT_FORMAT" > "$CALIBRATION_FILE"
echo -e "done"

##Correction

echo -e "\nCorrecting geometry.  This will take some time..."
for i in *.ppm; do
 if [ -e "$i" ]; then
   FILE=`basename "$i" .ppm`
   ppmunwarp $UNWARP_OPT -d calibration.bin "$i" > "$FILE$CORRECTED_POSTFIX.$OUTPUT_FORMAT"
 fi
done
echo -e "done"

###DPI Calculation###

if [ "$DPI_CALCULATION_METHOD" == "automatic" ];
	then
	##Calculating PPI in software
	echo -e "\nCalculating DPI from image: $CALIBRATION_IMAGE$CORRECTED_POSTFIX.$OUTPUT_FORMAT"
	PPI=$(( (ppmunwarp $UNWARP_OPT -m "$CONTROL_PPI.$OUTPUT_FORMAT" "$CALIBRATION_IMAGE$CORRECTED_POSTFIX.$OUTPUT_FORMAT") 1>/dev/null ) 2>&1)
	echo "$PPI"
	PPI=`echo "$PPI" | grep -o "Average: [0-9\.]*, calculated from [0-9]* of [0-9]* data points. PPI: [0-9]*" | sed -r 's/.*PPI: ([0-9]*).*/\1/g'`
	echo "Calculated PPI is: $PPI"

elif [ "$DPI_CALCULATION_METHOD" == "manual" ];
	then
	##Open Image in GIMP
	echo -e "\n$CALIBRATION_IMAGE$CORRECTED_POSTFIX.$OUTPUT_FORMAT is being opened in GIMP"
	echo "Manually determine the PixelsPerInch" 
	gimp "$CALIBRATION_IMAGE$CORRECTED_POSTFIX.$OUTPUT_FORMAT" &
	
	##Enter PPI
	echo -e "\nPixelsPerInch?: "
	read PPI
	echo "PPI is set to: $PPI"

elif [ "$DPI_CALCULATION_METHOD" == "autocheck" ];
	##Autodetecting DPI and comparing with a manual measurment in GIMP
	then
	
	##Calculating PPI in software
	echo -e "\nCalculating calibration data from image: $CALIBRATION_IMAGE$CORRECTED_POSTFIX.$OUTPUT_FORMAT"
	PPI=$(( (ppmunwarp $UNWARP_OPT -m "$CONTROL_PPI.$OUTPUT_FORMAT" "$CALIBRATION_IMAGE$CORRECTED_POSTFIX.$OUTPUT_FORMAT") 1>/dev/null ) 2>&1)
	echo "$PPI"

	##Checking with GIMP
	gimp "$CALIBRATION_IMAGE$CORRECTED_POSTFIX.$OUTPUT_FORMAT" &
	
	
	##Compare automatic and manual
	PPI=`echo "$PPI" | grep -o "Average: [0-9\.]*, calculated from [0-9]* of [0-9]* data points. PPI: [0-9]*" | sed -r 's/.*PPI: ([0-9]*).*/\1/g'`
	echo "Calculated PPI is: $PPI"
	echo "Is this correct? If not, insert corrected value now. If correct, leave empty and press enter"
	read PPI_CORRECTED
	if [ "$PPI_CORRECTED" != "" ]
	then
		PPI=$PPI_CORRECTED
	fi
	echo "PPI is set to: $PPI"
	

else
	#If config option is left blank or is incorrect, no DPI will be entered
	echo -e "\nDPI information has neither been calculated nor manually entered."
	echo "ImageMagick will now prepare images for Scantailor..."
	mogrify $IM_VERBOSE_OPTION $IM_FINAL_OUTPUT_OPTION *[$CORRECTED_POSTFIX].ppm
	rm *.ppm
	echo "done"
	exit
fi

	
###Preparing Images for Scantailor###
echo -e "\nImageMagick will now prepare images for Scantailor..."
mogrify $IM_VERBOSE_OPTION $IM_FINAL_OUTPUT_OPTION -density $PPI -units PixelsPerInch *[$CORRECTED_POSTFIX].ppm
echo "done"


###Housekeeping###

#Delete temporary files
echo -e "\nDeleting temporary files..."
rm *.ppm
echo "done"

echo -e "\nYour images are ready for Scantailor"
exit
) 2>&1 | tee preprocess.log

abmartin
Posts: 79
Joined: 15 Sep 2010, 15:33
Number of books owned: 2000
Country: USA
Location: Ohio

Re: Scantailor preprocess script

Post by abmartin » 17 Jun 2013, 22:30

Hmmm...

So it appears that the conversion to ppm is going okay. If my next suggestion is wrong, could you please confirm that the initial conversion is happening properly. (imagemagick should spit out an error message if it wasn't) Those files would appear in the working folder.

The first failure occurs when ppmunwarp tries to calibrate itself. The message it is giving us tells us that something is wrong with the syntax. I wonder if it is not able to find the calibration image. What have you named the calibration file? The script is trying to feed ppmunwarp with calibration.ppm which was converted from the original calibration.CR2. If you chose to call that file something different, you can change that name in the script config.

I sure hope it's that simple, or else I have some thinking to do.

I really should go back in and add exits on errors. (but I doubt that I will)

lab_rat
Posts: 6
Joined: 04 Jan 2013, 23:55
E-book readers owned: kindle
Number of books owned: 0
Country: USA

Re: Scantailor preprocess script

Post by lab_rat » 18 Jun 2013, 12:57

It appears to be able to find the calibration image without issue.
http://img14.imageshack.us/img14/5696/ea3i.jpg


I added verbose to the script... here is the log

Code: Select all

Your camera images are now being prepared for Scantailor

Determining RGB values of color.CR2...
Detected RGB values are 43, 26, 12
RGB values to be adjusted by 3.7674418604, 6.2307692307, 13.3333333333

Running Imagemagick to adjust colors and convert CR2 to ppm...
"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=/tmp/magick-gexOomHz.png" "/tmp/magick-SNC9KnXw"
/tmp/magick-gexOomHz.png PNG 3921x2611 3921x2611+0+0 16-bit DirectClass 47.19MB 1.140u 0:01.179
calibration.CR2=>/tmp/magick-gexOomHz.png CR2 3921x2611 3921x2611+0+0 16-bit DirectClass 47.19MB 0.010u 0:00.030
calibration.CR2=>calibration.ppm CR2 3921x2611 3921x2611+0+0 16-bit DirectClass 61.42MB 0.480u 0:00.469
"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=/tmp/magick-2d9YDotg.png" "/tmp/magick-uyvHArZZ"
/tmp/magick-2d9YDotg.png PNG 4770x3177 4770x3177+0+0 16-bit DirectClass 71.08MB 1.160u 0:01.160
color.CR2=>/tmp/magick-2d9YDotg.png CR2 4770x3177 4770x3177+0+0 16-bit DirectClass 71.08MB 0.010u 0:00.009
color.CR2=>color.ppm CR2 4770x3177 4770x3177+0+0 16-bit DirectClass 90.92MB 0.720u 0:00.510
"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=/tmp/magick-q45yXviB.png" "/tmp/magick-TxGIdCBc"
/tmp/magick-q45yXviB.png PNG 4770x3177 4770x3177+0+0 16-bit DirectClass 72.53MB 1.340u 0:01.339
IMG_0003.CR2=>/tmp/magick-q45yXviB.png CR2 4770x3177 4770x3177+0+0 16-bit DirectClass 72.53MB 0.010u 0:00.019
IMG_0003.CR2=>IMG_0003.ppm CR2 4770x3177 4770x3177+0+0 16-bit DirectClass 90.92MB 0.730u 0:01.140
"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=/tmp/magick-Vum1VR6o.png" "/tmp/magick-KuFJB9BB"
/tmp/magick-Vum1VR6o.png PNG 4770x3177 4770x3177+0+0 16-bit DirectClass 72.72MB 1.410u 0:01.419
IMG_0005.CR2=>/tmp/magick-Vum1VR6o.png CR2 4770x3177 4770x3177+0+0 16-bit DirectClass 72.72MB 0.020u 0:00.010
IMG_0005.CR2=>IMG_0005.ppm CR2 4770x3177 4770x3177+0+0 16-bit DirectClass 90.92MB 0.720u 0:00.480
"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=/tmp/magick-Biz0aDku.png" "/tmp/magick-zasyt82m"
/tmp/magick-Biz0aDku.png PNG 4770x3177 4770x3177+0+0 16-bit DirectClass 72.68MB 1.390u 0:01.390
IMG_0007.CR2=>/tmp/magick-Biz0aDku.png CR2 4770x3177 4770x3177+0+0 16-bit DirectClass 72.68MB 0.010u 0:00.010
IMG_0007.CR2=>IMG_0007.ppm CR2 4770x3177 4770x3177+0+0 16-bit DirectClass 90.92MB 0.700u 0:00.509
"ufraw-batch" --silent --create-id=also --out-type=png --out-depth=16 "--output=/tmp/magick-FaXfyvP4.png" "/tmp/magick-tEntiUBM"
/tmp/magick-FaXfyvP4.png PNG 4770x3177 4770x3177+0+0 16-bit DirectClass 72.89MB 1.390u 0:01.399
IMG_0009.CR2=>/tmp/magick-FaXfyvP4.png CR2 4770x3177 4770x3177+0+0 16-bit DirectClass 72.89MB 0.010u 0:00.019
IMG_0009.CR2=>IMG_0009.ppm CR2 4770x3177 4770x3177+0+0 16-bit DirectClass 90.92MB 0.730u 0:00.980
 Found a generic profile, type exif
 Found a generic profile, type exif
 Found a generic profile, type exif
 Found a generic profile, type exif
 Found a generic profile, type exif
 Found a generic profile, type exif
done

Calculating geometry calibration data from image: calibration.ppm
Usage: ppmunwarp [options] [--] [inpname or stdin]

Options:
  --version     Print program version.
  -h            Print program usage.
  -q            Suppress normal program messages.
  -c            Enforce calibration mode.
  -m            Set output PPM picture with checked calibration points.
  -cc (inpname) Set input PPM picture with calibration points.
  -cp           Set file name for textual calibration point list.
  -d  (stdout)  Set file name for binary deformation grid.
  -i  (inpname) Set input file name.
  -o  (stdout)  Set ouput file name.
  -gm (1)       Set additional margin around calibration points.
  -gs (2)       Set scale factor for deformation grid.
  -pc (ff0000)  Point color for calibration point detection.
  -ph (85)      Range [%] around point color for detection.
  -ps (50)      Saturation [%] for point detection.
  -pv (50)      Intensity volume threshold [%] for point detection.

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
done

Correcting geometry.  This will take some time...
Usage: ppmunwarp [options] [--] [inpname or stdin]

Options:
  --version     Print program version.
  -h            Print program usage.
  -q            Suppress normal program messages.
  -c            Enforce calibration mode.
  -m            Set output PPM picture with checked calibration points.
  -cc (inpname) Set input PPM picture with calibration points.
  -cp           Set file name for textual calibration point list.
  -d  (stdout)  Set file name for binary deformation grid.
  -i  (inpname) Set input file name.
  -o  (stdout)  Set ouput file name.
  -gm (1)       Set additional margin around calibration points.
  -gs (2)       Set scale factor for deformation grid.
  -pc (ff0000)  Point color for calibration point detection.
  -ph (85)      Range [%] around point color for detection.
  -ps (50)      Saturation [%] for point detection.
  -pv (50)      Intensity volume threshold [%] for point detection.

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
Usage: ppmunwarp [options] [--] [inpname or stdin]

Options:
  --version     Print program version.
  -h            Print program usage.
  -q            Suppress normal program messages.
  -c            Enforce calibration mode.
  -m            Set output PPM picture with checked calibration points.
  -cc (inpname) Set input PPM picture with calibration points.
  -cp           Set file name for textual calibration point list.
  -d  (stdout)  Set file name for binary deformation grid.
  -i  (inpname) Set input file name.
  -o  (stdout)  Set ouput file name.
  -gm (1)       Set additional margin around calibration points.
  -gs (2)       Set scale factor for deformation grid.
  -pc (ff0000)  Point color for calibration point detection.
  -ph (85)      Range [%] around point color for detection.
  -ps (50)      Saturation [%] for point detection.
  -pv (50)      Intensity volume threshold [%] for point detection.

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
Usage: ppmunwarp [options] [--] [inpname or stdin]

Options:
  --version     Print program version.
  -h            Print program usage.
  -q            Suppress normal program messages.
  -c            Enforce calibration mode.
  -m            Set output PPM picture with checked calibration points.
  -cc (inpname) Set input PPM picture with calibration points.
  -cp           Set file name for textual calibration point list.
  -d  (stdout)  Set file name for binary deformation grid.
  -i  (inpname) Set input file name.
  -o  (stdout)  Set ouput file name.
  -gm (1)       Set additional margin around calibration points.
  -gs (2)       Set scale factor for deformation grid.
  -pc (ff0000)  Point color for calibration point detection.
  -ph (85)      Range [%] around point color for detection.
  -ps (50)      Saturation [%] for point detection.
  -pv (50)      Intensity volume threshold [%] for point detection.

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
Usage: ppmunwarp [options] [--] [inpname or stdin]

Options:
  --version     Print program version.
  -h            Print program usage.
  -q            Suppress normal program messages.
  -c            Enforce calibration mode.
  -m            Set output PPM picture with checked calibration points.
  -cc (inpname) Set input PPM picture with calibration points.
  -cp           Set file name for textual calibration point list.
  -d  (stdout)  Set file name for binary deformation grid.
  -i  (inpname) Set input file name.
  -o  (stdout)  Set ouput file name.
  -gm (1)       Set additional margin around calibration points.
  -gs (2)       Set scale factor for deformation grid.
  -pc (ff0000)  Point color for calibration point detection.
  -ph (85)      Range [%] around point color for detection.
  -ps (50)      Saturation [%] for point detection.
  -pv (50)      Intensity volume threshold [%] for point detection.

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
Usage: ppmunwarp [options] [--] [inpname or stdin]

Options:
  --version     Print program version.
  -h            Print program usage.
  -q            Suppress normal program messages.
  -c            Enforce calibration mode.
  -m            Set output PPM picture with checked calibration points.
  -cc (inpname) Set input PPM picture with calibration points.
  -cp           Set file name for textual calibration point list.
  -d  (stdout)  Set file name for binary deformation grid.
  -i  (inpname) Set input file name.
  -o  (stdout)  Set ouput file name.
  -gm (1)       Set additional margin around calibration points.
  -gs (2)       Set scale factor for deformation grid.
  -pc (ff0000)  Point color for calibration point detection.
  -ph (85)      Range [%] around point color for detection.
  -ps (50)      Saturation [%] for point detection.
  -pv (50)      Intensity volume threshold [%] for point detection.

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
Usage: ppmunwarp [options] [--] [inpname or stdin]

Options:
  --version     Print program version.
  -h            Print program usage.
  -q            Suppress normal program messages.
  -c            Enforce calibration mode.
  -m            Set output PPM picture with checked calibration points.
  -cc (inpname) Set input PPM picture with calibration points.
  -cp           Set file name for textual calibration point list.
  -d  (stdout)  Set file name for binary deformation grid.
  -i  (inpname) Set input file name.
  -o  (stdout)  Set ouput file name.
  -gm (1)       Set additional margin around calibration points.
  -gs (2)       Set scale factor for deformation grid.
  -pc (ff0000)  Point color for calibration point detection.
  -ph (85)      Range [%] around point color for detection.
  -ps (50)      Saturation [%] for point detection.
  -pv (50)      Intensity volume threshold [%] for point detection.

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
done

Calculating DPI from image: calibration_corrected.ppm
Usage: ppmunwarp [options] [--] [inpname or stdin]

Options:
  --version     Print program version.
  -h            Print program usage.
  -q            Suppress normal program messages.
  -c            Enforce calibration mode.
  -m            Set output PPM picture with checked calibration points.
  -cc (inpname) Set input PPM picture with calibration points.
  -cp           Set file name for textual calibration point list.
  -d  (stdout)  Set file name for binary deformation grid.
  -i  (inpname) Set input file name.
  -o  (stdout)  Set ouput file name.
  -gm (1)       Set additional margin around calibration points.
  -gs (2)       Set scale factor for deformation grid.
  -pc (ff0000)  Point color for calibration point detection.
  -ph (85)      Range [%] around point color for detection.
  -ps (50)      Saturation [%] for point detection.
  -pv (50)      Intensity volume threshold [%] for point detection.

Simple calibration:  ppmunwarp [-m check.ppm] calib.ppm > deform.bin
Simple unwarping:    ppmunwarp -d deform.bin skewed.ppm > unwarped.ppm
Calculated PPI is: 

ImageMagick will now prepare images for Scantailor...
mogrify.im6: invalid argument for option `-units': -density @ error/mogrify.c/MogrifyImageCommand/4325.
done

Deleting temporary files...
done

Your images are ready for Scantailor

abmartin
Posts: 79
Joined: 15 Sep 2010, 15:33
Number of books owned: 2000
Country: USA
Location: Ohio

Re: Scantailor preprocess script

Post by abmartin » 18 Jun 2013, 13:48

My instinct is suggesting that the ppm files that imagemagick is making are somehow not compatible with ppmunwarp. Since your file manager doesn't seem to be thumbnailing them, I figure that might be the problem. (Of course, I'm not sure if your file manager thumbs ppm files anyway) Since we also seem to be getting 0 byte _corrected.ppm files, I think that backs this up even more. If you can send me the calibration.CR2 image, I'll play around with imagemagick to see if there is something special that needs done. I've sent you a PM with my e-mail.

*edit
I just RTFM....http://www.imagemagick.org/www/formats.html
It does look like imagemagick needs an additional piece of information when handling CR2 files.
"Requires an explicit image format otherwise the image is interpreted as a TIFF image (e.g. cr2:image.cr2)."

(It may be as simple as just manually editing line 64 -- at the end just make it cr2:*.$INPUT_FORMAT --- That may also need done to line 48: cr2:$COLOR_IMAGE.$INPUT_FORMAT)

abmartin
Posts: 79
Joined: 15 Sep 2010, 15:33
Number of books owned: 2000
Country: USA
Location: Ohio

Re: Scantailor preprocess script

Post by abmartin » 18 Jun 2013, 19:15

To report about the bug: The problem was happening in the initial conversion from the raw format. Then every other step failed too. (I really should add exit codes and errors, I just don't want too...) The solution, however, was pretty simple.

I just spent a few minutes playing with imagemagick and CR2 images. I've concluded that using imagemagick to do the initial raw conversion just isn't a great idea. The output that I get as ppm files have issues with ppmunwarp. (incorrect header)

The solution, and this seems to increase speed anyway, is to use ufraw-batch to do the initial raw conversion.

In this version, I have added two new color correction/conversion options. One for raw images without color correction (raw_no_color) and the other for raw images with automatic color correction (raw_color). The manual raw color correction option using ufraw and ufraw-batch is still available. (ufraw) The two remaining options are for non-raw images, like camera jpgs, one for automatic color correction (imagemagick) and the other for no color correction (leave config option blank)

In the new raw_color option, ufraw-batch makes tif files from the raw input. Imagemagick then determines appropriate color correction and applies that correction while converting into ppm files. (Then the intermediate tif files are deleted)

In raw_no_color, ufraw-batch just does the final conversion to ppm.

In all of the UFraw steps, I have added a --silent option. If you desire verbose output, delete it in the configuration.

I hope that this fixes everyone's RAW issues!
preprocessraw.txt
Updated preprocess script for better RAW functionality
(9.62 KiB) Downloaded 236 times
It still needs testing though!

(And one of these days, I'll get around to more sophisticated color profiling. See Misty's excellent thread about how that can be accomplished)

abmartin
Posts: 79
Joined: 15 Sep 2010, 15:33
Number of books owned: 2000
Country: USA
Location: Ohio

Re: Scantailor preprocess script

Post by abmartin » 18 Jun 2013, 19:49

It seems like UFRaw does something strange with tif files...I get error messages from imagemagick when it opens each tif. It still gets the job done though, so it doesn't really matter. If those error messages bother you when using the raw_color method, it's pretty simple to hide those messages. Just add a -quiet option to lines 106 and 122

It's definitely the fastest lossless format to use as the intermediate step, so I'm content with meaningless error messages. Pngs, for example, take at least 15 times longer, which in a long book would be pretty annoying.

I've tested the new version on a few different sets of raw files of my own, and it seems to work properly.

Post Reply