Daniel Reetz, the founder of the DIY Book Scanner community, has recently started making videos of prototyping and shop tips. If you are tinkering with a book scanner (or any other project) in your home shop, these tips will come in handy. https://www.youtube.com/channel/UCn0gq8 ... g_8K1nfInQ

Scantailor preprocess script

General discussion about software packages and releases, new software you've found, and threads by programmers and script writers.
lab_rat
Posts: 6
Joined: 04 Jan 2013, 23:55
E-book readers owned: kindle
Number of books owned: 0
Country: USA

Re: Scantailor preprocess script

Post by lab_rat » 18 Jun 2013, 23:06

I truly appreciate your help. I will try the new script and let you know if I am able to get it to work.

Thanks again!

adamzero
Posts: 6
Joined: 03 Jul 2013, 02:30
E-book readers owned: ipad
Number of books owned: 1000
Country: USA

Re: Scantailor preprocess script

Post by adamzero » 08 Jul 2013, 12:03

Hi everyone, I'm having trouble.

I'm running this on OS X, on jpgs, in automatic dpi mode, without color calibration. "calibration.bin" is zero bytes, and (thus?) so is every other "*_corrected.ppm" file. What's going wrong and how do I fix it?

Here's the log:

Code: Select all

Your camera images are now being prepared for Scantailor

No color correction will be done
Preparing images for ppmunwarp by converting jpg to ppm
done

Calculating geometry calibration data from image: calibration.ppm
Number of detected points: 4483
Only 2087 detected points used for calibration!
Average: 78.385221, calculated from 1622 of 2043 data points. PPI: 398
!!! Error in ppmunwarp:
!!! Distance too large for extrapolation!
done

Correcting geometry.  This will take some time...
!!! Error in ppmunwarp:
!!! Couldn't read deformation map file 'calibration.bin'!
!!! Error in ppmunwarp:
!!! Couldn't read deformation map file 'calibration.bin'!
!!! Error in ppmunwarp:
!!! Couldn't read deformation map file 'calibration.bin'!
!!! Error in ppmunwarp:
!!! Couldn't read deformation map file 'calibration.bin'!
!!! Error in ppmunwarp:
!!! Couldn't read deformation map file 'calibration.bin'!
!!! Error in ppmunwarp:
!!! Couldn't read deformation map file 'calibration.bin'!
!!! Error in ppmunwarp:
!!! Couldn't read deformation map file 'calibration.bin'!
!!! Error in ppmunwarp:
!!! Couldn't read deformation map file 'calibration.bin'!
!!! Error in ppmunwarp:
!!! Couldn't read deformation map file 'calibration.bin'!
!!! Error in ppmunwarp:
!!! Couldn't read deformation map file 'calibration.bin'!
done

Calculating DPI from image: calibration_corrected.ppm
!!! Error in ppmunwarp:
!!! PPM image file 'calibration_corrected.ppm' has wrong preamble!
Calculated PPI is: 

ImageMagick will now prepare images for Scantailor...
mogrify: invalid argument for option `-units': -density @ error/mogrify.c/MogrifyImageCommand/4340.
done

Deleting temporary files...
done

Your images are ready for Scantailor

abmartin
Posts: 79
Joined: 15 Sep 2010, 15:33
Number of books owned: 2000
Country: USA
Location: Ohio

Re: Scantailor preprocess script

Post by abmartin » 09 Jul 2013, 12:32

That's cool that it can run on Macs!

It looks like it is failing back at the ppmunwarp calibration stage.

Without a sample image, I can propose two issues that seem to be the most common. It's detecting A LOT of points, but only about half were considered good enough to use. Here are my guesses:

1. You photographed the calibration image with another calibration image on the facing page, meaning the camera is picking up two images rather than just one. That wreaks havoc with ppmunwarp.

2. ppmunwarp isn't detecting the correct color dots. Check out ppmunwarp's options and determine the correct options for your setup that will only get the correct colors. Determine the correct options by just using ppmunwarp outside of the script. Once you have it working, you can add those options into the configuration section of the script.

Basic command to calibrate:
ppmunwarp -m check.ppm calibration.ppm > calibration.bin

Using a photoeditor, determine the hex value of the dots' color. Using that, you can add in a specific color option -- then play with the range option, saturation, and intensity thresholds to get things working right for your setup. Once done, you shouldn't have to change it and can just enter that value into the script.

-pc (point color in hex)
-ph (range percentage)
-ps (saturation percentage)
-pv (intensity volume)

adamzero
Posts: 6
Joined: 03 Jul 2013, 02:30
E-book readers owned: ipad
Number of books owned: 1000
Country: USA

Re: Scantailor preprocess script

Post by adamzero » 09 Jul 2013, 16:50

Thanks for the response!

FYI, for any mac users who want to use the preprocess script, every instance of sed -r has to be replaced by sed -E.

Here is a link to my calibration image: http://s21.postimg.org/qlus584mv/calibration.jpg / it is also uploaded attached to this post.

The problem is not an image on the facing page; it is probably the colors being way off without any color correction. But I am trying to manually enter the values as you suggested and then ppmunwarp is only find a tiny number of dots and calculating completely wrong DPI. No matter what I'm trying for the values (hexes from various dots, different numbers for hsv), I'm actually getting worse results than I was getting before. (But then, aside from the hex value, I"m not really sure what the changes I'm making mean.)
Attachments
calibration.jpg

adamzero
Posts: 6
Joined: 03 Jul 2013, 02:30
E-book readers owned: ipad
Number of books owned: 1000
Country: USA

Re: Scantailor preprocess script

Post by adamzero » 11 Jul 2013, 01:13

More data for you: I thought maybe I should do geometry correction first, since the keystoning might be making it hard for ppmunwarp to find the dots. When I try to do geometry correction, note the following:

When I do

Code: Select all

./ppmunwarp -mul 5.08 -m check.ppm calibration.ppm > calibration.bin
Number of detected points: 4549
Only 2095 detected points used for calibration!
Average: 78.381044, calculated from 1616 of 2047 data points. PPI: 398
!!! Error in ppmunwarp:
!!! Distance too large for extrapolation!
It chokes and the output files are not generated.

If I use a dot color sampled from one in the middle of my calibration.jpg, then ppmunwarp runs but with terrible numbers.

Code: Select all

./ppmunwarp -pc B7362E -mul 5.08 -m check.ppm calibration.ppm > calibration.bin
Number of detected points: 108
Only 15 detected points used for calibration!
Average: nan, calculated from 0 of 10 data points. PPI: -2147483648
Deskewed picture size: 109 x 131   (2.37% x 3.80%)
Yet does generate the output files. Check.ppm, converted to jpg, is attached, and looks pretty good.
check.jpg
But when I then do the geometry correction, though, and look at calibration_corrected.ppm, it is all f-cked up. See again the attached.
calibration_corrected.jpg
calibration_corrected.jpg (4.69 KiB) Viewed 6379 times

abmartin
Posts: 79
Joined: 15 Sep 2010, 15:33
Number of books owned: 2000
Country: USA
Location: Ohio

Re: Scantailor preprocess script

Post by abmartin » 11 Jul 2013, 10:49

Thanks for the sample. I played with it a little, but I've got to take off and won't be able to get to it until late tonight. I managed to get pretty close. It's tricky because of the bottom right corner being so different in color to the rest. It's going to be finicky to get that to work, but could very well be possible.

(Just FYI, in the check file, the points that are being used have blue lines between them, so it's actually the page edge that was being used the generate the output, so that's why the results look so terrible)

However, you'd be better off getting closer with the colors when you photograph. It's pretty hard for the program to find red dots of a particular color when things are so far off. (The background, for example, which has almost 3 times the amount of red when compared to blue and double green to blue, makes it very difficult to get proper contrast between the red dots and the red-green paper.

Slowing down the shutter speed and adjusting the camera white balance seems to be in order. You should be able to get reasonably close to the real colors (say within 10 percent) with just the cameras. First, set your white balance to match your lighting. Then adjust the shutter speed and f-stop. (I like to open the aperture a lot) If you are setting the exposure to a book page, It's better if you overexpose it by a click or two, since light meters are expecting colors to be averaging a darker gray than what a book page averages.

If you look at the sample pages I did, with proper camera settings, the color is pretty close before I fine-adjust the whitebalance. I expect that you can get that close too.

Improving the colors also make a big difference with scantailor quality too. (And mixed-mode pages look significantly better)

adamzero
Posts: 6
Joined: 03 Jul 2013, 02:30
E-book readers owned: ipad
Number of books owned: 1000
Country: USA

Re: Scantailor preprocess script

Post by adamzero » 11 Jul 2013, 14:26

Thanks for your detailed reply!

I did kludge it to the point of working this morning. First I imagemagicked the colors to get the background near-white, then I manually recolored a couple dozen dots in that bottom right corner, passed ppmunwarp the pc flag with that color, and after that everything ran without a hitch!

In the future I'll play with the camera more. I just built the new standard scanner, but I know nothing about photography.

mhr
Posts: 37
Joined: 07 May 2012, 10:12
E-book readers owned: onyx-boox-m92 sony-trs-t1
Number of books owned: 500
Country: Germany

Re: Scantailor preprocess script

Post by mhr » 14 Aug 2013, 03:05

I updated my program ppmunwarp to version 1.1 with calculation of the correct PPI information for the unwarped pages.
You can find the program at http://www.diybookscanner.org/forum/vie ... 464#p16464.

Have fun!

mhr
Posts: 37
Joined: 07 May 2012, 10:12
E-book readers owned: onyx-boox-m92 sony-trs-t1
Number of books owned: 500
Country: Germany

Re: Scantailor preprocess script

Post by mhr » 16 Aug 2013, 09:19

There is another update of ppmunwarp to version 1.2 in
http://www.diybookscanner.org/forum/vie ... =19&t=2589, which eliminates a minor bug in option processing.
This last version is used in a tutorial, which I have written today about a new tool for white balancing color correction. The new
tool is in the same spirit as ppmunwarp and called ppmwhitebalancing.
You can find this tutorial in http://www.diybookscanner.org/forum/vie ... =19&t=2891.
This was inspired by the activities of abmartin and other users.

The usefulness of ppmunwarp seems to be approved by the reactions of the community in this forum.

The usefulness of ppmwhitebalancing, which I have written just because I got curious, is not quite clear to me though. I think its possibilities will become clearer after several tests, which some of the users here might carry out.

Have fun!

mhr
Posts: 37
Joined: 07 May 2012, 10:12
E-book readers owned: onyx-boox-m92 sony-trs-t1
Number of books owned: 500
Country: Germany

Re: Scantailor preprocess script

Post by mhr » 27 Aug 2013, 07:50

New versions 1.3 of ppmunwarp and ppmwhitebalance are available in the above mentioned tutorials.
They are capable of converting any raw ppm file now, including ones with 16 bit color depth. You can create such
files via e.g. dcraw from raw camera images.

Post Reply