Dewarping with a regular calibration point grid

Share your software workflow. Write up your tips and tricks on how to scan, digitize, OCR, and bind ebooks.

Moderator: peterZ

mhr
Posts: 37
Joined: 07 May 2012, 10:12
E-book readers owned: onyx-boox-m92 sony-trs-t1
Number of books owned: 500
Country: Germany

Dewarping with a regular calibration point grid

Post by mhr »

Hello all,

I refer to the post viewtopic.php?f=17&t=113&hilit=calibration, in which a calibration point grid is suggested to do a calibrated dewarping if the camera has a fixed position with respect to the scanned page. There are other approaches like viewtopic.php?f=13&t=784 to automatically recognize calibration information for dekeystoning. The latter seems appropriate for setups where the relative position is changing during scanning.

I have written a small command line program, which is capable to correct keystoning, page warping and camera lense distortion automatically by using a calibration point grid as suggested in the first referenced post from above. In the following posts I will discuss the approach taken and finally will supply the program, which is GPLed.

The general setup is the following:

Your scanning setup guarantees that the camera is in fixed geometrical position with respect to the platen. In contrast to all other setups, the platen need not be flat! A non-flat platen seems strange at first sight, but can be advantageous to avoid blur. I have estimated an absolute depth of focus of about 1 cm for a camera like mine (a Powershot A490), which seems to be roughly independent from the distance of the camera and the platen. Therefore, with the pythagorean theorem you can show, that noticeable blurring occurs quite fast for usual camera setups. If dR denotes the maximal depth of focus, R the vertical camera to platen distance and r the radius of a circle on a page with focused content, then the formula

Code: Select all

r = sqrt(dR * (2*R + dR))
gives the relationship between these values, providing that the camera is orthogonal to the platen. Some examples for dR = 1 cm:
  • R = 44 cm ==> r = 9.4 cm
  • R = 60 cm ==> r = 11 cm
  • R = 80 cm ==> r = 12.6 cm
  • R = 100 cm ==> r = 14.1 cm
If the platen is flat, 2*r is the maximal diameter of a page fully in focus, but if
the platen is part of a cylindrical surface, 2*r is the maximal height of a page fully in focus!
Therefore I plan to construct a scanner with a cylindrical platen to enhance sharpness.
The distortion will be no problem for me as the following posts should demonstrate.
mhr
Posts: 37
Joined: 07 May 2012, 10:12
E-book readers owned: onyx-boox-m92 sony-trs-t1
Number of books owned: 500
Country: Germany

Re: Dewarping with a regular calibration point grid

Post by mhr »

First of all, a suitable calibration grid has to be created. The program needs colored
points and defaults to red points to make point detection in images an easy task.

If you have no color printer, you can make one by hand using a gridded paper and marking a regular grid with a red marker pen. Note that the points should be filled circles with the corresponding corners located at the circle centers. This takes roughly half an hour if done with precision. My first version of a calibration grid was the following:
Hand made calibration grid with red points
Hand made calibration grid with red points
calibration_grid_hand_made.jpg (42.76 KiB) Viewed 20767 times
Note that the distance of the points must be regular. To calculate the DPI of the unwarped images more easily, a standard distance like 1 cm or 5 mm is a good choice. I used the latter distance finally. Finer grids are more precise in dewarping, but due to the underlying algorithm, distances like 5 mm are fine enough for the expected distortions.

If a color printer is available (I was able to print on a color laser printer), the following small postscript file

Code: Select all

% calibration_grid.ps
% Creates regular grid for camera calibration.
% This is for A4 pages, modify for other formats.
/dx  0.500 def
/dy  0.500 def
/xa  0.250 def
/xb 20.750 def
/ya  0.250 def
/yb 29.250 def
/r   0.125 def
/fac 72 2.54 div def
1 setlinewidth
0 0 0 setrgbcolor
xa dx xb { fac mul dup ya fac mul moveto yb fac mul lineto } for
ya dy yb { fac mul xa fac mul 1 index moveto
               xb fac mul exch lineto } for
stroke
1 0 0 setrgbcolor
ya dy yb { fac mul
           xa dx xb { fac mul 1 index r
                           fac mul 0 360 arc closepath fill } for
           pop } for
0 0 1 setrgbcolor
showpage
can be used and adjustments should be self-explanatory (the units are in cm).
The code above draws an A4-sized rectangular grid of red points (with radius 0.125 cm).
Last edited by mhr on 28 May 2012, 10:46, edited 1 time in total.
mhr
Posts: 37
Joined: 07 May 2012, 10:12
E-book readers owned: onyx-boox-m92 sony-trs-t1
Number of books owned: 500
Country: Germany

Re: Dewarping with a regular calibration point grid

Post by mhr »

To unwarp a series of pictures, first of all a picture has to be taken with the calibration grid at the defined paper position. The inplane rotation is not crucial, because programs like scantailor are very capable to correct these final inplane rotations perfectly. Then the book pages (or whatever stuff) can be scanned. For safety reasons (in case of unwanted changes in the relative positions of camera and book pages) I also take a calibration grid shot after scanning all book pages. If the book is large, more safety shoots may be advisable, depending on your confidence regarding the rigidity of your scanner setup.

The unwarp program ppmunwarp (see one of the following posts) needs all pictures in PPM format with raw picture data. Use a suitable converter like Imagemagick or NETPBM for conversion. I usually use the latter, because the conversions can be easily pipelined in one command like

Code: Select all

jpegtopnm page123.jpg | ppmunwarp -d deform.bin | pnmtopng > page123_unwarp.png
Note that grayscale PGM pictures need to be converted to real PPM images, though most other PNM utilities are capable of reading both formats while expecting color images. These conversions can be integrated into the pipeline with the filter programs pgmtoppm and ppmtopgm from the NETPBM package if necessary.

In the command from above the unwarping information of the calibration grid is contained in the file deform.bin. This has to be created first of course. If the picture of the calibration grid is contained in calib.jpg, then the command

Code: Select all

jpegtopnm calib.jpg | ppmunwarp -m check.ppm > deform.bin
creates this calibration data. The additional file check.ppm, created by ppmunwarp, is very useful to control the detected grid. If there are problems with point detection, a graphic editor like GIMP or PHOTOSHOP can be used to manually erase points (or even repair points in borderline cases) of the image file calib.jpg. Eventually you can help the program by changing the default values of the options -pc (point color), -ph (hue value range), -ps (saturation for point detection) and -pv (intensity threshold) to enhance point detection. If some few points are missing in the detected grid, this is acceptable, because the program tries to interpolate (and even extrapolate to a certain amount) missing points. Too irregularly shaped non-rectangular grids might give problems though. The program might even reject such calibration grid images and a correction of that file by a graphic editor is mandatory in such cases.

To easily detect the DPI of the unwarped pictures, it is advisable to unwarp the calibration grid itself with the command

Code: Select all

jpegtopnm calib.jpg | ppmunwarp -d deform.bin | pnmtopng > calib_unwarp.png
The grid points in calib_unwarp.png should be located in a perfect grid, parallel to the picture border. To calculate the DPI, look for the pixel count between the grid points in a graphic editor. If the physical grid has grid points at distance 5 mm and the number of pixels between e.g. 10 grid points (horizontally or vertically) is n, then the DPI can be calculated by

Code: Select all

DPI = (2.54 / 5) * n = 0.508 * n
Last edited by mhr on 28 May 2012, 10:51, edited 2 times in total.
mhr
Posts: 37
Joined: 07 May 2012, 10:12
E-book readers owned: onyx-boox-m92 sony-trs-t1
Number of books owned: 500
Country: Germany

Re: Dewarping with a regular calibration point grid

Post by mhr »

Some example images are given in this post for demonstration purposes. I scaled them down for file size. Note that this is just a rough test with limited precision. Precision is really increased by a perfect setup.

First the camera shot of the calibration grid:
Original warped calibration grid
Original warped calibration grid
calibration_grid_hand_made_warped.jpg (42.76 KiB) Viewed 20765 times
The generated control image (from ppmunwarp) with the detected grid points and their connectivity. This should be regularly connected though the grid is naturally deformed:
Check control image from ppmunwarp
Check control image from ppmunwarp
calibration_grid_check.jpg (28.26 KiB) Viewed 20765 times
The unwarped calibration grid (with ppmunwarp) with the perfectly rectangular array of grid points, orientated parallel to the image border:
Unwarped calibration grid
Unwarped calibration grid
calibration_grid_hand_made_unwarped.jpg (44.05 KiB) Viewed 20765 times
A test page (GPL license) shot by the camera:
Original warped test page (GPL)
Original warped test page (GPL)
gpl_page_warped.jpg (39.77 KiB) Viewed 20765 times
The unwarped test page (with ppmunwarp), which is adequately corrected regarding this simple and unprecise camera setup:
Unwarped test page (GPL)
Unwarped test page (GPL)
gpl_page_unwarped.jpg (50.04 KiB) Viewed 20765 times
mhr
Posts: 37
Joined: 07 May 2012, 10:12
E-book readers owned: onyx-boox-m92 sony-trs-t1
Number of books owned: 500
Country: Germany

Re: Dewarping with a regular calibration point grid

Post by mhr »

And here is the program called ppmunwarp, which does the actual calibration. I've chosen the GPL copyleft policy:
COPYING.txt
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
(34.32 KiB) Downloaded 713 times
The program source consists of one C++ file
ppmunwarp.cc.txt
The program source ppmunwarp.cc
(61.37 KiB) Downloaded 823 times
Please drop the .txt extension. That was necessary to allow this file as attachment for this post. The program depends on no additional libraries! It should compile well under Linux and Windows and very probably also under MAC OS (but I have zero experience with MAC OS). The header of the source file contains some information about command line compilation on Linux with g++. For other operating systems the procedure is similar (on Windows the natural extension is .cpp and not .cc).

The header also contains some information about available options. They are also listed by

Code: Select all

ppmunwarp -h
Note that the program is intensionally kept small. Every enhanced functionality like other image formats are omitted on purpose. It does what is does in a command line environment, which can flexible and powerful be augmented with suitable other functional unities. If you want a GUI, write one on your own and make a system call for ppmunwarp. The program is open source, therefore you are also invited to create your own version or integrate it into whatever program you like.

Just for the curious ones the algorithmic stages of the program while generating calibration data:
  • Detect grid point domains as connected pixel areas of suitable colored pixels.
  • Find centers of these connected point domains as grid point locations by weighted average of pixel intensities.
  • Search for initial grid cell (the mostly rectangular cell of neighboring grid points).
  • Extend the seed grid cell in all directions by extrapolating grid positions and searching for closely located not yet connected grid points.
  • Write out test file (if selected) with detected grid points, connected grid points and lines visualizing grid connectivity. The grid center pixels are colored in white, the background color is black.
  • Select missing grid points by the method of moving least squares (MLS). If the envolved matrices are numerically singular, program is aborting.
  • Write out calibration data. There are binary (the normal choice) and ascii exports available.
The actual dewarping is done for each of the three color channels by cubic interpolation between (per default oversampled by a factor of two) grid points. The resolution of the output image is chosen in a way, that the number of pixels (pixel area) of the unwarped and warped images inside the detected grid are approximately equal.

Have fun!
User avatar
daniel_reetz
Posts: 2812
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Dewarping with a regular calibration point grid

Post by daniel_reetz »

I can add .cc (or any other extension) to the list of allowed filetypes, if you like... send me a PM if so!
abmartin
Posts: 79
Joined: 15 Sep 2010, 15:33
Number of books owned: 2000
Country: USA
Location: Ohio

Re: Dewarping with a regular calibration point grid

Post by abmartin »

I only discovered this post about two weeks ago. I want to offer my most sincere thanks to the author. This is marvelous. I was as excited as a tween at a Justin Bieber concert when I found this inspired little program! For anyone who wants geometrically-correct images, this is the fastest, simplest tool I have seen. For fun, I have been taking photos off-axis and with badly-distorting lenses to try and trip this program up, and it works great every time. (Although obviously, having the camera aimed properly is still important to have the image in focus over an entire page)

The trickiest thing needed to use this program is to make a series of calibration images of different sizes to aid the automatic cropping stage. A quick trip to kinkos with a few sizes of calibration images and a paper cutter, I made about 5 standard sizes that should cover all sizes I might need. (Including one the complete size of the platen for those jumbo books) I have included eps files in this post to make printing easy for those of us using US page sizes. To adjust for your needed print size, change the bounding box size and adjust the xb and yb coordinates as appropriate.

To run this program on a directory of images, after creating the calibration information, you can use a simple bash script to run it on an entire directory.

Code: Select all

 
#!/bin/bash
for i in *.ppm; do
 if [ -e "$i" ]; then
   file=`basename "$i" .ppm`
   /path/to/ppmunwarp -d calibration.bin "$i" > "$file.png"
 fi
done
Then covert the resulting output (still actually a pixel map file at this point) to your output file format of choice with imagemagick. (since the calibration grid has known dimensions, you can use gimp to measure the pixelspercentimeter, multiply by 2.54 and apply that DPI to the converted images too at this stage)

Code: Select all

mogrify -format tif -density ??? -units PixelsPerInch -compress lzw *.png
I plan on posting a script that does all of this in the next few days in a new tutorial thread. Stay tuned.

Here are some eps files for three different US print sizes that you can convert to PDF to take to kinkos or print yourself on your color printer. (For some reason, my local Kinkos couldn't handle the EPS files themselves. Pretty ironic that they use Adobe software that can't handle a format designed by Adobe...) Just copy the text into a blank file, rename the extension to .eps, then convert it to pdf to print. (I used epstopdf, one of the ghostscript tools) If there is a call for it, I can post actual pdf files, but I'd rather not use up more of Dan's bandwidth than I have to.

Letter (8.5x11)

Code: Select all

%!PS-Adobe-3.0 EPSF-3.0
%%Origin: 0 0
%%BoundingBox: 0 0 612 792
%%LanguageLevel: 2 

/dx  0.500 def
/dy  0.500 def
/xa  0.250 def
/xb  21.590 def
/ya  0.250 def
/yb  27.940 def
/r   0.125 def
/fac 72 2.54 div def
1 setlinewidth
0 0 0 setrgbcolor
xa dx xb { fac mul dup ya fac mul moveto yb fac mul lineto } for
ya dy yb { fac mul xa fac mul 1 index moveto
               xb fac mul exch lineto } for
stroke
1 0 0 setrgbcolor
ya dy yb { fac mul
           xa dx xb { fac mul 1 index r
                           fac mul 0 360 arc closepath fill } for
           pop } for
0 0 1 setrgbcolor
Legal (8.5x14)

Code: Select all

%!PS-Adobe-3.0 EPSF-3.0
%%Origin: 0 0
%%BoundingBox: 0 0 612 1008
%%LanguageLevel: 2 

/dx  0.500 def
/dy  0.500 def
/xa  0.250 def
/xb 21.590 def
/ya  0.250 def
/yb 35.560 def
/r   0.125 def
/fac 72 2.54 div def
1 setlinewidth
0 0 0 setrgbcolor
xa dx xb { fac mul dup ya fac mul moveto yb fac mul lineto } for
ya dy yb { fac mul xa fac mul 1 index moveto
               xb fac mul exch lineto } for
stroke
1 0 0 setrgbcolor
ya dy yb { fac mul
           xa dx xb { fac mul 1 index r
                           fac mul 0 360 arc closepath fill } for
           pop } for
0 0 1 setrgbcolor
Tabloid (11x17)

Code: Select all

%!PS-Adobe-3.0 EPSF-3.0
%%Origin: 0 0
%%BoundingBox: 0 0 792 1224
%%LanguageLevel: 2 

/dx  0.500 def
/dy  0.500 def
/xa  0.250 def
/xb  27.940 def
/ya  0.250 def
/yb 43.180 def
/r   0.125 def
/fac 72 2.54 div def
1 setlinewidth
0 0 0 setrgbcolor
xa dx xb { fac mul dup ya fac mul moveto yb fac mul lineto } for
ya dy yb { fac mul xa fac mul 1 index moveto
               xb fac mul exch lineto } for
stroke
1 0 0 setrgbcolor
ya dy yb { fac mul
           xa dx xb { fac mul 1 index r
                           fac mul 0 360 arc closepath fill } for
           pop } for
0 0 1 setrgbcolor
mhr
Posts: 37
Joined: 07 May 2012, 10:12
E-book readers owned: onyx-boox-m92 sony-trs-t1
Number of books owned: 500
Country: Germany

Re: Dewarping with a regular calibration point grid

Post by mhr »

I just saw a few days ago the activities of Abmartin.
In the thread http://www.diybookscanner.org/forum/vie ... 5&start=20
an updated version of my program ppmunwarp is presented to obtain PPI information.
I updated my program to calculate this information in a hopefully more precise manner.
It now takes information of the unwarped grid to give PPI information in x-direction, y-direction and a geometric mean of them
as a printout on stderr.

Instead of the -mul 5.08 option, you can use the -gx 5.08 option. Another option -gy is also available if
the calibration grid points do have different physical distances. I don't know if this might be useful to someone later.
If only -gx is set, the two factors are simultaneously set equal to the given factor.
The number indicates, how many calibration points makes it to one inch.
The output has two more lines, one indcates the different PPI numbers in x- and y-direction.
The more important line looks like

Code: Select all

PPI: <number>
and unreveales the wanted PPI number as a geometric means of the calculated ones in the axial directions.

And here is the updated program:
ppmunwarp.cc.txt
The version 1.1 of ppmunwarp with PPI features
(63.15 KiB) Downloaded 569 times
An update of the scripts of abmartin should be easy as I kept the main ideas there.

Have fun!
mhr
Posts: 37
Joined: 07 May 2012, 10:12
E-book readers owned: onyx-boox-m92 sony-trs-t1
Number of books owned: 500
Country: Germany

Re: Dewarping with a regular calibration point grid

Post by mhr »

Just another update with a minor bugfix.
I noted that the option '-gm' doesn't accept a zero value, which is useful to avoid boundaries
around detected grid points.
ppmunwarp.cc.txt
Version 1.2 of ppmunwarp
(63.18 KiB) Downloaded 580 times
mhr
Posts: 37
Joined: 07 May 2012, 10:12
E-book readers owned: onyx-boox-m92 sony-trs-t1
Number of books owned: 500
Country: Germany

Re: Dewarping with a regular calibration point grid

Post by mhr »

I just decided to enable ppmunwarp to be able to convert 8 bit and 16 bit ppm files.
So this version 1.3 of ppmunwarp should recognize and convert all valid raw ppm files.
You can create 16 bit ppm files from raw camera dng files via a program like dcraw.
The target color depth can be specified with the new option -od:
ppmunwarp.cc.txt
Version 1.3 of ppmunwarp
(66.66 KiB) Downloaded 606 times
Post Reply