Laser Scan Methods

DIY Book Scanner Skunk Works. Share your crazy ideas and novel approaches. Home of the "3D structure of a book" thread.

Moderator: peterZ

duerig
Posts: 388
Joined: 01 Jun 2014, 17:04
Number of books owned: 1000
Country: United States of America

Laser Scan Methods

Post by duerig »

This thread is to collect resources and discussion about how to use line focused lasers to improve book scanning.

Overall idea:

Line-focused lasers projected at an angle towards an open book will describe a curve based on the shape of the pages of that book. An 'ideal' book with perfectly straight pages will have a 'V' shape with the bottom in the spine. A flat surface will have a single straight line. The height of the book at that point will determine the y-displacement of the line compared to a flat surface.

Because of this, we could take a picture of each page of the book twice, once with laser lines and once without. Then by analyzing the laser lines, we can dewarp the page to remove any page curl or keystoning. Variations of this might be to take a laser picture every Nth page and using the most one picture to dewarp multiple pages.

Benefits:

Using this method, it is possible to simplify the construction of scanners and streamline the scanning process. A platen would no longer be needed to flatten the pages, removing a source of glare and artifacts. Cradle placement would no longer be as important and there would be less need to worry about exact camera placement and angle because minor keystoning or other artifacts would be automatically corrected.

Resources:

Forum thread where this is discussed along with other ideas: http://www.diybookscanner.org/forum/vie ... f=17&t=788
Python script for dewarping: https://github.com/Blender3D/Laser-based-Book-Dewarping
Line lasers available for purchase: http://www.dx.com/p/red-laser-module-fo ... m-5mw-5928
AC power controller (for floodlights): http://www.powerswitchtail.com/Pages/default.aspx

Videos:

Demo of a laser scanner system: http://www.youtube.com/watch?v=zS68EhEUfSY
Dan's videos:
http://www.youtube.com/watch?v=0GNIQlIZ9Pc
http://www.youtube.com/watch?v=ka4u7oKKSRI
http://www.youtube.com/watch?v=o-Ufwn08HGE
duerig
Posts: 388
Joined: 01 Jun 2014, 17:04
Number of books owned: 1000
Country: United States of America

Re: Laser Scan Methods

Post by duerig »

Hardware and build considerations:

In addition to straightening out the curve of pages, the platen also keeps them in place. With a laser-based system, you no longer have a platen. This means that you will be holding the pages in place with your hands. This means that a foot pedal or other hands-free triggering mechanism is now required.

In order to take good scans, you need an extremely well-lit page. But the floodlights or other lighting mechanism will overwhelm the relatively dim light of the lasers. This means that you will need a way to turn on/off the floodlights whenever there is a laser shot. Above, I linked to the PowerSwitch Tail II in resources which allows you to do this from an Arduino or Raspberry Pi board. But it could be a manual switch as well. There are also home automation outlets that are coming onto the market that allow AC power based on a wireless signal. These are a bit pricey and tend to take a bit longer than I'd like to switch on and off (a few seconds from the time the command is issued to actually switching).

While the exact position and angles of the camera don't matter as much, it is still useful to get it approximately right. The more dewarping that happens, the worse the quality will be. So it is best to try to ensure that a minimum of dewarping is necessary. So it is still best to have two cameras, each angled towards a facing page in a cradle.

On the standard build, there is no need for the cradle to be rolling or for the platen mounts or platen construction. Since these are some of the more complicated pieces to make, it should be much easier.

The lasers should be at a 45 degree angle relative to the plane of your base.

Dan and the other people who have done this previously have mounted two lasers at 45 degree angles but from opposite directions relative to the camera. My build at the moment has multiple lasers in parallel instead. I am experimenting to try to determine which of these is the best and why.
duerig
Posts: 388
Joined: 01 Jun 2014, 17:04
Number of books owned: 1000
Country: United States of America

Re: Laser Scan Methods

Post by duerig »

An analysis of the python dewarping script:

In the previous exploration of this space, a Python script for dewarping was developed and is available on Github here: https://github.com/Blender3D/Laser-based-Book-Dewarping

I have tweaked and tinkered with this script and there are three basic pieces. The main script is 'dewarp.py' from what I can see. The 'warp.py' is a copy of another script which uses different methods for dewarping a page.

There are three major pieces to this script which are run in order.

(1) Analyze a picture, isolate out the laser lines, and return two arrays (top and bottom) of points which describe the two expected laser lines. The laser pixels will be bright green, so it splits the image into the three color channels. First, it subtracts the blue channel from the green channel so that white/grey pixels that have non-green coloration will have a lower value. Then it takes everything that still has a fair amount of green and sets its value to 255. Finally, it uses a smoothing algorithm to try to remove any random speckles in the picture that might remain. Now it can iterate through each column of the picture and find the two lines.

(2) Transform the top and bottom lines based on geometry. This is the part that is fuzziest to me. I understand the pieces, but as a whole it doesn't gel. There are several relatively arbitrary parameter choices which likely need to be tweaked for different book setups. The overall idea is to project a line from the camera, through a point in the top/bottom, and see where it intersects with the plane the book is on. After this, there are two arrays (line_bottom, line_top) which are the transformed coordinates of top and bottom.

(3) The actual dewarping of the image now occurs. This is done column by column. The area between line_bottom and line_top is cropped then stretched to be the whole height of the image. Regardless of whether steps 1/2 are correct, this needs to be reworked in some way since the dewarping should only stretch/contract pieces to new proportions and should not be cropping any piece of the image.
duerig
Posts: 388
Joined: 01 Jun 2014, 17:04
Number of books owned: 1000
Country: United States of America

Re: Laser Scan Methods

Post by duerig »

Laser Deskew and Spine Detection:

In addition to the more complicated problem of dewarping generally, analysis of the laser pattern should allow easy deskewing and spine detection. The extreme minimum/maximum of the two lines provides us with two points that show us the spine. The spine detection allows us to crop/split the pages without reference to heuristics that try to determine the spine by looking at vertical lines. This should be a more reliable method.

Further, the angle of the line between the two points should describe a vertical line if the camera and book are properly aligned with respect to one another. By determining the relative slope of the line, we can determine the skew of the image and deskew it automatically with no heuristics.

While heuristic techniques for page splitting and deskewing work well for most pages, the laser analysis should work on all pages, including those that are photos or other shapes that might otherwise cause a bogus result.
duerig
Posts: 388
Joined: 01 Jun 2014, 17:04
Number of books owned: 1000
Country: United States of America

Re: Laser Scan Methods

Post by duerig »

Some promising results:

I have been tinkering with a number of more or less naive dewarping algorithms using the raw data provided by lasers as the basis. I have not had very much success trying to use a 45 degree angle as suggested by Dan and others.

I have also now tried something new. I have suspended lasers above the pages, nearly perpendicular to the plane that the book sits on. At this angle, the laser tracks any straight lines across a page instead of skewing through them vertically. So far the results look quite promising.

Our initial image (ignore the bad lighting): https://files.app.net/23j1bLOU-

First pass where the laser pixels are detected and set to white while everything else is set to black: https://files.app.net/23j1sXrXx

Second pass where a single set of coordinates is created for each laser line: https://files.app.net/23j1dr9_g

Third pass where an estimated line is determined where the laser 'ought' to be if there were no warping: https://files.app.net/23j14htzn

---

There are two algorithms I have tried running based on the results above. Both are naive since I don't know much about image processing. But the results are surprisingly good and I would wager that we could get even better with better algorithms.

Naive tri-partite stretching. For each column, stretch the top middle and bottom to match where the image 'ought' to be: https://files.app.net/23j1neQlI

Naive offset interpolation. For each column, set pixels by interpolating an offset weighted by distance to the two laser lines and their offsets: https://files.app.net/23j1ti9bl

---

If you compare the original to the stretching and the interpolation, make sure to zoom in and look at the four corners and the middle. If there are any libraries to try out or algorithms, please suggest them here along with an explanation of how to pass in the laser points in a way that makes sense.
duerig
Posts: 388
Joined: 01 Jun 2014, 17:04
Number of books owned: 1000
Country: United States of America

Re: Laser Scan Methods

Post by duerig »

I've been working on getting the software side of laser scanning working. After spending a lot of time trying various naive algorithms or converting equations from academic papers into code, I hit upon a simpler solution.

I have modified ScanTailor CLI to accept the laser curves as a distortion model. This lets me use ScanTailor's existing dewarping algorithm which is pretty good. Below are some sample page scans using this method. I apologize for the inconsistent lighting in these images. I'm still tweaking that part of my rig.

In each album, you can see the original image, the image with lasers taken just before, and three resulting dewarp candidates for comparison.

Auto: This uses ScanTailor with dewarping=auto and deskew=auto (I had to fix the argument parsing for dewarping in my version because there was a bug which caused this parameter to be ignored).
Laser: This uses ScanTailor with a new dewarping=manual option and passes in an XML file of the laser curve sampled once every fifty pixels. deskew=manual is also set because deskewing before the manual dewarping phase seems to cause problems.
Laser-deskew: This combines the above two techniques. The image is passed through ScanTailor twice. The first time, it uses the manual laser dewarping data. The second time it is passed through with dewarping=auto and deskew=auto.

Test Page 001: https://www.flickr.com/photos/126962164 ... 126525758/
Test Page 003: https://www.flickr.com/photos/126962164 ... 128101040/
Test Page 005: https://www.flickr.com/photos/126962164 ... 486040016/
Test Page 007: https://www.flickr.com/photos/126962164 ... 544232365/
Test Page 009: https://www.flickr.com/photos/126962164 ... 526386611/
Test Page 011: https://www.flickr.com/photos/126962164 ... 123684157/
Test Page 013: https://www.flickr.com/photos/126962164 ... 128326750/

Analysis: Laser-dewarping on its own usually does not square paragraphs and make headings completely horizontal. This is for two potential reasons. One may be a calibration issue with the lasers or camera. My current prototype involves a lot of tape and the two lasers are certainly at subtly different angles even if they form parallel lines on a flat surface. The second problem is that if a page is skewed (the top is curled slightly more than the bottom or vice versa), this is not really reflected in the laser lines. The lasers can only be used to determine if the book as a whole is skewed by determining the line of the spine of the book. See Page 001 for example.

Auto-dewarping in ScanTailor does not deal with cylindrical page shapes as well as the laser-dewarping does. It is much more likely to flatten or deform the characters, leaving them foreshortened. The laser-dewarping algorithm successfully reconstructs them in many more cases, bringing their proportions and shape back to where it should be. See page 007 for an example of this. Look along the left edge, especially near the top-left corner.

Combining laser-dewarping and auto-dewarping in a two-pass phases seems to retain both of these advantages. The paragraphs and lines are very close to how they are in the auto-dewarp case. While individual letters are decurled similar to how they look in the laser-dewarping scenario.

Sometimes if there is an extreme amount of warping involved, none of the dewarping methods are completely satisfactory. See page 013 for an example.

All in all, these results make me think that it is feasible to use lasers together with ScanTailor to successfully scan books without a platen. In order to do this, the lasers must be nearly perpendicular to the plane that the book is resting on. And both the lasers and lights must be controlled by computer. Next steps include adding support for even pages (you might notice these are all odd pages), cleaning up the code, reinforcing and calibrating my laser supports, testing lasers every nth page turn, and more.

Please let me know if you are interested in collaborating and I can share my code and more details of how to set up a laser-based scanner.
User avatar
daniel_reetz
Posts: 2812
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Laser Scan Methods

Post by daniel_reetz »

I have modified ScanTailor CLI to accept the laser curves as a distortion model. This lets me use ScanTailor's existing dewarping algorithm which is pretty good. Below are some sample page scans using this method. I apologize for the inconsistent lighting in these images. I'm still tweaking that part of my rig
Holy crap, that's brilliant.

I would *LOVE* to collaborate with you on this, though I am not very useful with code. I can definitely help with the optical side of things, though. I've always felt terrible that laser-based dewarping was never taken further...
Tulon
Posts: 687
Joined: 03 Oct 2009, 06:13
Number of books owned: 0
Location: London, UK
Contact:

Re: Laser Scan Methods

Post by Tulon »

Feeding the laser curves into Scan Tailor is a great idea, however Scan Tailor's distortion model also needs the vertical boundaries of content in addition to those curves. In the source code it's assumed that vertical boundaries pass through the endpoints of horizontal curves. Unfortunately, lasers can't provide that kind of information. BTW, these boundaries don't strictly have to be content boundaries, though they do have to be vertical relative to the content. That's the reason you are benefiting from doing a separate pass with deskew - it tends to make the content boundaries more vertical in the picture.
Scan Tailor experimental doesn't output 96 DPI images. It's just what your software shows when DPI information is missing. Usually what you get is input DPI times the resolution enhancement factor.
duerig
Posts: 388
Joined: 01 Jun 2014, 17:04
Number of books owned: 1000
Country: United States of America

Re: Laser Scan Methods

Post by duerig »

Tulon, that is absolutely right. What I do is spine detection. I had surmised that the endpoints were supposed to be aligned with the content. By aligning them with the spine, I can make sure that they are correctly aligned relative to the skew of the book as a whole. Which means we no longer need an auto-centering cradle in this setup. But when the page is inconsistently curled, that means that my spine-alignment isn't right with respect to the content. Because the top part of the text is closer to the spine than the bottom or vice versa.

In addition to my current patch where I just pass in the distortion model and ST runs it unchanged, I have considered trying to patch it so that there is an option to pass in two curves across the whole page and let ST try to figure out the left and right edges of the content and cut off the curves accordingly. I'm not sure how hard this hybrid mode would be. And running the same thing through ST a second time on dewarp=auto seems to achieve the same effect.
guitarguy
Posts: 7
Joined: 12 Sep 2014, 11:44
Number of books owned: 0
Country: Germany

Re: Laser Scan Methods

Post by guitarguy »

Hi there!

What is the current status of your project? Are you satisfied with the results your Scan Tailor modification is giving you?

As I said in the Hello thread, I would love to participate!
You said there were some papers with promising methods you want to try out?

Cheers
Chris
Post Reply