ideas on ST Picture Zone handling in mixed mode

[dtic]

Imagine you want to process a 200 page book that contains mostly only plain black text but with around 10 pages with both text and grayscale diagrams, diagrams that look best in Scan Tailors mixed mode. What is the quickest workflow for that? I do this:
1. Process all pages in "black and white" mode. (For extra speed use a script to run Scan Tailor command line in parallell to make use of multiple CPU cores.)
2. Browse the Scan Tailor GUI, or the thumbnails for the .tif output, for pages with diagrams. One by one, change those pages to "mixed" mode and adjust the picture zones to fit the diagrams.

A bottleneck here is the picture zone adjustment. Diagram areas in books are often rectangular. But Scan Tailor often by default detects a non-rectangular picture zone. For example some text within a diagram may be detected as text while other text parts are detected as images.

In those cases it takes many manual steps to modify an incorrect default zone.
1. in the Output step mixed mode click "Picture zones" tab.
2. click five times to trace a rectangle that cover all of the incorrect default zones.
3. right click the rectangle > properties > "subtract from auto layer" > ok. This removes all the default made zones.
4. click five times to trace a rectangle that cover the area you want as picture zone
5. right click the rectangle > properties (if the two rectangles cover each other, make sure to click the right menu item here!) > "add to auto layer" > ok

Steps 2 3 are of course only needed if some default areas are outside of the rectangular area you want.

Here are some ideas on how this picture zoning could be streamlined
A. An option for mixed mode to set no default picture zone at all or to set a default only if Scan Tailor detects a roughly rectangular picture area
B. Allow drawing a zone rectangle by click-and-hold, draw and click-release.
C1. Add a hotkey to without prompt delete all picture zones for the current image.
C2. Add a hotkey to without prompt delete a currently selected picture zone.
D. Allow operation B to be done on top of the output preview image even when in black and white mode. For example the user could do control + click-and-hold, draw and click-release on top of the preview image. Scan Tailor would then react to that by switching to mixed mode, applying the drawn picture zone, outputting the tif and displaying the new tif preview.
E. Allow Scan Tailor command line to take a .scantailor file as input and output an image according to the project file data. This way third party scripts/tools could be used for selecting picture zones, thus generating picture zone data, writing that data into a Scan Tailor project file, which Scan Tailor then processes. (I think I've read that Pejuko had plans for ST enhance to take project files as input but I don't know how much work that would involve.)

What do people here think of these suggestions? Any other ideas on how to adjust picture zones quicker?

[jera2]

Hi dtic,

I think your suggestions are good! And I didn't know one could delete the default picture zone. That always annoyed me to no end. So thanks for pointing that out!

I use about the same workflow, starting off with everything in B/W mode and then changing to mixed-mode on pages as needed.

I really like suggestion D, allowing a rubber-band rectangle to be drawn across the preview image, thus changing to mixed-mode and applying that rectangle as the picture zone.

And I think that the default picture-zone method should be a rectangle. A bit of user-interface work, for sure, but it would be much appreciated.

[dtic]

I discovered that Scan Tailor Enhanced already can take a project file as input and output a tif. After that I made QuickPicZone, a small windows helper program that has the features of D and E above. See the QuickPicZone thread here.

[jaffamuffin]

I have a project whereby I need to as one part of it, extract the pictures from my scanned books.

What is the best way to do this ? Is it marking out the picture areas with the red box and then somehow reading back the coords from the XML file? The problem with that is it seems to refer to the master image (I scan open book and use ST to split as well) This master image is undeskwed etc so the output isn't as nice.

I thought to use the automasks, but they don't add the user added masks seemingly. The automask does indeed pick up a lot of my pictures, so having to go through and manually draw round each one (with 4 clicks instead of eg 2 TL and BL or 1, click and drag) is a royal pain.

Any ideas ?

[dtic]

Let me first see if I understand your question.

Do you want ST to output tif files that contain the whole book page processed in mixed mode (text parts processed as black and white and illustrations on the book page processed as color)? That is what QuickPicZone assists with.

Or do you want to extract only pictures from book pages and remove all the surrounding text?

[jaffamuffin]

Thanks. I will look at the software today.

To be complete i need:
From scanned open book images i am generally happy with the tif outputs after running throuagh all stages of scan tailor. I get binary text and colour pictures (mixed). Sometimes i have to zone extra pics and ither areas sometimes st picks them up up. (These files goto ocr through abbyy)

Also addition to this i also need:
Each picture seperate in its own file

And....

A text file of the caption


I somehow need to extract all the pictures and know which page they came from so i can key the caption (into a custom php app i wrote)

Do i need to mark all pics with the zone or can i somehow extract even auto detected pics from the st project?

Previously i had processed the images in st then used used a custom php process to 1)tag pages with pics and the number of pics2) then display only those pages and keyed the caption and the 'fig' number 3) ran a process to copy x number of copies of that page (one for each pic) named after the figure number 4) manually cropped in irfanview the relevant pic according to the filename and save snd overwrite. 5) export the caption data and creat a 3 colum csv with image path,fig no, caption


As you can see very fiddly and if i can auto extract the pics as im already zoning them it would save a bunch of work

Thanks

[dtic]

Ok, I think you will have to do those two things separately.

QuickPicZone can help you set custom picture zones in mixed mode with fewer clicks that doing so in the Scan Tailor GUI. But several things has to be in place for that to work. First you need only one book page per image. Second, if that images needs a lot of deskewing you will have to do that in advance too, or else the picture zones will be misplaced. Third, you must run each image in scan tailor black and white mode and save a separate .scantailor project file for each image. You can do that using the cli version of Scan Tailor Enhanced.

Once that is done you input to QuickPicZone and set picture zones.

To split photos with two book pages you can use BookCrop.

Your second goal was to extract only the illustration parts from book page photos. I haven't seen anyone here do that. However if you have a .scantailor project files with the picture zones you want set then you should be able to extract the data you need to copy the illustrations. Search for <zone> tags in the .scantailor file and you'll see picture zone coordinates. If your project file also has deskew settings then things get more complicated.

Another idea is to try to extract the illustrations from the mixed mode output tif files, using some different tool. Perhaps you can use OpenCV or another computer vision tool to detect and extract all and only grayscale/color segments of those files.

[jaffamuffin]

Ok, I think you will have to do those two things separately.

QuickPicZone can help you set custom picture zones in mixed mode with fewer clicks that doing so in the Scan Tailor GUI. But several things has to be in place for that to work. First you need only one book page per image. Second, if that images needs a lot of deskewing you will have to do that in advance too, or else the picture zones will be misplaced. Third, you must run each image in scan tailor black and white mode and save a separate .scantailor project file for each image. You can do that using the cli version of Scan Tailor Enhanced.

Once that is done you input to QuickPicZone and set picture zones.

OK, that explains it a little, but it looks to me that actually in order to save some clicks in ST I have to do a whole bunch of other work first , i.e. deskew, splitting, running through to black and white, and then I can click less to set a picture zone.

My workflow is generally batch run through on mixed mode, then open job and make tweaks / i.e. fix some deskews, fix picture zones, re-adjust some content areas.

To split photos with two book pages you can use BookCrop.

Interesting app, will check this out. I actually find scantailor splitting to be pretty excellent, just there seems to be no way to JUST split the images and dump, anything in ST has to go through it's process.(content and margins)

Your second goal was to extract only the illustration parts from book page photos. I haven't seen anyone here do that. However if you have a .scantailor project files with the picture zones you want set then you should be able to extract the data you need to copy the illustrations. Search for <zone> tags in the .scantailor file and you'll see picture zone coordinates. If your project file also has deskew settings then things get more complicated.

Yeah, I noticed this - also, they seem to refer to to master image so I have e.g. a coord reference at 3,500 pixels in - this is the right hand page of the double page spread, seems to me no way to get the coords of the 'processed' file, just the input file.

Another idea is to try to extract the illustrations from the mixed mode output tif files, using some different tool. Perhaps you can use OpenCV or another computer vision tool to detect and extract all and only grayscale/color segments of those files.

Yeah, but that will diverge my workflow - Not only do I have to grab colour pics, but black and white diagrams etc as well. I'm probably better off approaching from the same angle as I did before. If only ST just had some kind of 'zoning' and you could arbitraily assign particular processing mode (colour/b&W) and a 'tag' i.e. (pic/diagram/table/text) then I could zip through and zone the book, and then just extract all the stuff.

My other thought was to use something like this : http://iakob.com/canvas-area-draw/demo.html and code something aroudn it but obviously that is a bit of work required.

My main issue is I run the images through scan tailor and zone the pictures etc, but then I also have to manually go through and crop all the pictures out... The images are already zoned I should be able to extract the data, even the data that is auto generated in mixed mode.

[cday]

Another idea is to try to extract the illustrations from the mixed mode output tif files, using some different tool. Perhaps you can use OpenCV or another computer vision tool to detect and extract all and only grayscale/color segments of those files.

That might be possible using Adobe Acrobat, by first converting the tiff page images to a PDF file using the ClearType option, which will save the images on the tiff pages separately from the text, then extracting those images from the resulting PDF file using Acrobat again...

If jaffamuffin has Acrobat...

I can't test that as I don't have a modern version of Acrobat, but the PDF file that is created must contain the image content as streams, and I'm sure that it is possible to extract the images in a file as separate images using Acrobat.

[dtic]

@jaffamuffin
Yeah your workflow is a bit different compared to the one I made QuickPicZone to fit.

Some more ideas though. You could first use Scan Tailor to separate, deskew and crop pages and then output them as color/grayscale tif files. Then batch process those tifs in ScanTailor black and white mode on the command line and save individual .scantailor files. Use the command line parameters to disable content detection and set margins to zero, in order to save the whole page. Then use QuickPicZone to manually set picture zones. (For this to work I'll have to update QPZ to accept not only jpg but also tif source files but that is easy to do.) The coordinates saved by QuickPicZone should then be usable to also extract all picture zones as individual images.

Search for <zone> tags in the .scantailor file and you'll see picture zone coordinates.

Yeah, I noticed this - also, they seem to refer to to master image so I have e.g. a coord reference at 3,500 pixels in - this is the right hand page of the double page spread, seems to me no way to get the coords of the 'processed' file, just the input file.

Well the .scantailor file also includes deskew data, search for <deskew . But you'd have to do some math operations with that and the coordinates to get the real picture zone coordinates on the output tif files.

@cday Good thinking! I bet the same could be done with free tools too, both to generate the pdf and to extract the images. This thread has some leads http://stackoverflow.com/questions/2693 ... -in-python