Daniel Reetz, the founder of the DIY Book Scanner community, has recently started making videos of prototyping and shop tips. If you are tinkering with a book scanner (or any other project) in your home shop, these tips will come in handy. https://www.youtube.com/channel/UCn0gq8 ... g_8K1nfInQ

Scan Tailor

Scan Tailor specific announcements, releases, workflows, tips, etc. NO FEATURE REQUESTS IN THIS FORUM, please.
Locked
Tulon
Posts: 687
Joined: 03 Oct 2009, 06:13
Number of books owned: 0
Location: London, UK
Contact:

Re: Scan Tailor

Post by Tulon » 02 Apr 2010, 03:52

Amjad wrote:The thing is, here I wanted to modify the text, namely, to make it bolder.
I see. First of all, you could adjust ST's binarization threshold. I've extended the adjustment range to [-30 .. +30] in 0.9.8.
Separation into text / non text images and postprocessing using 3rd party software would be your last resort.
Amjad wrote:Would it really be that hard to add manual zones to the mask?
cache/automask files are meant to contain auto detected pictures only. They exist to be able to go in picture zone editing mode without having to go through the whole output process every time. And no, you can't just take the automask file and draw polygons from the project file on top of it. That's because those polygons are in the coordinate system of the input, not output image.

Anyway, there is a simple (for a programmer) way to extract picture mask right from the output file. You just read it pixel by pixel and set the corresponding pixel of the mask to 1 if your output pixel is not pure black or pure white, and 0 otherwise. Scan Tailor enforces that assumption. In fact, that's exactly what Layer Tailor and ST Separator do. I could code my own version in an hour, though I am busy with other things right now.
Scan Tailor experimental doesn't output 96 DPI images. It's just what your software shows when DPI information is missing. Usually what you get is input DPI times the resolution enhancement factor.

Amjad
Posts: 8
Joined: 31 Mar 2010, 01:49

Re: Scan Tailor

Post by Amjad » 02 Apr 2010, 05:32

Tulon wrote:I've extended the adjustment range to [-30 .. +30] in 0.9.8.
Separation into text / non text images and postprocessing using 3rd party software would be your last resort.
Great! ST's output will be even better! I'm actually resorting to postprocessing, because it's so hard to get the pictures right with my scanner, that I tuned its settings in a way that kinda messes the text a bit. My reasoning is that it's much easier to fix the text later (hence the need to get it in a separate image) + once the first page is fixed, a Photoshop action can do the rest for me.
Tulon wrote:And no, you can't just take the automask file and draw polygons from the project file on top of it. That's because those polygons are in the coordinate system of the input, not output image.
I think you mean the opposite. I'm using the coordinates from the <output><page id><zones><zone><spline> node, and do the processing on the output from ST configured with : "Color / Grayscale", no descpeckling (I go in Mixed mode just to get the zones, save the project, and then do the processing in color mode without saving the project again. I know, I'm :evil: like that).
Tulon wrote:Anyway, there is a simple (for a programmer) way to extract picture mask right from the output file. You just read it pixel by pixel and set the corresponding pixel of the mask to 1 if your output pixel is not pure black or pure white, and 0 otherwise.
I know, that's exactly what I did!
As I said, I already coded it. It's easy with CxImage, see:

Code: Select all

    DWORD height = mask.GetHeight();
    DWORD width = mask.GetWidth();
    
    POINT points[4];
    points[0].x=width-1059; points[0].y=height-2621;
    points[1].x=width-2400; points[1].y=height-2598;
    points[2].x=width-2309; points[2].y=height-3094;
    points[3].x=width-1077; points[3].y=height-3098; 
    image.SelectionAddPolygon(points, 4);
    
    imPic.Copy(image); // picture layer
    
    // Apply mask. image becomes text layer:
    RGBQUAD pixColor, rgbWhite = {255,255,255,0};
    for (DWORD x = 1; x <= width; x++) {
        for (DWORD y = 1; y <= height; y++) {
            pixColor = mask.GetPixelColor(x, y, false);
            if (pixColor.rgbBlue || image.SelectionIsInside(x, y)) // black==0, white==255
                image.SetPixelColor(x, y, rgbWhite, true);  
            else 
                imPic.SetPixelColor(x, y, rgbWhite, true);
        }
    }
    
    // Save result
    if ( image.Save("image_txt.tif", CXIMAGE_FORMAT_TIF) )
        cout << "Successfully saved text layer to image_txt.tif" << endl;
    else { cout << "Saving text layer failed !" << endl; ret = 1; }
    
    if ( imPic.Save("image_pic.tif", CXIMAGE_FORMAT_TIF) )
        cout << "Successfully saved picture layer to image_pic.tif" << endl;
    else { cout << "Saving picture layer failed !" << endl; ret = 1; }
It works perfectly.
(Regarding the extraction of the coordinates from the project file, don't ask me :oops: I should use some XML parser, but I rather flatten the file up with rxp (using Cygwin) and pipe it into an Awk script, The awk script is ugly, but I'm actually so good with Unix scripting (ksh, awk, sed, & co.) that it costs me almost zero effort. Laziness is sweet :mrgreen: ).

Bottom line, I'm only using ST for automatic deskew, content selection & graphics zones identification.
I take care of ALL the rest myself! You should be ashamed!
:mrgreen: :mrgreen:

Seriously, you already did a tremendous amount of work, so don't bother yourself with this, and focus on 0.9.8!

Tulon
Posts: 687
Joined: 03 Oct 2009, 06:13
Number of books owned: 0
Location: London, UK
Contact:

Re: Scan Tailor

Post by Tulon » 03 Apr 2010, 17:25

Scan Tailor 0.9.8 has been released. Take it from the official website.

I was going to focus on dewarping next, but I changed my mind. Recent discussions here made me think there is a more important task at hand. That task would be assembiling post-processed pages into a PDF or DJVU file. I am going to write a simple GUI program to do just that. Right now this task requires either lots of efforts, or costly commercial software, or you end up with huge files. My program would only target DJVU, at least initially. I think a couple of months should be enough to build it.
Scan Tailor experimental doesn't output 96 DPI images. It's just what your software shows when DPI information is missing. Usually what you get is input DPI times the resolution enhancement factor.

phaedrus
Posts: 56
Joined: 04 Mar 2014, 00:52

Re: Scan Tailor

Post by phaedrus » 04 Apr 2010, 05:24

Tulon. Thanks for all your work, I'll give 0.9.8 a try in the next few days I hope :-)

I'm interested in your comments re DJVU, PDF & de-warping. I can't comment on DJVU creation but I do find it fairly simple to use a combination of Irfanview & Cutepdf to produce a very reasonable PDF version of anything ST has output. The only reason I use Irfanview is simply to select all the files in the directory (thumbnail view) and print them to the Cutepdf printer I have loaded. If ST were to have an option for simply printing its output in order then one could do away with the Irfanview step and print directly to Cutepdf. Seems a fairly simple and elegant step to me that may possibly not require a lot of coding? Maybe there's a similar option for 'printing' to DJVU files if a 'print output' button were available (I'm not at all familiar with DJVU but perhaps there's a 'Cutedjvu' app somewhere!).

My reason for discussing this is that, to me, the PDF (and possibly DJVU) output is manageable reasonably easily with currently - and freely - available options and perhaps by adding a 'print output' option could be simplified even further. On the other hand de-warping is not really manageable by anyone but a coding-god or other software masochist (at least that's the way it seems to me!). As such I wonder if your skills would be better employed dealing with this need and maybe a print-output option rather than actually having ST produce PDF or DJVU files directly? It's probably obvious I know nothing about coding but I thought I'd bring up what I thought when I read your post :-)

Rgds, P.

Tulon
Posts: 687
Joined: 03 Oct 2009, 06:13
Number of books owned: 0
Location: London, UK
Contact:

Re: Scan Tailor

Post by Tulon » 04 Apr 2010, 06:23

phaedrus wrote:I can't comment on DJVU creation but I do find it fairly simple to use a combination of Irfanview & Cutepdf to produce a very reasonable PDF version of anything ST has output.
You would end up with huge PDFs then, no? What would be the size of a 300 page book with no pictures?
When the size is big, it's not suitable for internet distribution. Internet distribution is what I care about. I've just went to a certain underground website just to collect some statistics. They have 217 books on linear algebra. Scientific literature is their specialization. Only 4 of them are over 10MB and none is over 15. Note that most of those books are not OCRed. Surely, hosting 600MB books would not be an option for them.
Scan Tailor experimental doesn't output 96 DPI images. It's just what your software shows when DPI information is missing. Usually what you get is input DPI times the resolution enhancement factor.

dtic
Posts: 463
Joined: 06 Mar 2010, 18:03

Re: Scan Tailor

Post by dtic » 04 Apr 2010, 06:27

Tulon wrote:Recent discussions here made me think there is a more important task at hand. That task would be assembiling post-processed pages into a PDF or DJVU file. I am going to write a simple GUI program to do just that. Right now this task requires either lots of efforts, or costly commercial software, or you end up with huge files. My program would only target DJVU, at least initially. I think a couple of months should be enough to build it.
For djvu and OCR there is already fine FOSS available it seems to me. See http://www.diybookscanner.org/forum/vie ... ?f=3&t=319 .
I'm no programmer but I made a GUI frontend that transforms Scan Tailor output into OCR'ed djvu in one manual step. I will post it very shortly.

Amjad
Posts: 8
Joined: 31 Mar 2010, 01:49

Re: Scan Tailor

Post by Amjad » 04 Apr 2010, 07:25

Tulon: thanks for the release, and for your continued efforts.
The new version is under torture, and any secrets revealed will be on your desktop asap!
Would you mind a few words on the new features/most important fixes, and/or what's important to test?

phaedrus & dict: regarding the DjVu GUI, what Tulon is offering is actually completely new. Under Windows, at least, there is no such free GUI (to my knowledge). The only available option is software from Celartem.com (Document Express), which is expensive.

Having to press buttons instead of crafting scripts is certainly nice (well, me I can script during my sleep, so I'm not the best client), but I want to let Tulon know that the most prominent feature in the commercial solutions is actually the ability to fine-tune DjVu encoder options, visually. That is, to change the parameters and press a button to immediately see the effect on a test image of your choice (encoded image & size), before you commit that set of params (profile) to an 800 files encode task!
While I'm almost sure you would consider making some nice default "profiles" for everyone to have something to cover their general needs (for average Joe, there is no point in having a GUI without those, actually), the visual fine-tuning widget, now THAT would really bring DjVu to the masses by unleashing the true power of the beast, not hiding its great potential behind eerie-looking intimidating gazillion non-intuitive command-line options.
Just in the hope that this one "little" feature maybe would make it to your draw-board ;)

But once again, thank you very much for the awesome job, there is definitely a world before and after Scan Tailor for the whole scanning community.
Where have been hiding before?! :mrgreen:

Tulon
Posts: 687
Joined: 03 Oct 2009, 06:13
Number of books owned: 0
Location: London, UK
Contact:

Re: Scan Tailor

Post by Tulon » 04 Apr 2010, 08:58

dtic wrote:For djvu and OCR there is already fine FOSS available it seems to me. See http://www.diybookscanner.org/forum/vie ... ?f=3&t=319. I'm no programmer but I made a GUI frontend that transforms Scan Tailor output into OCR'ed djvu in one manual step. I will post it very shortly.
I'd certainly like to see that. I expect serious shortcomings though, like inability to handle halftone pictures.
Scan Tailor experimental doesn't output 96 DPI images. It's just what your software shows when DPI information is missing. Usually what you get is input DPI times the resolution enhancement factor.

User avatar
daniel_reetz
Posts: 2780
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Scan Tailor

Post by daniel_reetz » 04 Apr 2010, 10:10

Amjad wrote: Would you mind a few words on the new features/most important fixes, and/or what's important to test
Amjad, I'm sure you saw the changelog?

FWIW, I would *love* to see the DJVU/potential PDF program you're describing, Tulon. Great idea.

I put up a news post about the new version. Cheers, and thanks, as always.

Tulon
Posts: 687
Joined: 03 Oct 2009, 06:13
Number of books owned: 0
Location: London, UK
Contact:

Re: Scan Tailor

Post by Tulon » 04 Apr 2010, 10:34

Amjad wrote:Would you mind a few words on the new features/most important fixes, and/or what's important to test?
There is a more detailed changelog here: http://scantailor.git.sourceforge.net/g ... a=shortlog
Scan Tailor experimental doesn't output 96 DPI images. It's just what your software shows when DPI information is missing. Usually what you get is input DPI times the resolution enhancement factor.

Locked