Spreads for Post Processing

Johannes Baiter's Spreads and SpreadPi are the latest control systems and postprocessors for DIY scanning. http://spreads.readthedocs.org

Moderator: peterZ

RrangerRich
Posts: 5
Joined: 03 Mar 2015, 16:05
Number of books owned: 5000
Country: USA

Spreads for Post Processing

Post by RrangerRich »

I want to thank everyone for literally “tons” of valuable information.

I have run into a snag using Spreads only for Post Processing.

I have multiple books scanned in as .jpg images and want to use Spreads to Post Process them.

I have been able to Spreads to work fine (as far as I know), but when I launch it in Post Process mode and point it to a directory with the .jpg images, it keeps returning that the directory was empty and not doing anything.

Is there a file structure or specific files that are needed for Spreads to process the images?

Thanks,
Rich
duerig
Posts: 388
Joined: 01 Jun 2014, 17:04
Number of books owned: 1000
Country: United States of America

Re: Spreads for Post Processing

Post by duerig »

Spreads builds up a fair amount of metadata about the files while it is scanning. But I'm not entirely sure how much is necessary when post-processing. Here is what to try first:

Make a book folder.
Inside of that, make a folder called 'data'.
Inside of that, make a folder called 'raw' and put all your .jpg images inside of it.

This may be enough for it to work properly. If not, then I or Johannes will have to look deeper into the code to see how to make it work.

-D
RrangerRich
Posts: 5
Joined: 03 Mar 2015, 16:05
Number of books owned: 5000
Country: USA

Re: Spreads for Post Processing

Post by RrangerRich »

Thanks for the information.
I had already tried that after several experimentations.
After running the spread postprocess {project directory} command, the application creates 4 files (bag-info.txt bagit.txt manifest-md5.txt tagmanifest.md5.txt) and moves the raw directory down several directories ( {project directory}/data/data/data/raw ) and then quits.
Here is the output:

~$ spread postprocess ./scans/FarCry2test1/
Workflow: Initializing workflow ./scans/FarCry2test1/
bagit: Adding path /tmp/tmpIlEZQZ/bag-info.txt to payload
bagit: Adding path /tmp/tmpIlEZQZ/bag-info.txt to payload
bagit: Adding path /tmp/tmpIlEZQZ/bag-info.txt to payload
bagit: Adding path /tmp/tmpIlEZQZ/bag-info.txt to payload
bagit: Copying path scans/FarCry2test1/data to paylod directory
bagit: Adding path /tmp/tmpIlEZQZ/bag-info.txt to payload
spreads encountered an error:
Traceback (most recent call last):
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/main.py", line 318, in main
run()
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/main.py", line 308, in run
args.subcommand(config)
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/cli.py", line 352, in postprocess
workflow = spreads.workflow.Workflow(config=config, path=path)
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/workflow.py", line 431, in __init__
self.slug = util.slugify(unicode(self.path.name))
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/workflow.py", line 492, in slug
self.bag.info['spreads-slug'] = value
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/vendor/bagit.py", line 684, in __setitem__
self.save()
File "/home/engineer/.spreads/local/lib/python2.7/site-packages/spreads/vendor/bagit.py", line 752, in save
with open(self._path, "wb") as fp:
IOError: [Errno 2] No such file or directory: u'/tmp/tmpIlEZQZ/bag-info.txt'

I think that you are right and that some metadata or information about the images is missing and it is confusing the system.

Thanks for the help. 8-)
User avatar
jbaiter
Posts: 98
Joined: 17 Jun 2013, 16:42
E-book readers owned: 2
Number of books owned: 0
Country: Germany
Location: Munich, Germany
Contact:

Re: Spreads for Post Processing

Post by jbaiter »

  • What version of spreads are you using?
  • Did you scan with spreads?
  • What's the content of "./scans/FarCry2test1/" before you run postprocessing?
spreads: Command-line workflow assistant
RrangerRich
Posts: 5
Joined: 03 Mar 2015, 16:05
Number of books owned: 5000
Country: USA

Re: Spreads for Post Processing

Post by RrangerRich »

Hello.
Sorry it took forever for me to respond. I got “surged” at work and lost most of my free time.

The version of Spreads is reporting as 0.5git20150203.220d

I didn’t perform the scans with Spreads. I captured them with a sheet feed scanner and a flatbed scanner. So the images are all .jpg files.

The contents of the “./scan/FarCry2test1/” folder before running Spreads in postprocessing is this:

scans
|
data
|
raw
|
FarCry001.jpg
FarCry002.jpg
.
.
.
FarCry020.jpg

Make sense?

There are no other files in the folders other than the .jpg files.

Thanks for all the help.
User avatar
jbaiter
Posts: 98
Joined: 17 Jun 2013, 16:42
E-book readers owned: 2
Number of books owned: 0
Country: Germany
Location: Munich, Germany
Contact:

Re: Spreads for Post Processing

Post by jbaiter »

Thank you, I think you found a bug :-) I'll look more closely into it and hopefully write a test and a fix and get back to you!
spreads: Command-line workflow assistant
User avatar
jbaiter
Posts: 98
Joined: 17 Jun 2013, 16:42
E-book readers owned: 2
Number of books owned: 0
Country: Germany
Location: Munich, Germany
Contact:

Re: Spreads for Post Processing

Post by jbaiter »

RrangerRich, this bug has already been fixed a few weeks back, you should just have to update to a newer version. (See here: https://github.com/DIYBookScanner/sprea ... cbee29962a).

Note, though, that the pages won't show up in the web interface and you won't be able to do anything with them. I have not yet finished the code for that, but it should be ready soon.
spreads: Command-line workflow assistant
RrangerRich
Posts: 5
Joined: 03 Mar 2015, 16:05
Number of books owned: 5000
Country: USA

Re: Spreads for Post Processing

Post by RrangerRich »

Hello,

I updated to the latest nightly build. spreads 0.5git20150426.f8de

When I run the post processing command now, there are no thrown exceptions.

I get the following text:

Workflow: Initializing workflow ./scans/FarCry2test1/
bagit: Adding path /home/engineer/scans/FarCry2test1/bag-info.txt to payload
bagit: Adding path scans/FarCry2test1/config.yml to payload
Workflow: Starting postprocessing...%
Workflow: Running 'process' hooks
spreadsplug.autorotate: Rotating images
spreadsplug.scantailor: Generating ScanTailor configuration
spreadsplug.scantailor: /usr/bin/scantailor-cli --start-filter=1 --end-filter=5 --layout=1.5 -o=/tmp/tmpFGq0s9.ScanTailor --margins-top=2.5 --margins-right=2.5 --margins-bottom=2.5 --margins-left=2.5 /tmp/st-outlcujtf
.
.
.
The a good deal off text about the ScanTailor options. (Omitted for brevity)
.
.
.
spreadsplug.tesseract: Performing OCR%
spreadsplug.tesseract: Language is "osd"
bagit: Path scans/FarCry2test1/data/done is an empty directory , will be skipped.
bagit: Adding path /home/engineer/scans/FarCry2test1/bag-info.txt to payload
bagit: Adding path scans/FarCry2test1/pagemeta.json to payload
Workflow: Done with postprocessing!

A "data" directory and a "done" directory are created in the "data" directory and the "raw" directory is moved into the created "data" directory.

The "done" directory is empty.

I appreciate all the help and hope that I am not causing you extra work because of my own ineptitude. ;)

Thanks,

Rich
User avatar
jbaiter
Posts: 98
Joined: 17 Jun 2013, 16:42
E-book readers owned: 2
Number of books owned: 0
Country: Germany
Location: Munich, Germany
Contact:

Re: Spreads for Post Processing

Post by jbaiter »

Rich, can you please upload the whole log somewhere? Ideally when running with the "--verbose" flag.
spreads: Command-line workflow assistant
RrangerRich
Posts: 5
Joined: 03 Mar 2015, 16:05
Number of books owned: 5000
Country: USA

Re: Spreads for Post Processing

Post by RrangerRich »

Hello.

Here are the commands that I ran and the resulting log files.

spread --loglevel degug --logfile ./FarCry2test1.log postprocess ./scans/FarCry2test1/

spread --loglevel debug --logfile ./FarCry2test1_out.log output ./scans/FarCry2test1/

Thanks,

Rich
Attachments
FarCry2test1_out.log
Output log file
(592 Bytes) Downloaded 498 times
FarCry2test1.log
Post Processing log file
(1.34 KiB) Downloaded 509 times
Post Reply