Daniel Learns About Spreads (on Windows, at least)

[daniel_reetz]

Thanks abmartin. I think you're right, but unfortunately the YAML installer doesn't work for me, because it can't find the Python 2.7 registry strings (damnit, I have 2.7.5 installed!!!)

[jbaiter]

Phew, I have a bad conscience now :-/
Currently spreads has been solely developed and tested on a Linux-platform, so it's highly likely that the code in its current form is not going to work on Windows, no matter how it's installed.

There are a few reasons for this:

• Python has, historically seen, never been very friendly to the Windows-Platform (I think there's currently a single developer working on CPython for Windows)
• Spreads heavily relies on the Python ecosystem (Pip+PyPi) to handle its numerous dependencies
• To deal with devices, Spreads uses a C-extension that has to be compiled (major problem on Windows) and requires a libusb-compatible USB-driver. The Linux driver is so by default, but for Windows you will need to install a custom USB driver to get the camera communication to work.
• (Purely personal and probably prejudiced:) I find the whole Windows experience confusing and frustrating (especially if you try to develop something with it), so my motiviation hasn't exactly been very high to tackle that issue

Currently, my recommendation for Windows users is to either use Spreads from a VM that runs Linux and supports USB-passthrough (VirtualBox comes to mind) or (if your CPU doesn't have virtualization extensions) from a LiveCD. I can prepare either of those plus a tutorial (VM image is the easiest method for me, LiveCD will take more time), if there is interest.

Not that I want to keep you from trying, Daniel, your experiences will be valuable once I get to the point where I start work on making it usable on Windows
That being said, here are a few pointers that might help you:

• Follow this guide for setting up Python+Pip+virtualenv on your machine: With this, you have the equivalent of my Python development environment (sans the dependencies, of course, but that's what pip is for)
• Once you got that, try "pip install spreads", this should download and install spreads with all its dependencies. This will most probably fail. Probably you won't have a compiler installed, etc. Post your experience here (or even better, open an issue on GitHub) and maybe I can find an easy fix that brings us closer to a solution.
• Do not bother with those installers for the dependencies, try to use pip as much as possible.

Sorry if this is all confusing, but spreads is very much in alpha (or even pre-alpha) state right now, the architecture is likely going to change a lot over the next few months, and currently it's very much guided by my own needs and setup (although I really try to keep it as open and modular as possible!). Rest assured, though, I'm really excited and enthusiastic about it and the possibilities (and grateful for every bit of feedback I get!) and will try to continue working on it as much as I can find mental resources and time for it.

Plus, to my defense vis à the whole Windows-issue, we as advocates of open knowledge (I get that vibe from here ) should clearly favor free and open platforms as much as possible.

[jbaiter]

After spending the last three hours trying to get the Spreads VM to recognize the cameras under Windows 7, I hereby officially give up.
Windows seems to do some stuff with the cameras in the background at all times, which prevents VirtualBox from accessing them. It does, however, correctly handle every USB device *besides* the cameras .
I've also tried installing spreads on Windows directly, however it seems to require (as expected) a compiler. Funnily enough, that compiler seems to be Visual Studio 2008, which I neither have nor intend to buy...

I've got more important things to do than do deal with crap (sorry) like that, so spreads will stay *nix-only for the moment. If you want to use it, grab a recent version of Ubuntu, install it on an external hard-disk (or your internal one, if you can spare the space) and use that for scanning.

[daniel_reetz]

Yeah, please don't spend any more time on Windows - though I will try to find a way to make VM work, you shouldn't spend any time on it. I'll just boot to my Ubuntu install and work from that.

Thanks for trying, means a lot to me.

Plus, to my defense vis à the whole Windows-issue, we as advocates of open knowledge (I get that vibe from here ) should clearly favor free and open platforms as much as possible.

I agree, and this is a VERY Open Source/Free Software/Free Culture-heavy community, but I do want to pontificate a little. When I started the project, I knew that the first version only had to support Windows, even though that seemed backward/even a little hypocritical to me. Why? Because the majority of the world, in 2009, was running Windows, especially in libraries and universities. What I needed was to get together a critical mass of people, and I made that decision deliberately. Additionally, the politics around books were and are so thick and heavy that I felt that introducing another political layer around software made things damn near impenetrable. There's another point here, that I don't want to spend a lot of time on, that people in Latin America, Russia, and China (which have huge communities of book scanning people) are running less-than-registered copies of Windows XP. Convincing these people to install Linux is basically a non-starter, especially in less-educated communities with more pressing needs. I know this firsthand from some of the more crazy/poor/critical projects I've worked on. It's not my happy place.

But times have changed, and drastically. Now Android is everywhere, and a much larger percentage of developers are developing for Mac or Linux. I'll tell you right now, I spend 80% of my time on Windows 7 X64. Why? Not because I think Windows is a good option, but because I have to - lots of the software for engineering (Solidworks, Zemax, CorelDraw, Adobe CS) are only usable, for me, under Windows, and that's what I do at my day job. For me personally, Win 7 is probably the end of my Windows usage because of Microsoft's ugly missteps, both current and historical. But I can't deny that there are thousands of PCs out there that could be put to use as scanners, and thousands of people who can't pip install anything. Many of them with really great book collections and lots of motivation but little technical ambition.

I still think it would be dreamy to just have a flash drive that boots a basic version of Linux Mint or Busybox or anything like that, and is a complete scanning environment with all necessary software.

I also think it would be extra dreamy to offload the linux-y parts to a Raspberry pi or Android or similar cheap, tiny computer. Scan Tailor already runs on every platform, so people stuck in the Windows world could still deal. I know work has begun on this.

But the most dreamiest situation, philosophically speaking, would be to work the same on all platforms. That's not something I want to ask of any developer. Rather, I'd like to see the platform made feature complete in your environment of choice, where you are the happiest and where things are most Free, and then, perhaps, put a bounty on making it work on Windows. Or take up a collection to get it to work (like Kickstarter or something). That way, we include the most people possible and get the most scanning done.

[daniel_reetz]

Note to self: Here's A Comprehensive Comparison of Linux Development Boards.

[victoriaaustralia]

Spreads does look interesting and obviously jbaiter has done a heap of work on it. I think Daniel has the Model T ford of the bookscanner in the Hackerspace kits and this Spreads will probably be debugged and used in that environment.

From my point of view the image capture/post processing is well taken care of mechanically (camera button triggers) and with ScanTailor. The most difficult component for me is compilation and compression. Homer is the only program I have used that does this and does it so well. I just this morning finished a book and Homer took it from 613pages, 1.06GB to a 30.9MB OCR'd pdf with a mixture of grayscale images and black and white. This is my current workflow using Windows Freeware:
http://www.diybookscanner.org/forum/vie ... =19&t=2835

love the work jbaiter - more bookscanning, more of the time by more people!!

[jbaiter]

I still think it would be dreamy to just have a flash drive that boots a basic version of Linux Mint or Busybox or anything like that, and is a complete scanning environment with all necessary software.

No need to be dreamy, that is completely feasible with a moderate amount of work. The basic steps are basically using another Live distribution, modifying it and creating a new image. Maybe create a little artwork and test it on a range of machines.

I also think it would be extra dreamy to offload the linux-y parts to a Raspberry pi or Android or similar cheap, tiny computer. Scan Tailor already runs on every platform, so people stuck in the Windows world could still deal. I know work has begun on this.

Indeed, it has, and that's also my primary goal right now, to get to a point where we can just drop a scanner with a RPi (or any other small, cheap SoC) somewhere, do the scanning there with minimal setup efforts (turn on Pi, wait until booted, start scanning workflow from a smartphone or maybe a connected touchscreen display, trigger capture via foot-pedal) and transfer the captured images to a more powerful machine (either via HTTP or external storage) to do the post-processing. I'm fairly confident that I can get that post-processing part to work on windows, as it doesn't have to deal with any device drivers.

[dtic]

... highly likely that the code in its current form is not going to work on Windows
There are a few reasons ... relies on the Python ecosystem (Pip+PyPi) to handle its numerous dependencies ...a C-extension that has to be compiled ... a libusb-compatible USB-driver.

I haven't gotten around to test spreads yet but will give it a shot using a linux boot disc some day or other.
But I agree with Daniel that long term the ideal would be a cross platform solution. I got one idea from reading the above. Do you jbaiter think it might be easier to get some parts of spreads to work in Windows rather than all of spreads? That is, are the issues related only to some of the steps? I ask because at least some things you mentioned looks to be related to the capture stage. For that step there are already some working solutions. A Windows version of spreads that for the time being does all steps after capture might still be useful, even if that means having to use spreads + one other tool.
In Windows photo capture can be done using chdkptp, once libusb-win32 is installed for any CHDK compatible Canon camera. chdkptp is easy to control from the command line. There is now also a remote shoot command that bypasses the camera memory card and saves directly to the computer. There are chdkptp versions available for Windows, Linux and raspi.

[jbaiter]

But I agree with Daniel that long term the ideal would be a cross platform solution. I got one idea from reading the above. Do you jbaiter think it might be easier to get some parts of spreads to work in Windows rather than all of spreads? That is, are the issues related only to some of the steps? I ask because at least some things you mentioned looks to be related to the capture stage. For that step there are already some working solutions. A Windows version of spreads that for the time being does all steps after capture might still be useful, even if that means having to use spreads + one other tool.

Yep, that is indeed feasible. My main problem at the moment under Windows is the whole communication with the devices, as this requires a C-extension that has to be compiled and for which custom drivers have to be installed, which is less than ideal from a usability perspective.
A postprocessing-only version is definitely possible and will be included in one of the next versions of spreads, most likely in the form of a little batch-script that will run the server component and host a small web-interface through which the user can manage all queued postprocessing tasks. This is part of my "Scanning on a RPi, post-processing on another machine"-plan. The architecture, as I currently envision it, will look like this:



The user will be able to start/stop/monitor/configure the scanning process on the "client" machine (RPi + Scanner), which will combine the files from the devices and send them to the postprocessing server (which will be platform-independent). Here, the task enters a queue, which the server will process one item (=bundle of scanned images plus configuration) at a time. The user can monitor and manage the queue through a web interface. It will include an option to download a ZIP-file containing the raw images and the pre-generated ScanTailor configuration and another option to upload the manually adjusted ScanTailor configuration.

A few advantages of that architecture:

• No need for screens on clients + servers, everything can be controlled from a web browser (Smartphone/Tablet/PC)
• Scalable: Many scanners could use a single post-processing machine, where everything can be controlled via a single interface, might be useful for long-running digitization projects at smaller libraries or archives.
• Loose coupling of scanning and postprocessing: The postprocessing server doesn't have to be on the same network, as everything is done via RESTful HTTP APIs. Take the scanner to another location with internet access, do the scanning, submit the images to the server and enjoy the results when you're home.

---
edit:
Concerning chdkptp, the pyptpchdk module is already using the modified libptp2-version from that project. I plan to explore options like remote capture without an SD-Card in the future, but I don't have much experience dealing with C code (and libptp doesn't have the clearest code...), much less with device drivers. Running the chdkptp-tool as an external command could be another option, I'll try to find a way to fall back to that solution in case the Python module is not available.

[spamsickle]

Based on your implied recommendation, I took a look at the Spreads documentation. It looks to me like Spreads is a wrapper for Scan Tailor, with post-Scan Tailor options for creating a variety of ebook formats.

I'm currently using CHDK scripts which fire my cameras in-sync every five seconds, so a setup tethered to my computer which requires me to press a button is not desirable. After I've scanned a book, I'm manually copying the memory chip images to Left and Right folders on my computer; it looks like Spreads could automate this step, if I connect both cameras to the computer with a pair of USB cables or a USB hub.

At the present time, my post-processing consists of using YASW to rotate, de-keystone, and crop the images, then export them to a new set of JPEG images, which I read with a standard image viewer. Since YASW doesn't have a command-line option, I expect all a wrapper could do would be to call it.

I'm convinced that, long-term, ebooks will be stored as marked-up text, images, and optionally (in formats like PDF) some way of specifying the original layout. To go from a physical book to that ebook will require highly accurate OCR. I have Omnipage, Abbyy, I.R.I.S., and Acrobat. I would like to correlate the OCR output from all four, accept the text for which all agree, and present the text alternatives along with the original page image to allow human decision where they differ. Is it conceivable that Spreads would be able to facilitate something like that? I'm saving all my original JPEGs when I scan, for the day when that will become possible. In the meantime, I'm content to read YASW's JPEG output. Disk space is cheap. 64 GB thumb drives are affordable, and can easily hold all the books I'm reading at any given time even when each "ebook" requires several hundred megabytes.

It's not my intention to derail this thread or annoy anyone. Feel free to move or delete this question if it doesn't belong here. I haven't explored Spreads at all, and I realize my own direction hasn't been mainstream for quite a while now.