Daniel Reetz, the founder of the DIY Book Scanner community, has recently started making videos of prototyping and shop tips. If you are tinkering with a book scanner (or any other project) in your home shop, these tips will come in handy. https://www.youtube.com/channel/UCn0gq8 ... g_8K1nfInQ

Dire camera situation 2017

Everything camera related. Includes triggers, batteries, power supplies, flatbeds and sheet-feeding scanners, too.
User avatar
Aegre Reminiscens
Posts: 36
Joined: 04 Mar 2014, 00:53
E-book readers owned: kindle
Number of books owned: 0
Country: Germany
Contact:

Dire camera situation 2017

Post by Aegre Reminiscens » 02 Jan 2017, 11:11

Heyho,

first of all - I wish you a happy new year!

After lots of research, I have to say that the Camera situation seems to be dire.

As it stands, there is not a single camera that is:
a) still being made / available
b) supported by CHDK

The last one that was still available in december but is not anymore (talked to canon support today) was the SX410IS.

Sure, one can buy expensive Canon DSLRs, because for them there is the Canon SDK, and i read in many places that not only do they work nice with gphoto2, but Canon offers an SDK for them as well. Unfortunately, the DSLRs are too expensive for most bookscanner-owners. Also the DSLRs have mechanical parts in them that move for every single photo, which leads me to the conclusion that the total lifetime of such a camera is significantly lower then the lifetime of a cheaper, non-dslr "point'n'shoot" camera.

Here is an overview of the cameras that are supported by gphoto:
http://www.teaser.fr/~hfiguiere/linux/digicam.html
this overview is nice, because it also contains the megapixel of the camera. if you look through the list, you'll find quickly that not a single one in the list (apart from the DSLRs) is still available.

Here is a different overview, of those that are supported by libgphoto2:
http://gphoto.org/proj/libgphoto2/support.php
i searched through there and spent a lot of time searching for cameras that are on that list, but also could not find a single one that is available to be bought.

And here is the overview for CHDK:
http://chdk.wikia.com/wiki/For_Developers
I linked the "for developer" version of the list, as it contains also the cameras that are currently available, for example the SX420IS, but as you can see from the list, for this camera the original firmware hasnt even been aquired, and so no work has started on portint CHDK for it.

What is the impact of this?
First and foremost, our community, the DIY Bookscanner Community has a big problem, if we can not fix this, as all the nice software (spreads, piscan..) requires the camera to be controlled by the computer.

secondly, the NGO Searchwing (https://www.searchwing.org/), which are building drones to save lives in the mediterranean sea is also desperatly trying to find a camera that a linux computer can control, which they want to put on the drone itself.

I guess many other communitys and NGOs will see the impact of this problem.

What can we do?

I think we as a community need to decide which approach we want to support, I see two options:
  • CHDK
  • GPhoto2
When we have made a decision (for example by voting) we need to find a way to support that community, so either the community of GPhoto2 or CHDK, and we need to support the community. We are many and we all need the same thing - Cameras that are available on the market and that can be controlled by linux.

I dont know if monetary support is the right thing, but I do also believe, that if we can settle on a camera (for example the SX420IS or the IXUS285, then we can pay someone to develop an integration into either CHDK or GPhoto2. If everyone of us chips in a small amount (5-50€) or so, then i think we can offer a nice payment for porting a current camera.

The upside of using gphoto2 is that it works with most high-end cameras already nicely. The downside is, that most of the current bookscanner software is developed to use CHDK (spreads, piscan). As the last years have shown, CHDK support for a camera might be available, but only becomes available months or years after that camera is already taken off the market and is not being sold anymore - so we need to ask ourselves wether CHDK is the right horse to bet on for the years to come.

I dont know which community is more active and quicker with integrating new cameras, and I dont know much about the hurdles to switch piscan or spreads to gphoto2, which is why i opened this thread.

Lets have a lively and productive discussion how we as a community can solve this problem.

Cheers
Aegre Reminiscens
All my projects are documented here: http://www.ijon.me

User avatar
Aegre Reminiscens
Posts: 36
Joined: 04 Mar 2014, 00:53
E-book readers owned: kindle
Number of books owned: 0
Country: Germany
Contact:

Re: Dire camera situation 2017

Post by Aegre Reminiscens » 03 Jan 2017, 04:31

I had a conversation on the IRC-Server freenode in the channel #chdk with a user named reyalp, who is very active in the CHDK-Forum, definitly one of the more active users of the CHDK-Community.

Maybe you like to read it.

16:19 < ij0n> heyho, i wish you all a happy new year
16:20 < ij0n> We, the DIY Bookscanner Community have a problem and we need your help. I have outlined the problem in this thread: https://www.diybookscanner.org/forum/vi ... =15&t=3385 and I would love to
read some input from some CHDK-Developers on this problem
16:20 < ij0n> also, the same problem that we have is currently affecting the rescuing of lives in the mediterranean sea by the organisation seawatch, they also need what the diy bookscanner community needs
16:20 < ij0n> I hope to read from you soon!
19:38 * reyalp ret
19:39 < reyalp> hi ij0n
19:44 < reyalp> as far as CHDK goes, porting to recent non-digic 6 cameras should not be terribly difficult for someone with the right skill set
19:58 < reyalp> the "CHDK project" is not organized in a way that supports doing ports for donations, but it's possible you might find an individual developer who would
19:59 < reyalp> I would like to add some more recent cameras, but I don't currently have time
20:06 < reyalp> as far as crowd funding goes, people tend to underestimate what a fair price is
20:07 < reyalp> for someone with decent reverse engineering / C skills but new to CHDK, a port might take 50 hours. That skillset is easily worth $100/hr
23:44 < ij0n> reyalp: yes, i agree on this
23:45 < ij0n> reyalp: but then there are people who are very familiar with CHDK, who could, for example, do the change from the recently finished SX410IS to the SX420IS - it is the direct successor and I just
hope (maybe naive?) that the port cant be that big...
23:45 < ij0n> and maybe that person then only needs a full workday, because that person has done it before..
23:46 < reyalp> it doesn't really make much difference that it's the next model in the series
23:46 < ij0n> how can I find out who did the port for the SX410IS?
23:46 < reyalp> the porting thread in the forum for that camera
23:47 < reyalp> https://chdk.setepontos.com/index.php?topic=12948.0
23:47 < reyalp> looks like it was blackhole and srsa_4c mostly
23:47 < ij0n> reyalp: thx
23:49 < ij0n> reyalp: what do you think about codebounty programs like https://www.bountysource.com/ for example?
23:49 < reyalp> as far as porting effort goes, cameras released at different dates tend to be more different
23:49 < reyalp> no direct experience
23:49 < ij0n> reyalp: how big (roughly..) is the community of people who regularly do CHDK-Ports?
23:50 < reyalp> currently... between 0 and 3 maybe, depending how you define regularly?
23:51 < ij0n> ..."contributed every couple of months a little bit and then maybe a bigger project once in a while or more"
23:51 < reyalp> in general, ports are done by owners of the particular camera who also happen to be programmers
23:52 < reyalp> there was a time when one developer (nafraf) did a bunch of blind ports
23:52 < ij0n> oh becoming an owner of a currently being sold camera is the easy part - if someone would only express the wish to get a camera with the promise of trying to port for it, i am pretty sure that that
person would get that camera pretty quick
23:53 < reyalp> well yeah, but a $100-200 camera isn't really the point
23:53 < ij0n> i mean it would help if that person did a port before, to establish the confidence that the attempt would not be futile, because that person did not have the right skillset
23:53 < ij0n> reyalp: i know that this is not enough or "the point"..which is why i seeded the idea of doing a crowdfunding over there in the other forum
23:54 < reyalp> what I meant is a lot of the ports in the past were done by someone who had a camera, and thought it would be fun to do the port for this thing they had
23:54 < ij0n> ok, i see
23:55 < ij0n> reyalp: so what skillset is needed to do a port? i maybe can try to convince some of my friends to do it
23:55 < reyalp> I think the decline of this is related to the decline of P&S overall, and the fact that other programmable cameras are more common
23:55 < ij0n> i am not a programmer, i am more the mechanical guy...building bookscanners, and floating drones
23:56 < ij0n> reyalp: so the logical step is to switch to android phones as cameras for bookscanners?
23:56 < ij0n> i messed with the thought...
23:57 < reyalp> raspberry pi or something like that
23:57 < ij0n> rpi-cameras dont have the resolution
23:57 < ij0n> one needs 12MP or more for a decent bookscan
23:58 < ij0n> i would love to use 16 or 20, like the SX410IS - or its successor
23:59 < ij0n> reyalp: what programming languanges should a person be able to write to attempt a port?
Day changed to 03 Jan 2017
00:00 < reyalp> they should know C, have some knowledge of assembler, and some experience with disassembly
00:00 < ij0n> oh.
00:01 < reyalp> don't specifically have to know ARM assembler, but if not they will have to learn a bit
00:01 < ij0n> reyalp: the lua part is just for scripting on a running chdk, but has nothing to do with actually porting it?
00:01 < reyalp> correct
00:01 < ij0n> damn
00:02 < ij0n> reyalp: see, i know next to nothing about the underlying software, but i am a very happy user
00:02 < reyalp> people who didn't know much C/ASM have made working ports, but it's a lot harder
00:04 < ij0n> ok, but that kind of attempt takes a lot longer then if someone does it who already knows C and ASM and has worked on CHDK before
00:04 < reyalp> yes, and if a new model does something that needs real reverse engineering, they will probably be stuck
00:04 < ij0n> hm, i dont know, in the end it doesnt really matter what skillset a person has, as long as the goal of working ports for currently available cameras is somehow reached
00:05 < reyalp> there was a time when we had tools that would automate a lot of the steps, and Canon didn't change their firmware for a few generations
00:05 < reyalp> so ports were relatively easy
00:06 < reyalp> but canons stuff changes over time, so when the old technique stop workings and someone with a lot more knowledge is needed to fix it
00:06 < ij0n> is canon now changing the firmware more often, or did they change it once or twice and the tools just werent pushed past that step?
00:09 < reyalp> a combination of things I guess
00:12 < reyalp> for some time, the things that canon changed between releases didn't happen to affect chdk very much
00:47 < ij0n> reyalp: thx for your input, i'll go to sleep now
09:07 * reyalp zzz


On different news, Canon called me back and confirmed that they removed PTP-Support on all the small Cameras, only the EOS-Devices keep PTP-Support via USB. This also means that the decision between CHDK/GPhoto2 that I oulined in the other post has been made for us. CHDK it is, wether we like it or not, because GPhoto relies on a working PTP in the camera firmware, which canon does only ship with eos-kameras.
All my projects are documented here: http://www.ijon.me

spamsickle
Posts: 596
Joined: 06 Jun 2009, 23:57

Re: Dire camera situation 2017

Post by spamsickle » 03 Jan 2017, 19:09

I don't think 12 MP is required. I use 5 MP cameras I buy on eBay for under $100. Unless you're scanning large-format books, or books with really tiny glyphs, that's adequate, IMO.

rkomar
Posts: 86
Joined: 12 May 2013, 16:36
E-book readers owned: PRS-505, PocketBook 902, PRS-T1, PocketBook 623, PocketBook 840
Number of books owned: 3000
Country: Canada

Re: Dire camera situation 2017

Post by rkomar » 03 Jan 2017, 21:59

I haven't bought a camera for a few years, so I'm out of date. What are the Canon cameras using if PTP is disabled? WiFi?

User avatar
Aegre Reminiscens
Posts: 36
Joined: 04 Mar 2014, 00:53
E-book readers owned: kindle
Number of books owned: 0
Country: Germany
Contact:

Re: Dire camera situation 2017

Post by Aegre Reminiscens » 04 Jan 2017, 10:39

spamsickle wrote:I don't think 12 MP is required. I use 5 MP cameras I buy on eBay for under $100. Unless you're scanning large-format books, or books with really tiny glyphs, that's adequate, IMO.
If 5MP is enough for your purposes, then great! I think it helps to have a finer resolution, especially for deskewing-algorithms. also if one has a better resolution then one does not need to adjust the zoom on the camera for every book, but can go with a "one-fits-all-maximum area" setting.

The central problem here is still valid though - the further we move away (in released camera models) from chdk-supported cameras, the less cameras will be available for us on the market of used cameras. the DIY-person needs to buy two of the same camera, which might still be possible a year from now, if we are talking about used cameras on ebay. Everyone that wants to create Kits and sell them has a different problem, as that person has to buy more then two at the same time and most buyers of a kit would not accept to get a used camera in an unknown state in a nice kit.

I dont think i need to elaborate how important it is that we as a community have both - the opportunity to buy kits and the opportunity for people to DIY whatever they want with our help.
rkomar wrote:I haven't bought a camera for a few years, so I'm out of date. What are the Canon cameras using if PTP is disabled? WiFi?
They use the USB as a mass storage device, so people can still download images, but they do not ship any PTP anymore, so if you want to remote shoot a picture (from a linux system for example) you a forced to buy one of the more expensive EOS-DSLRs.
All my projects are documented here: http://www.ijon.me

duerig
Posts: 371
Joined: 01 Jun 2014, 17:04
Number of books owned: 1000
Country: United States of America

Re: Dire camera situation 2017

Post by duerig » 04 Jan 2017, 11:29

This has been a growing problem. For smaller books, a Pi camera (at 8MP) is likely to be sufficient. So I have been working on adding support for them to Pi Scan. Similarly, for those who want a high quality camera, the DSLRs and mirrorless cameras that libgphoto supports are the way to go and so I've started adding support for libgphoto to Pi Scan.

Alas, this still leaves the problem in the middle range. For the moment, I have enough ELPH 160's on hand to continue providing them in the Archivist Quill kits. But they are no longer being sold by Canon. Unfortunately, it is unclear if there is a suitable Canon replacement camera even if we do port CHDK. The ELPH 180 which is its successor is currently listed as on 'backorder' at Canon which is a worrying sign. And the SX* cameras are twice as expensive or more.

But this is also the time of year when Canon has historically released its new camera models. Hopefully we will be able to see in a few days what they are and coordinate on which one might be the most useful as a replacement. Then work on getting CHDK ported to it over the next few months.

-Jonathon Duerig

dpc
Posts: 298
Joined: 01 Apr 2011, 18:05
Number of books owned: 0
Location: Issaquah, WA

Re: Dire camera situation 2017

Post by dpc » 05 Jan 2017, 16:45

I'd be happy when Canon releases a mirrorless EOS camera that uses the EF lenses. There are ways to lock the mirror up on the EOS cameras but I've yet to figure out how to do this with their SDK. Doing that would help with the wear factor when scanning thousands of pages. Commercial scanners use DSLRs though so they have either found a way to lock the mirror up or it's not the troubling issue that we think it is.

In continuing to attempt to support the lower-end cameras you may be fighting a losing battle. The reverse-engineering effort required to continually support future Canon Point-and-Shoot camera models could be more work than people are willing to do and there are some legal gray areas here of reverse-engineering copyrighted code owned by Canon which may somewhat affect your success at hiring someone to do this work. Finally, if Canon wanted to prevent you from being able to load your modified firmware on the device, they could easily do this. What's your plan going to be if that happens?

Commercial scanners that rely on DSLRs such as ATIZ BookDrive use the Canon and Nikon SDKs which at least offer a support path going forward and don't require modifying the camera's firmware (at least the Canon SDK doesn't). It's the reason I bit the bullet and broke out my wallet to buy the more expensive cameras. I figured that those commercial scanner manufacturers have smarter people than me designing their book scanners and they aren't messing around trying to support the less expensive Point-and-Shoot cameras they likey did that for a good reason.

User avatar
Aegre Reminiscens
Posts: 36
Joined: 04 Mar 2014, 00:53
E-book readers owned: kindle
Number of books owned: 0
Country: Germany
Contact:

Re: Dire camera situation 2017

Post by Aegre Reminiscens » 08 Jan 2017, 08:08

Thanks for all your opinions regarding this topic.

Please tell me - how much would YOU be willing to donate towards a current CHDK-Port? We need to buy at least a camera for the volunteer who wants to do it, but we might want to give a bit more then that.

If you are uncomfortable saying it out loud here, please DM me.
All my projects are documented here: http://www.ijon.me

User avatar
Aegre Reminiscens
Posts: 36
Joined: 04 Mar 2014, 00:53
E-book readers owned: kindle
Number of books owned: 0
Country: Germany
Contact:

Re: Dire camera situation 2017

Post by Aegre Reminiscens » 10 Jan 2017, 08:09

Dear all,

yesterday I went to a meeting from the organisation searchwing, which is building a very simple fixed wing drone to capture images of radar targets. The idea here is that the multiple NGOs that are currently rescuing lives of refugees in the mediteranean sea want to verify quicker wether a boat full of refugees needs help or not.

They have the exact same problem then we do, they also need a working CHDK-Port. Luckily, the organisation searchwing is mostly comprised of european hackers, most of them have a strong connection to the CCC. ( https://en.wikipedia.org/wiki/Chaos_Computer_Club in case you haven't heard of them)

Obviously, the CCC does not like it very much when requests from outside to do or hack something are brought in, as that happends waaaay too often. In this case, the situation is different, because searchwing has been founded (member wise) very close to the core of the CCC.

Searchwing yesterday acknowledged the need and they will help to push this demand "please port this or that camera" in the right circles of CCC - i have very high hopes that this will lead to a timely port of a current version of CHDK for this camera model.

Lets see if this works...

Cheers
All my projects are documented here: http://www.ijon.me

murgen
Posts: 16
Joined: 22 Sep 2012, 03:45
E-book readers owned: Kindle
Number of books owned: 1000
Country: Belgium

Re: Dire camera situation 2017

Post by murgen » 20 Jan 2017, 09:42

I am browsing aliexpress.com for the camera and contacting chinese producer in order to locate which product could do the following 3 features:

- remote configure (to set foccus)
- video flow (for online monitors)
- remote shooting

Stuff like that is maybe all we need : http://fr.aliexpress.com/item/AMKOV-AMK ... 67009f51b5

IP OEM camera with high resolution : at 5mp I have my doubts, but they are progressing:
https://fr.aliexpress.com/item/8MP-3264 ... 0ec8099d2a


There are quite a few chear 'DSLR' but it is hard to figure out if they have a SDK and if yes which one:
https://fr.aliexpress.com/item/Compact- ... 51c8&tpp=1

Post Reply