Daniel Reetz, the founder of the DIY Book Scanner community, has recently started making videos of prototyping and shop tips. If you are tinkering with a book scanner (or any other project) in your home shop, these tips will come in handy. https://www.youtube.com/channel/UCn0gq8 ... g_8K1nfInQ

Microsoft Kinect: infrared depth maps for dewarping?

DIY Book Scanner Skunk Works. Share your crazy ideas and novel approaches. Home of the "3D structure of a book" thread.
User avatar
daniel_reetz
Posts: 2786
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by daniel_reetz » 21 Nov 2010, 12:43

Yeah, that TIFF looks perfect! I just loaded it in a viewer. Obviously, I'm not seeing all the depth info, because my display is only 8 bit, but that's the kind of data we need to be feeding to dewarping software.

What was your procedure for saving it out? (rough idea is fine; I'm a total n00b with programming, I think I can do it with Linux using a small mod to the existing code)

If you get a chance, throw some books under that thing! I'm really excited to graph the output and see how much better it looks than my first test.

vitorio
Posts: 138
Joined: 30 Oct 2010, 23:56
Number of books owned: 0
Location: Austin, Texas, USA
Contact:

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by vitorio » 21 Nov 2010, 19:34

Okay, I tidied up the code.

The Windows libfreenect work isn't there yet, and Zephod's drivers and C++ sample code aren't really made to be expanded upon (unless you like programming in C++), so I installed the closed-source Code Laboratories NUI Platform drivers and used their DLL:

http://codelaboratories.com/nui/

I used stock Python 2.5 on Windows and copied the CLNUIDevice.dll into a directory alongside the following Python script:

http://python.pastebin.com/ZQw6ttzZ

That will give you a PNG from the RGB camera and what I think is a 16-bit greyscale TIFF from the depth camera. I'll post more samples after I test the 16-bit output some more. If you're on Linux or OS X, it should be straightforward to rewrite the script against the libfreenect Python bindings that they include with the package.

vitorio
Posts: 138
Joined: 30 Oct 2010, 23:56
Number of books owned: 0
Location: Austin, Texas, USA
Contact:

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by vitorio » 06 Dec 2010, 00:32

The minimum focal distance of the Kinect is around half a meter or so, and it occurred to me that perhaps I could use e.g. reading glasses to correct that, but actually trying it resulted in no depth image at all. From some discussion in #openkinect:
(08:59:33 PM) marcan: not going to work
(08:59:44 PM) Vito`: why's that
(08:59:48 PM) marcan: the depth cam uses very accurate factory calibration of the depth pattern
(08:59:57 PM) marcan: *any* distortion or shift and it just stops working
(09:00:06 PM) brandyn: Vito`, maybe figure out how to flash the firmware?
(09:00:16 PM) Vito`: brandyn: with what?
(09:00:18 PM) brandyn: Vito`, they use a test shot I think, so you could possibly do the same
(09:00:35 PM) brandyn: No idea, just saying if they store an image inside to calibrate it
(09:00:40 PM) marcan: also, the focal distance issue isn't actually about focus
(09:00:42 PM) brandyn: then you can possibly figure out how to do the same
(09:00:52 PM) marcan: the IR projection pattern consists of roughly collimated laser beams
(09:01:01 PM) marcan: at close distance, those beams converge into one big blob
(09:01:20 PM) marcan: it doesn't matter what you do to the camera, the information just isn't there when you're too close
(09:01:58 PM) marcan: you could try altering the focus of the projector assembly into something non-collimated (you'd probably have to reduce brightness too), but that would involve modifying that bit of optics
(09:02:08 PM) marcan: and you'd still have to recalibrate everything, which we don't know how to do
(09:02:12 PM) brandyn: he was adding glasses to it
(09:02:28 PM) brandyn: he got some new eyeware for his kinect
(09:12:19 PM) Vito`: marcan: through lenses, the projection and the view of the projection should be consistent with the stored state, except closer, though, shouldn't they?
(09:12:31 PM) Vito`: marcan: I'm not sure I buy that it's not possible without recalibration
(09:13:15 PM) Vito`: but yes, I am currently getting no or screwy depth information when I put glasses in front of the projector and camera
(09:14:38 PM) marcan: iff you can somehow manage to make the camera see *precisely* the same constellation pattern (same exact size and shape and alignment) then yes, it should work. However, that also won't let you see closer
(09:14:42 PM) marcan: you need to solve three things at once
(09:15:08 PM) marcan: 1) maintain the precise pattern, in rotation and shape and alignment, but
(09:15:16 PM) marcan: 2) shift the pattern horizontally to shift the depth range
(09:15:45 PM) marcan: 3) while solving the "blob" issue of the constellation converging into nothing at close range, i.e. refocusing the beams (glasses probably won't do a good enough job here)
(09:15:58 PM) marcan: oh, and 4) reducing power and 5) making sure the camera is in focus too
(09:16:14 PM) marcan: it might be physically possible but I don't think it's practical with a set of glasses :P
(09:16:58 PM) Vito`: why would 1 change and why would 2 need to happen?
(09:17:23 PM) marcan: any lense is going to cause distortion
(09:17:26 PM) marcan: *lens
(09:17:52 PM) marcan: and the kinect isn't going to return depth data that close, you need to trick it into thinking near isn't that near
(09:17:58 PM) marcan: i.e. shift the depth response
(09:18:42 PM) brandyn: Vito`, you are at a university right?
(09:19:15 PM) brandyn: contact someone in optics and see if they want to help
(09:19:35 PM) CuBeD_: Brandyn What is the demo I need to run?
(09:19:36 PM) marcan: personally I think it's completely impractical without recalibration
(09:22:13 PM) amiller: i think it would work even without recalibration
(09:22:20 PM) amiller: you'd end up with a projective distortion of your 3d image
(09:23:02 PM) amiller: i see your point about it being too near, but mirrors might fix that too
The Windows drivers have a libfreenect-compatible wrapper now, so I'll update my Python script so you can run it on your Mac/Linux setups, bring my level in and take some shots of books from 18"+ away.

spamsickle
Posts: 596
Joined: 06 Jun 2009, 23:57

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by spamsickle » 06 Dec 2010, 04:40

I'm not sure I buy the "any distortion and it just stops working" claim. Conceptually, the device is ALWAYS seeing distortion -- that's how it's determining depth, right? Yes, a lens is probably moving dots independently of depth, but I'd still expect you to get information back. You might not be able to depend on the accuracy of that information if the lens is distorting too much, but if you're getting nothing I'd be more inclined to suspect that the lens is reflecting or attenuating the IR light than that "lens distortion" is blowing its mind.

We don't really need true depth, just relative depth. For our purposes, it doesn't matter if the device reports that the page is 18" away or 3 feet, as long as it has enough precision to give accurate depth separation for the various parts of the page.

User avatar
daniel_reetz
Posts: 2786
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by daniel_reetz » 03 Jan 2011, 11:29

Looks like the PrimeSensor (kinect platform) will be spreading to many different products... maybe we can work with them to get a close-range unit going.

http://image-sensors-world.blogspot.com ... based.html

Vitorio, I'm really glad you checked things out with Marcan, I don't agree with all the conclusions but it's definitely a tougher problem than it looked at first. I just got a textbook on speckle and I'm learning everything I can about shifting the pattern for close range use.

vitorio
Posts: 138
Joined: 30 Oct 2010, 23:56
Number of books owned: 0
Location: Austin, Texas, USA
Contact:

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by vitorio » 04 Jan 2011, 13:05

Primesense recently opened up their whole SDK and are offering development kits for sale, so it might be worth it to ask them.

Alternately, I wonder if all of that structured light is really necessary for book dewarping. Could you get away with, say, three laser levels across the book (top 1/3, middle 1/3, bottom 1/3)? Would that provide enough structured light that you could photograph that, and then photograph the book, and dewarp from that?

User avatar
daniel_reetz
Posts: 2786
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by daniel_reetz » 04 Jan 2011, 13:08

It's possible. I spent much of the day yesterday working on a post about different ways we could extract 3D data from books. I'll get it done soon and post it here.

My primary objection to most structured light stuff is the number of shots necessary. If we could get a 1-3 shot system going, that would likely be worthwhile.

User avatar
daniel_reetz
Posts: 2786
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by daniel_reetz » 11 Jan 2011, 23:53

Here's another PrimeSensor in the wild... now if only they'd make a close-up version, we'd be in business...

http://www.engadget.com/2011/01/11/asus ... ed-at-ces/

User avatar
daniel_reetz
Posts: 2786
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by daniel_reetz » 06 Jun 2011, 19:30

Holy crap! There's a set of "zoom lenses" for the Kinect that supposedly cause it to operate at close range - I *really* want to know if this works!

http://gizmodo.com/5809139/nyko-zoom-fo ... d-quarters

We need one of these!
Attachments
zoomkinect.jpg
zoomkinect.jpg (16.46 KiB) Viewed 7043 times

Nexxxeh

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by Nexxxeh » 25 Sep 2011, 08:18

Has anyone tried this with the Nyko Zoom Lenses? The Nyko website suggests they are readily available in the US and I can import them via eBay into the UK.
PC Mag review suggests a less than 40% improvement but I've no idea how useful that would be for mapping book surface. At the very least, presumably that means the concept is sound. Anyone good with building custom optics?

Post Reply