Daniel Reetz, the founder of the DIY Book Scanner community, has recently started making videos of prototyping and shop tips. If you are tinkering with a book scanner (or any other project) in your home shop, these tips will come in handy. https://www.youtube.com/channel/UCn0gq8 ... g_8K1nfInQ

Microsoft Kinect: infrared depth maps for dewarping?

DIY Book Scanner Skunk Works. Share your crazy ideas and novel approaches. Home of the "3D structure of a book" thread.
User avatar
rob
Posts: 773
Joined: 03 Jun 2009, 13:50
E-book readers owned: iRex iLiad, Kindle 2
Number of books owned: 4000
Country: United States
Location: Maryland, United States
Contact:

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by rob » 07 Nov 2010, 12:13

univurshul wrote:Where are you guys getting your laser cutters?
I use the one at my local makerspace.
The Singularity is Near. ~ http://halfbakedmaker.org ~ Follow me as I build the world's first all-mechanical steam-powered computer.

User avatar
daniel_reetz
Posts: 2785
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by daniel_reetz » 10 Nov 2010, 14:18

Well, this turned out to be an interesting story. The guy that made the high quality hacked driver for the PS3 cam wrote code to control it right away, but didn't release it. Instead, he made a fund asking for $10,000 (including the Adafruit money), and when that was hit, he would release code.

http://www.engadget.com/2010/11/07/kine ... ame-video/

Unfortunately for AlexP, the open source approach steamrolled him. Within a few hours of the european launch of Kinect, hacker Hector Martin released working code for Linux.

http://www.engadget.com/2010/11/10/open ... -download/

I hope Adafruit offers Hector the prize. I just ordered a Kinect for myself. Looking forward to making this my first serious coding project.

vitorio
Posts: 138
Joined: 30 Oct 2010, 23:56
Number of books owned: 0
Location: Austin, Texas, USA
Contact:

Structured light for dewarping

Post by vitorio » 10 Nov 2010, 15:22

Both the Kinect and the page-flipping demo use the same techniques for figuring out depth and alignment: structured light.

http://en.wikipedia.org/wiki/Structured_light

The links in the Wikipedia article include other references, such as using it to automatically align projectors (where I first heard about it), and algorithms and software. The Makerbot 3D scanner has its own software that performs the same functions.

The IEEE Spectrum article about the book flipping scanner says that it uses a custom camera that does 1280x1024 at 500fps, but I'm not sure you really need a) that high a framerate or b) a custom camera. Elphel makes ~$1k video cameras that can do at least 640x480 at 100fps, but maybe 1024x768 at 44fps is enough, or something in between. For lower-resolution video, you could theoretically use superresolution techniques to interpolate multiple video frames together into a higher-resolution or more detailed image.

"Just" adding macro lenses to the Kinect hardware to refocus the IR light on a page or across the entire platen might give you enough resolution to dewarp a book, even if you're not using the RGB camera to capture the images, and instead just turning pages and photographing one or both pages with a still camera. Reconstructing two pages from a single photograph of a "spread" alleviates a lot of the issues with dual cameras if you don't mind the resolution loss and it doesn't affect OCR.

Given that there are at least two pieces of software for reference out there for doing this, I'd guess it's probably doable by a software developer with a few weeks of free time, regardless of whether you're using Kinect or making it yourself with a projector like the Makerbot 3D scanner.

User avatar
rob
Posts: 773
Joined: 03 Jun 2009, 13:50
E-book readers owned: iRex iLiad, Kindle 2
Number of books owned: 4000
Country: United States
Location: Maryland, United States
Contact:

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by rob » 10 Nov 2010, 23:41

daniel_reetz wrote:...Within a few hours of the european launch of Kinect, hacker Hector Martin released working code for Linux.
Ha ha HA! Hector indeed got the $3k bounty. I doubt AlexP even had a working driver.

However, the depth output seems rather coarse...
The Singularity is Near. ~ http://halfbakedmaker.org ~ Follow me as I build the world's first all-mechanical steam-powered computer.

User avatar
daniel_reetz
Posts: 2785
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by daniel_reetz » 10 Nov 2010, 23:45

I know... so great.

Staring at my Kinect now... I'd start hacking on it, but I'm working on another book scanner design. :)

univurshul
Posts: 496
Joined: 04 Mar 2014, 00:53

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by univurshul » 11 Nov 2010, 00:58

I'm buying my girlfriend a Kinect for Christmas. hehehe...

User avatar
daniel_reetz
Posts: 2785
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by daniel_reetz » 11 Nov 2010, 01:15

Let me guess... she doesn't have an XBOX? Perfect!

User avatar
daniel_reetz
Posts: 2785
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by daniel_reetz » 12 Nov 2010, 00:52

I just had a meetup with Rob and Spamsickle, and Spamsickle said there was a video of the Kinect infrared lighting... I hadn't seen it, but I went looking and found this: http://www.youtube.com/watch?v=nvvQJxgykcU

Pretty interesting stuff...

User avatar
daniel_reetz
Posts: 2785
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by daniel_reetz » 12 Nov 2010, 02:25

Alright, so I am now working to get the Kinect up and running. Warning, this is going to be super-verbose and mostly notes to myself.

First things first: Need to get the code onto my machine. I'm using Ubuntu Linux for my starting point.

The code is stored on a Git server. Git is a version control system for code. I have never used it before. Using this tutorial as a starting point, I entered the following magic incantation:

Code: Select all

fake@polar:~$  git clone git://git.marcansoft.com/libfreenect.git
Which got me the following error message/solution:

Code: Select all

The program 'git' is currently not installed.  You can install it by typing:
sudo apt-get install git-core
fake@polar:~$ sudo apt-get install git-core
I installed Git and tried the command again.

Code: Select all

fake@polar:~$  git clone git://git.marcansoft.com/libfreenect.git
Initialized empty Git repository in /home/fake/libfreenect/.git/
remote: Counting objects: 74, done.
remote: Compressing objects: 100% (71/71), done.
remote: Total 74 (delta 37), reused 0 (delta 0)
Receiving objects: 100% (74/74), 17.44 KiB, done.
Resolving deltas: 100% (37/37), done.
fake@polar:~$ 
Now it's time to build this code. But damn, I don't have the right tools. Just trying make or gcc fails badly.Lookslike people are building this code with cmake. So, to install cmake:

Code: Select all

sudo apt-get install cmake
Then I made a directory called "build"

Code: Select all

fake@polar:~$ cd libfreenect
fake@polar:~/libfreenect$ mkdir build
fake@polar:~/libfreenect$ ls
build  CMakeLists.txt  examples  include  lib  Modules  README.txt
fake@polar:~/libfreenect$ cd build
fake@polar:~/libfreenect/build$ cmake
Ugh, so all that and there's apparently no cmake file, blah. I am going to clone the OpenKinect git repository, instead, and start with that -- it looks to be where the action is happening.

Code: Select all

fake@polar:~$ git clone https://github.com/qdot/OpenKinect.git
Initialized empty Git repository in /home/fake/OpenKinect/.git/
remote: Counting objects: 182, done.
remote: Compressing objects: 100% (155/155), done.
remote: Total 182 (delta 77), reused 54 (delta 15)
Receiving objects: 100% (182/182), 352.37 KiB | 180 KiB/s, done.
Resolving deltas: 100% (77/77), done.
fake@polar:~$ ls
fake@polar:~$ cd OpenKinect
fake@polar:~/OpenKinect$
hmm, more trouble in paradise, blarg:

Code: Select all

fake@polar:~/OpenKinect/c$ cd build
fake@polar:~/OpenKinect/c/build$ cmake ..
-- The C compiler identification is GNU
-- The CXX compiler identification is unknown
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
CMake Error: your CXX compiler: "CMAKE_CXX_COMPILER-NOTFOUND" was not found.   Please set CMAKE_CXX_COMPILER to a valid compiler path or name.
-- Looking for include files CMAKE_HAVE_PTHREAD_H
-- Looking for include files CMAKE_HAVE_PTHREAD_H - found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- checking for module 'libusb-1.0>=1.0.3'
--   found libusb-1.0, version 1.0.6
-- Found USB 
-- Found USB 
-- Configuring incomplete, errors occurred!
fake@polar:~/OpenKinect/c/build$ ls
CMakeCache.txt  CMakeFiles  examples  lib
fake@polar:~/OpenKinect/c/build$
Huh, I guess that means that GCC is not installed. Well, that's not so bad. More installing. This time the necessary package is called "build-essential"

Code: Select all

fake@polar:~/OpenKinect/c/build$ sudo apt-get install build-essential
Installed fine. Now to try that command again...

Code: Select all

fake@polar:~/OpenKinect/c/build$ cmake ..
-- The CXX compiler identification is GNU
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found USB 
-- Found USB 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/fake/OpenKinect/c/build
fake@polar:~/OpenKinect/c/build$ 
WHEW, that was RIDICULOUS. Now I have a makefile?? So I need to run MAKE:

Code: Select all

fake@polar:~/OpenKinect/c/build$ make
[ 25%] Generating inits.c
Scanning dependencies of target freenect
[ 50%] Building C object lib/CMakeFiles/freenect.dir/cameras.c.o
[ 75%] Building C object lib/CMakeFiles/freenect.dir/inits.c.o
Linking C static library libfreenect.a
[ 75%] Built target freenect
Scanning dependencies of target glview
[100%] Building C object examples/CMakeFiles/glview.dir/glview.c.o
/home/fake/OpenKinect/c/examples/glview.c:24:21: error: GL/glut.h: No such file or directory
/home/fake/OpenKinect/c/examples/glview.c:25:19: error: GL/gl.h: No such file or directory
/home/fake/OpenKinect/c/examples/glview.c:26:20: error: GL/glu.h: No such file or directory
/home/fake/OpenKinect/c/examples/glview.c:47: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘gl_depth_tex’
/home/fake/OpenKinect/c/examples/glview.c:48: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘gl_rgb_tex’
/home/fake/OpenKinect/c/examples/glview.c: In function ‘DrawGLScene’:
/home/fake/OpenKinect/c/examples/glview.c:68: warning: implicit declaration of function ‘glClear’
/home/fake/OpenKinect/c/examples/glview.c:68: error: ‘GL_COLOR_BUFFER_BIT’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:68: error: (Each undeclared identifier is reported only once
/home/fake/OpenKinect/c/examples/glview.c:68: error: for each function it appears in.)
/home/fake/OpenKinect/c/examples/glview.c:68: error: ‘GL_DEPTH_BUFFER_BIT’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:69: warning: implicit declaration of function ‘glLoadIdentity’
/home/fake/OpenKinect/c/examples/glview.c:71: warning: implicit declaration of function ‘glEnable’
/home/fake/OpenKinect/c/examples/glview.c:71: error: ‘GL_TEXTURE_2D’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:73: warning: implicit declaration of function ‘glBindTexture’
/home/fake/OpenKinect/c/examples/glview.c:73: error: ‘gl_depth_tex’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:74: warning: implicit declaration of function ‘glTexImage2D’
/home/fake/OpenKinect/c/examples/glview.c:74: error: ‘GL_RGB’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:74: error: ‘GL_UNSIGNED_BYTE’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:76: warning: implicit declaration of function ‘glBegin’
/home/fake/OpenKinect/c/examples/glview.c:76: error: ‘GL_TRIANGLE_FAN’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:77: warning: implicit declaration of function ‘glColor4f’
/home/fake/OpenKinect/c/examples/glview.c:78: warning: implicit declaration of function ‘glTexCoord2f’
/home/fake/OpenKinect/c/examples/glview.c:78: warning: implicit declaration of function ‘glVertex3f’
/home/fake/OpenKinect/c/examples/glview.c:82: warning: implicit declaration of function ‘glEnd’
/home/fake/OpenKinect/c/examples/glview.c:84: error: ‘gl_rgb_tex’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:95: warning: implicit declaration of function ‘glutSwapBuffers’
/home/fake/OpenKinect/c/examples/glview.c:56: warning: unused variable ‘fcnt’
/home/fake/OpenKinect/c/examples/glview.c: In function ‘keyPressed’:
/home/fake/OpenKinect/c/examples/glview.c:102: warning: implicit declaration of function ‘glutDestroyWindow’
/home/fake/OpenKinect/c/examples/glview.c: In function ‘ReSizeGLScene’:
/home/fake/OpenKinect/c/examples/glview.c:109: warning: implicit declaration of function ‘glViewport’
/home/fake/OpenKinect/c/examples/glview.c:110: warning: implicit declaration of function ‘glMatrixMode’
/home/fake/OpenKinect/c/examples/glview.c:110: error: ‘GL_PROJECTION’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:112: warning: implicit declaration of function ‘glOrtho’
/home/fake/OpenKinect/c/examples/glview.c:113: error: ‘GL_MODELVIEW’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c: In function ‘InitGL’:
/home/fake/OpenKinect/c/examples/glview.c:118: warning: implicit declaration of function ‘glClearColor’
/home/fake/OpenKinect/c/examples/glview.c:119: warning: implicit declaration of function ‘glClearDepth’
/home/fake/OpenKinect/c/examples/glview.c:120: warning: implicit declaration of function ‘glDepthFunc’
/home/fake/OpenKinect/c/examples/glview.c:120: error: ‘GL_LESS’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:121: warning: implicit declaration of function ‘glDisable’
/home/fake/OpenKinect/c/examples/glview.c:121: error: ‘GL_DEPTH_TEST’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:122: error: ‘GL_BLEND’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:123: warning: implicit declaration of function ‘glBlendFunc’
/home/fake/OpenKinect/c/examples/glview.c:123: error: ‘GL_SRC_ALPHA’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:123: error: ‘GL_ONE_MINUS_SRC_ALPHA’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:124: warning: implicit declaration of function ‘glShadeModel’
/home/fake/OpenKinect/c/examples/glview.c:124: error: ‘GL_SMOOTH’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:125: warning: implicit declaration of function ‘glGenTextures’
/home/fake/OpenKinect/c/examples/glview.c:125: error: ‘gl_depth_tex’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:126: error: ‘GL_TEXTURE_2D’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:127: warning: implicit declaration of function ‘glTexParameteri’
/home/fake/OpenKinect/c/examples/glview.c:127: error: ‘GL_TEXTURE_MIN_FILTER’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:127: error: ‘GL_LINEAR’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:128: error: ‘GL_TEXTURE_MAG_FILTER’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:129: error: ‘gl_rgb_tex’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c: In function ‘gl_threadfunc’:
/home/fake/OpenKinect/c/examples/glview.c:140: warning: implicit declaration of function ‘glutInit’
/home/fake/OpenKinect/c/examples/glview.c:142: warning: implicit declaration of function ‘glutInitDisplayMode’
/home/fake/OpenKinect/c/examples/glview.c:142: error: ‘GLUT_RGBA’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:142: error: ‘GLUT_DOUBLE’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:142: error: ‘GLUT_ALPHA’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:142: error: ‘GLUT_DEPTH’ undeclared (first use in this function)
/home/fake/OpenKinect/c/examples/glview.c:143: warning: implicit declaration of function ‘glutInitWindowSize’
/home/fake/OpenKinect/c/examples/glview.c:144: warning: implicit declaration of function ‘glutInitWindowPosition’
/home/fake/OpenKinect/c/examples/glview.c:146: warning: implicit declaration of function ‘glutCreateWindow’
/home/fake/OpenKinect/c/examples/glview.c:148: warning: implicit declaration of function ‘glutDisplayFunc’
/home/fake/OpenKinect/c/examples/glview.c:149: warning: implicit declaration of function ‘glutIdleFunc’
/home/fake/OpenKinect/c/examples/glview.c:150: warning: implicit declaration of function ‘glutReshapeFunc’
/home/fake/OpenKinect/c/examples/glview.c:151: warning: implicit declaration of function ‘glutKeyboardFunc’
/home/fake/OpenKinect/c/examples/glview.c:155: warning: implicit declaration of function ‘glutMainLoop’
/home/fake/OpenKinect/c/examples/glview.c: In function ‘rgbimg’:
/home/fake/OpenKinect/c/examples/glview.c:221: warning: unused variable ‘i’
make[2]: *** [examples/CMakeFiles/glview.dir/glview.c.o] Error 1
make[1]: *** [examples/CMakeFiles/glview.dir/all] Error 2
make: *** [all] Error 2
fake@polar:~/OpenKinect/c/build$ 
Alright, well, something bad happened, it had something to do with GLUT. I'm going to try installing that and then I am going to quit for the night, I need to be well rested tomorrow.

Code: Select all

/home/fake/OpenKinect/c/examples/glview.c:24:21: error: GL/glut.h: No such file or directory
BAGH. No. Stupid linux.

I went and installed everything related to FreeGLUT and GLUTG3. Result:

Code: Select all

fake@polar:~/OpenKinect/c/build$ make
[ 75%] Built target freenect
[100%] Building C object examples/CMakeFiles/glview.dir/glview.c.o
/home/fake/OpenKinect/c/examples/glview.c: In function ‘DrawGLScene’:
/home/fake/OpenKinect/c/examples/glview.c:56: warning: unused variable ‘fcnt’
/home/fake/OpenKinect/c/examples/glview.c: In function ‘rgbimg’:
/home/fake/OpenKinect/c/examples/glview.c:221: warning: unused variable ‘i’
Linking C executable glview
[100%] Built target glview
I think that means it built. Gonna try to run it now...

User avatar
daniel_reetz
Posts: 2785
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Microsoft Kinect: infrared depth maps for dewarping?

Post by daniel_reetz » 12 Nov 2010, 02:41

Moar learning.

So, the make process built a file called "glview". That was the sum of all that effort. But you can't just stupidly type "glview" and expect it to work:

Code: Select all

fake@polar:~/OpenKinect/c/build/examples$ glview
No command 'glview' found, did you mean:
 Command 'vlview' from package 'lipsia' (universe)
 Command 'mlview' from package 'mlview' (universe)
 Command 'mglview' from package 'mathgl' (universe)
 Command 'gview' from package 'vim-gnome' (main)
 Command 'gview' from package 'vim' (main)
 Command 'gview' from package 'vim-nox' (universe)
 Command 'gview' from package 'vim-gtk' (universe)
 Command 'gview' from package 'vim-tiny' (main)
 Command 'gqview' from package 'gqview' (universe)
glview: command not found
fake@polar:~/OpenKinect/c/build/examples$ ^C
fake@polar:~/OpenKinect/c/build/examples$ 
Nope, to run an executable you have to type "./" before the command. So...

Code: Select all

fake@polar:~/OpenKinect/c/build/examples$  ./glview
Kinect camera test
libusb couldn't open USB device /dev/bus/usb/002/019: Permission denied.
libusb requires write access to USB device nodes.
Could not open device
fake@polar:~/OpenKinect/c/build/examples$ 
AGGHHH. It needs to run as root.

Code: Select all

fake@polar:~/OpenKinect/c/build/examples$ sudo ./glview
Success!
Screenshot.png
Some bad news, I tried a few normal lenses on the IR projector and camera, and in both cases it immediately stopped working. Now I need to look into IR lenses...

Post Reply