Daniel Reetz, the founder of the DIY Book Scanner community, has recently started making videos of prototyping and shop tips. If you are tinkering with a book scanner (or any other project) in your home shop, these tips will come in handy. https://www.youtube.com/channel/UCn0gq8 ... g_8K1nfInQ

Adjust color automaticly

General discussion about software packages and releases, new software you've found, and threads by programmers and script writers.
Post Reply
debuti
Posts: 2
Joined: 12 Nov 2013, 10:51
E-book readers owned: kindle
Number of books owned: 1000
Country: Spain

Adjust color automaticly

Post by debuti » 12 Nov 2013, 17:29

Hi!

I'm new to this forum, but not really new to book scanners, anyway, i made my own book scanner, it looks like a little house ;) (IMG_20131112_222021.jpg)

I use a pendulum to change camera orientation, I'll make a post to explain it, but by now, i'm stuck in color postprocessing. I've uploaded thre photos:
- IMG_2404_orig.jpg is the output from the camera (trimmed)
- Dibujo.jpg is the output from my HP scanner, i want the output to look like this, or like the output by scantailor
- IMG_2404.jpg is the best postprocessed page that i get...

My postprocessing script is

Code: Select all

#!/usr/bin/env bash

# Parameters
DATE=`date +%Y%m%d`
LOG_PATH=""
LOGLEVEL="DEBUG" #"INFO"

# Constants

# Global variables
log="$LOG_PATH"
inputLine=""
tempFolder="/tmp/scanBooks"
rights_parity=""
rotation_rights=""
rotation_lefts=""
degrees_rights=""
degrees_lefts=""
perspective_rights=""
perspective_lefts=""
trim_rights=""
trim_lefts=""
color_palette=""
threshold=""
resize=""
pages_input=""
pages_output=""

# Error declaration

 
# Usage function
function usage() {
  # Tell the user how to use me
  echo "$0 <rights_parity> <rotation_rights> <rotation_lefts> <degrees_rights> <degrees_lefts> <perspective_rights> <perspective_lefts> <trim_rights> <trim_lefts> <color_palette> <threshold/bitres> <resize> <pages_input> <pages_output>"
  
  echo "  <rights_parity>       odd or even"
  echo "  <rotation_rights>     in degrees"
  echo "  <rotation_lefts>      in degrees"
  echo "  <degrees_rights>      in degrees"
  echo "  <degrees_lefts>       in degrees"
  echo "  <perspective_rights>  "
  echo "  <perspective_lefts>   in format Sx1,Sy1 Dx1,Dy1   Sx2,Sy2 Dx2,Dy2   Sx3,Sy3 Dx3,Dy3   ...   Sxn,Syn Dxn,Dyn "
  echo "  <trim_rights>         "
  echo "  <trim_lefts>          in size and position in geometry (X0,Y0;X1,Y1)"
  echo "  <color_palette>       one of Bilevel, Grayscale"
  echo "  <threshold/bitres>    Bilevel threshold 0-100 (69 is ok) or Grayscale bit resolution"
  echo "  <resize>              resize percentage"
  echo "  <pages_input>         input directory"
  echo "  <pages_output>        output pdf"
  echo ""
  echo "HINT: ./scanBooks.sh odd \"90\" \"90\" \"355\" \"358\" \"560,460 0,0  2200,265 1640,0  585,2930 0,2470  2240,3000 1640,2470\" \"1200,240 0,0  3000,280 1640,0  1200,2930 0,2470  3020,2890 1640,2470\" \"0,0;1640,2470\" \"0,0;1640,2470\" Grayscale 69 \$INFOLDER \$INFOLDER/libro.pdf"
}
  
# Input validation function (getopts)
function checkInput() {
  # Check input and store params in global variables to use them from main or call usage()
  if [ $# -ne 14 ]; then
    echo "Recognized $# parameters"
    usage $0
    exit -1
  else
    rights_parity="$1"
    rotation_rights="$2"
    rotation_lefts="$3"
    degrees_rights="$4"
    degrees_lefts="$5"
    perspective_rights="$6"
    perspective_lefts="$7"
    trim_rights="$8"
    trim_lefts="$9"
    color_palette="${10}"
    threshold="${11}"
    resize="${12}"
    pages_input="${13}"
    pages_output="${14}"
    return 0
  fi 
}


# Dependencies validation
function checkDependencies() {
  # Repeat this foreach dependency
  command="convert"
  package="imagemagick"
  if ! which $command > /dev/null; then
    echo -e "Dependency not found. Install it by typing"
    echo -e "  sudo apt-get install $package"
    exit -1
  fi  

  # Repeat this foreach dependency
  command="pdftk"
  if ! which $command > /dev/null; then
    echo -e "Dependency not found. Install it by typing"
    echo -e "  sudo apt-get install $command"
    exit -1
  fi 
}
  
  
# Helper functions
function makeFolders() {     
  inFolder="$1"
  if [ "`ls -1 \"$inFolder\"| grep -i .jpg | wc -l`" -ne 0 ]; then
    echo "Creating directories in $inFolder"
    mkdir "$tempFolder"
    mv "$inFolder"/* "$tempFolder/."
    mkdir "$inFolder/1"
    mkdir "$inFolder/2"
    mkdir "$inFolder/3"
    mkdir "$inFolder/4"
    mkdir "$inFolder/5"
    mkdir "$inFolder/6"
    mv "$tempFolder"/* "$inFolder/1/."
  else
    rm -rf "$inFolder/2"/*
    rm -rf "$inFolder/3"/*
    rm -rf "$inFolder/4"/*
    rm -rf "$inFolder/5"/*
    rm -rf "$inFolder/6"/*
  fi
}

function rotate() { 
  inPhoto="$1"
  rotation="$2"
  outPhoto="$3"
  convert "$inPhoto" -rotate "$rotation" "$outPhoto" \
    && echo "  Applied $rotation degrees to $inPhoto"
}

function trim() { 
  inPhoto="$1"
  trim="$2"
  outPhoto="$3"
  function pointsToConvert(){
    input="$1"
    x0=`echo "$input" | cut -d";" -f1 | cut -d"," -f1`
    y0=`echo "$input" | cut -d";" -f1 | cut -d"," -f2`
    x1=`echo "$input" | cut -d";" -f2 | cut -d"," -f1`
    y1=`echo "$input" | cut -d";" -f2 | cut -d"," -f2`
    
    xn=`echo "${x1}-${x0}" | bc`
    yn=`echo "${y1}-${y0}" | bc`
    
    echo "${xn}x${yn}+${x0}+${y0}"
  }
  e=`pointsToConvert "$trim"`
  convert "$inPhoto" -crop "$e" "$outPhoto" \
    && echo "  Applied $e crop to $inPhoto"
}
  
function perspective() {
  inPhoto="$1"
  parameters="$2"
  outPhoto="$3"
  convert -verbose "$inPhoto" -matte -virtual-pixel transparent -distort perspective "$parameters" "$outPhoto" >/dev/null 2>/dev/null \
    && echo "  Applied $parameters distorsion to $inPhoto" \
    || echo "  Can not apply $parameters distorsion to $inPhoto"
}

function decolor() { 
  inPhoto="$1"
  type="$2"
  outPhoto="$3"
  #convert "$inPhoto" -type "$type" "$outPhoto" \
  #  && echo "  Applied $type colorspace to $inPhoto"
  
  convert "$inPhoto" \
          -edge 1 -negate -normalize \
          -colorspace Gray -blur 0x.5 -contrast-stretch 0x50% \
          "$outPhoto"
}

function correctcolor() {
  inPhoto="$1"
  type="$2"
  param="$3"
  outPhoto="$4"

  if [ "$type" == "Bilevel" -o "$type" == "Grayscale" ]; then
    convert "$inPhoto" -alpha copy -channel A -negate +channel "$tempFolder/"color-test.png  >/dev/null 2>/dev/null
    convert "$tempFolder/"color-test.png -background wheat -flatten "$tempFolder/"color-test.jpg  >/dev/null 2>/dev/null
    if [ "$type" == "Grayscale" ]; then
      #convert "$tempFolder/"color-test.jpg -colorspace Gray -depth 8 -resample 200x200 -verbose "$outPhoto"  >/dev/null 2>/dev/null 
      convert "$tempFolder/"color-test.jpg -colorspace Gray -depth "$param" "$outPhoto"  >/dev/null 2>/dev/null \
       && echo "  Applied "$param"bit Grayscale color to $inPhoto" \
       || echo "  Can not apply "$param"bit Grayscale color to $inPhoto"
    fi
    if [ "$type" == "Bilevel" ]; then
      convert "$tempFolder/"color-test.jpg -colorspace Gray -threshold "$param"% "$outPhoto"  >/dev/null 2>/dev/null \
       && echo "  Applied "$param"% threshold Bilevel color to $inPhoto" \
       || echo "  Can not apply "$param"% threshold Bilevel color to $inPhoto"
    fi
    rm "$tempFolder/"*
  fi
}

function resizr() {
  inPhoto="$1"
  value="$2"
  outPhoto="$3"
  
  convert "$inPhoto" -resize "$value"% "$outPhoto"  >/dev/null 2>/dev/null \
    && echo "  Applied "$value"% resize to $inPhoto" \
    || echo "  Can not apply "$value"% resize to $inPhoto"
}

function correctcolor1() {
  inPhoto="$1"
  type="$2"
  outPhoto="$3"

TRUE_RED=162
TRUE_GREEN=162
TRUE_BLUE=160
    cp "$inPhoto" "$outPhoto"
    
	echo "  Determining RGB values of "$inPhoto"..."
	#Convert RAW file extension and crop an area in the center of calibration image
	convert "$inPhoto" -gravity center  "$tempFolder/"color-test.png

	#Determine average colors in cropped area
	SOURCE_RED=$(convert "$tempFolder/"color-test.png -resize 1x1 -format "%[fx:int(255*p{10,10}.r)]" info:)
	SOURCE_GREEN=$(convert "$tempFolder/"color-test.png -resize 1x1 -format "%[fx:int(255*p{10,10}.g)]" info:)
	SOURCE_BLUE=$(convert "$tempFolder/"color-test.png -resize 1x1 -format "%[fx:int(255*p{10,10}.b)]" info:)
	echo "  Detected RGB values are $SOURCE_RED, $SOURCE_GREEN, $SOURCE_BLUE"
	
	#Calculate necessary adjustments
	RED_ADJUST="$(echo "scale=10; $TRUE_RED/$SOURCE_RED" | bc)"
	GREEN_ADJUST="$(echo "scale=10; $TRUE_GREEN/$SOURCE_GREEN" | bc)"
	BLUE_ADJUST="$(echo "scale=10; $TRUE_BLUE/$SOURCE_BLUE" | bc)"
	echo "  RGB values to be adjusted by $RED_ADJUST, $GREEN_ADJUST, $BLUE_ADJUST"

	#Adjust Colors and Output Raw images as ppm files
	echo "  Running Imagemagick to adjust colors and convert $INPUT_FORMAT to $OUTPUT_FORMAT..."
	mogrify -format ppm $IM_VERBOSE_OPTION  -color-matrix "$RED_ADJUST 0 0 0 $GREEN_ADJUST 0 0 0 $BLUE_ADJUST" "$outPhoto"
	
	#Remove temporary file
	rm "$tempFolder/"color-test.png
}

function makebook() { 
  inFolder="$1"
  outFile="$2"
  filename=$(basename "$outFile")
  filename="${filename%.*}"
  
  rm -rf "$tempFolder"/*
  for i in `ls -1 "$inFolder"`; do
    convert -compress jpeg "$inFolder"/$i "$tempFolder"/$i.pdf
  done;
  pdftk "$tempFolder"/*.pdf cat output "$outFile" && rm "$tempFolder"/*.pdf 
  
  # Add line as a comment
  pdftk "$outFile" dump_data output "$tempFolder/"info.txt \
  && echo "InfoKey: Title
InfoValue: "$filename"
InfoKey: Subject
InfoValue: "$inputLine"
" | cat - "$tempFolder/"info.txt > "$tempFolder/"temp \
  && mv "$tempFolder/"temp "$tempFolder/"info.txt \
  && pdftk "$outFile" update_info "$tempFolder/"info.txt output "$tempFolder/"temp.pdf \
  && mv "$tempFolder/"temp.pdf "$outFile" \
  && rm "$tempFolder/"info.txt \
  && echo "Done $outFile"

}

  
# Main function
function main() {
  rotation=""
  perspect=""
  trim=""
  
  # Hacer carpetas para cada paso
  makeFolders "$pages_input"
  
  for file in `ls -1 "$pages_input/1"`; do
    if [ `echo "$file" | egrep -i *[02468].jpg | wc -l` -eq 1 ]; then 
      if [ $rights_parity == "odd" ]; then  
        rotation=`echo "$rotation_rights + $degrees_rights" | bc`
        perspect="$perspective_rights"
        trim="$trim_rights"
      fi;
      if [ $rights_parity == "even" ]; then 
        rotation=`echo "$rotation_lefts + $degrees_lefts" | bc`
        perspect="$perspective_lefts"
        trim="$trim_lefts"
      fi;
    fi;
    if [ `echo "$file" | egrep -i *[13579].jpg | wc -l`  -eq 1 ]; then 
      if [ $rights_parity == "odd" ]; then 
        rotation=`echo "$rotation_lefts + $degrees_lefts" | bc`
        perspect="$perspective_lefts"
        trim="$trim_lefts"
      fi;
      if [ $rights_parity == "even" ]; then 
        rotation=`echo "$rotation_rights + $degrees_rights" | bc`
        perspect="$perspective_rights"
        trim="$trim_rights"
      fi;
    fi;
    
    echo "$file: "
    
  # Reorientar y alinear
    rotate "$pages_input/1/$file" "$rotation" "$pages_input/2/$file";
  # Retomar perspectiva
    perspective "$pages_input/2/$file" "$perspect" "$pages_input/3/$file";
  # Recortar
    trim "$pages_input/3/$file" "$trim" "$pages_input/4/$file"; 
  # Escala de grises
    correctcolor "$pages_input/4/$file" "$color_palette" "$threshold" "$pages_input/5/$file"
  # Resize
    resizr "$pages_input/5/$file" "$resize" "$pages_input/6/$file"
  done;
  
  # Generar salida  
  makebook "$pages_input/6"  "$pages_output"
}
  
# Entry point
checkDependencies
checkInput "$@"
inputLine="$@"
main

  
Any ideas?

Thanks a lot (and sorry for my english)
Attachments
IMG_20131112_222021.jpg
My scanner
IMG_2404_orig.JPG
My scanner PreProcessed file
IMG_2404.JPG
My scanner PostProccessed file
Dibujo.JPG
HP scannered page

debuti
Posts: 2
Joined: 12 Nov 2013, 10:51
E-book readers owned: kindle
Number of books owned: 1000
Country: Spain

Re: Adjust color automaticly

Post by debuti » 13 Nov 2013, 09:16


rkomar
Posts: 87
Joined: 12 May 2013, 16:36
E-book readers owned: PRS-505, PocketBook 902, PRS-T1, PocketBook 623, PocketBook 840
Number of books owned: 3000
Country: Canada

Re: Adjust color automaticly

Post by rkomar » 13 Nov 2013, 13:29

Based on your script, it looks like you just need to set a few parameters to clean up the image. You should probably "trim" the side next to the spine of the book to get rid of the shadows there. You should also either set the "threshold" high enough to make the background white, or just set the "colour_palette" to Bilevel to make the whole image black and white.

dpc
Posts: 314
Joined: 01 Apr 2011, 18:05
Number of books owned: 0
Location: Issaquah, WA

Re: Adjust color automaticly

Post by dpc » 13 Nov 2013, 18:22

I'm looking forward to your post describing your camera pendulum scanner in more detail. I thought about doing something like this a few years ago with a standard scanner but not invert the platen as you have done. I wanted to use a better camera and thought that by swinging the camera over to the other side I would only need one of them. In the end, I just skipped a few restaurant dinners and bought another DSLR. Besides, shooting all the even number pages then flipping the book around and scanning the odd number pages was easier on the camera than swinging it back and forth so I forgot about this idea entirely.

One thing I did want to mention is that in the images that you've posted from your scanner I can see a reflection of the lighted page from the adjacent platen panel. This shows up as a sort of trapezoid from the left side of the image toward the center of the page. Depending your lighting method, you can typically eliminate this by increasing the angle between the two platen halves to something greater than 90 degrees (from Dan's investigations 100 deg. works well). This isn't much of an issue if your going to scan mostly text and convert it to a black & white image, but if you scan things that contain images (graphic novels, magazines, etc.) it becomes more important.

For scanners such as yours that have a fixed camera position relative to the platen surface, you can scan a calibration page (gray card, or simply a blank white page) then make a note of the post processing per-pixel in the image that needs to be done to make that page appear the correct color (usually white). That same post processing can be applied later to your scanned images and should help with image artifacts that result from uneven lighting.

User avatar
daniel_reetz
Posts: 2797
Joined: 03 Jun 2009, 13:56
E-book readers owned: Used to have a PRS-500
Number of books owned: 600
Country: United States
Contact:

Re: Adjust color automaticly

Post by daniel_reetz » 15 Nov 2013, 11:55

Cool stuff debuti! Any chance you should share a short video clip of how the scanner works?

steve1066d
Posts: 296
Joined: 27 Nov 2010, 02:26
E-book readers owned: PRS-505
Number of books owned: 1250
Location: Minneapolis, MN
Contact:

Re: Adjust color automaticly

Post by steve1066d » 15 Nov 2013, 20:43

Debuti,

I was wondering if you could post or send me an image of a blank page from your scanner. I think BSW could normalize the lighting over the page, but I'd like to have a sample image to see how well it would work.

It works by using an image of a blank page to see where it is brighter and darker, and using that information adjusts the brightness of the other images.
Steve Devore
BookScanWizard, a flexible book post-processor.

victoriaaustralia
Posts: 54
Joined: 07 Nov 2011, 16:22
E-book readers owned: newton
Number of books owned: 2
Country: Australia
Location: Castlemaine, Victoria, Australia

Re: Adjust color automaticly

Post by victoriaaustralia » 20 Nov 2013, 20:44

That is a really unique approach to a single camera inverted scanner!

Other people have used the similar roof gable design but with a single stationary camera, aimed at the spine of the open inverted book:

http://diybookscanner.org/forum/viewtop ... 8150#p8150

http://www.diybookscanner.org/forum/vie ... f=14&t=821

http://www.diybookscanner.org/forum/vie ... hilit=roof

This has the gable design but with two stationary cameras:
http://www.diybookscanner.org/forum/vie ... ner#p13843

And this design is the reverse of yours - stationary camera, rotating upside down book:

http://www.diybookscanner.org/forum/vie ... gle+camera

Apparently the bright lights shining up at the operator can be fatigue inducing with these designs.
Freeware Windows workflow: 40Mb 400pg OCR'd A4 book pdfs
http://www.diybookscanner.org/forum/vie ... =19&t=2835

Post Reply