Daniel Reetz, the founder of the DIY Book Scanner community, has recently started making videos of prototyping and shop tips. If you are tinkering with a book scanner (or any other project) in your home shop, these tips will come in handy. https://www.youtube.com/channel/UCn0gq8 ... g_8K1nfInQ

Automatic colour calibration using Argyll CMS

General discussion about software packages and releases, new software you've found, and threads by programmers and script writers.
User avatar
Misty
Posts: 481
Joined: 06 Nov 2009, 12:20
Number of books owned: 0
Location: Frozen Wasteland

Automatic colour calibration using Argyll CMS

Post by Misty » 20 Sep 2012, 17:43

I've been looking at options for automatic colour calibration of book scanner images. I don't trust camera's auto colour calibration, or software's white balancing, to really produce a decent colour calibration. So I decided to give the open source colour calibration software Argyll CMS a try.

I started with an incredibly crappy, handheld photo of an IT8 chart. IT8 is a standard colour chart with a set of predefined colours; I already had one on hand that I use with my scanner. This photo is really terrible and probably acts as a good stress test just because of how bad it is - it's not perfectly flat, it's kinda out of focus, it's not really a straight on view. Just awful.
it8chart-fromcamera.jpg
it8chart-fromcamera.jpg (99.18 KiB) Viewed 8984 times
The camera did... okay. For auto-white balance, anyway. The tones aren't really right, it's got a colour cast going on, etc. Time to feed it into Argyll!

First, I converted the .CR2 raw file into a TIFF using dcraw, the practically universal raw format decoder. To get as raw a conversion as possible, I used these options:

Code: Select all

  -4: linear 16-bit. Doesn't convert the image from its native gamma to a standard curve, uses a fixed white level, and writes 16 bits per pixel instead of 8 bits.
  -T: output as a TIFF
-q 3: Highest image quality
-o 0: Don't try to convert the colours into a standard colour space, leave them as is
The result is the rawest TIFF I could get, which doesn't look like much to the eye since it's totally uncalibrated:
itchart-rawtiff.jpg
itchart-rawtiff.jpg (59.49 KiB) Viewed 8984 times
Eww. Time to throw Argyll at it!

To get a colour profile from Argyll, I worked in two steps. First, I created a native Argyll colour information file using argyll's "scanin" tool:

Code: Select all

scanin photo.tiff it8.cht R1009005.txt
"it8.cht" is a reference to the IT8 colours. Since IT8 is a standard, a suitable it8.cht file comes with Argyll. "R1009005.txt" is a set of colour readings from the chart; this was provided by the chart's manufacturer.

The result of this was "photo.ti3"; TI3 is an internal format of Argyll's, which isn't really useful to us. So I converted that into a standard ICC profile using Argyl's "colprof" tool:

Code: Select all

colprof -v -D"My awesome colour profile!" -qm -ax photo
This produced a LUT profile; Argyll supports a few modes, but I found that this was the most accurate.

So, I went and applied that ICC profile to the TIFF I got out of dcraw, and this is what it looks like:
it8chart-icc.jpg
it8chart-icc.jpg (90.87 KiB) Viewed 8984 times
Pretty good! Slight green tinge in a particular set of the greys, otherwise quite accurate - which given the totally uncontrolled lighting and poor imaging, is quite good performance. Hoping I can improve my results further.

If anyone else has had experience with this - please chime in!
The opinions expressed in this post are my own and do not necessarily represent those of the Canadian Museum for Human Rights.

User avatar
Misty
Posts: 481
Joined: 06 Nov 2009, 12:20
Number of books owned: 0
Location: Frozen Wasteland

Re: Automatic colour calibration using Argyll CMS

Post by Misty » 20 Sep 2012, 17:57

I just ran another ICC conversion, this time using a LUT profile with LAB coordinates instead of XYZ coordinates (which is what -ax produces). The results are quite superior to the last attempt:
it8chart-icc2.jpg
it8chart-icc2.jpg (92.89 KiB) Viewed 8981 times
Very good colour accuracy without colour cast; definitely a nice improvement. (Unfortunately not all of that will show up in the photo - SRGB can't represent all those colours!)
The opinions expressed in this post are my own and do not necessarily represent those of the Canadian Museum for Human Rights.

User avatar
Misty
Posts: 481
Joined: 06 Nov 2009, 12:20
Number of books owned: 0
Location: Frozen Wasteland

Re: Automatic colour calibration using Argyll CMS

Post by Misty » 21 Sep 2012, 10:24

One more photo. This one also uses a LUT profile with LAB coordinates (-al) along with the -un setting, which a) avoids clipping values whiter than the "white" on the test chart, and b) does not extrapolate colour values outside what the test chart shows. I would have expected the extrapolated lighter and darker values (from -u) to improve the image quality, but in fact the results were quite poor - maybe because the camera was capturing a gamut much wider than what the test chart could represent.

Here's the result of -al -un:
it8chart-icc3.jpg
it8chart-icc3.jpg (89.66 KiB) Viewed 8965 times
I'm really pleased with the results; this results are very, very true to the real test chart with no clipping.
The opinions expressed in this post are my own and do not necessarily represent those of the Canadian Museum for Human Rights.

User avatar
Misty
Posts: 481
Joined: 06 Nov 2009, 12:20
Number of books owned: 0
Location: Frozen Wasteland

Re: Automatic colour calibration using Argyll CMS

Post by Misty » 17 May 2013, 12:39

Now that I have a proper camera and a real scanning environment, I've been doing some more testing with Argyll CMS. Here's the latest experiment:

This shot was produced on a Bencher Copymate copystand with fluorescent lighting, with a Nikon D800 using the Nikkor 60mm 1:2.8 macro lens. Let me first say that this camera is *sweet*. Incredible detail, lovely colours, etc. I am in love.

So, I shot a test chart and processed it using argyll 1.5.1, the latest release. The .ti3 file was converted into an ICC profile using the following settings:

Code: Select all

colprof -v -D"2013-05-17" -al DSC_0028 -O 2013-05-17.icc
Here's the colour chart with the profile embedded:
2013-05-17.jpg
2013-05-17.jpg (227.62 KiB) Viewed 8347 times
And an sRGB version, for those people with monitors that don't do colour profiles:
2013-05-17-srgb.jpg
2013-05-17-srgb.jpg (150.27 KiB) Viewed 8347 times
These results are *really* good. They do a fantastic job of reproducing the original colours! Unfortunately the sRGB file can't properly show how good it looks (some of the colours are lost in the conversion).

Soon I'll dig some laserdiscs out to shoot so I have sample colour-corrected real images to show you! Unfortunately the stuff I'm scanning for work right now I'm not able to post publicly ATM.
The opinions expressed in this post are my own and do not necessarily represent those of the Canadian Museum for Human Rights.

abmartin
Posts: 79
Joined: 15 Sep 2010, 15:33
Number of books owned: 2000
Country: USA
Location: Ohio

Re: Automatic colour calibration using Argyll CMS

Post by abmartin » 17 May 2013, 19:56

Misty,

Thanks a lot for showing us that! I have one small question. From what I remember, those scanner-calibration cards are pretty reflective. Is there an issue with the reflective nature of the material interfering with the color detection? I've had problems with glossy book pages in the past. Judging by your results, I suspect that it isn't a huge problem. although I see in your recent version that there are a few colors that seem a bit washed out by reflections in the bottom corners. It seems like the correction was still done properly though. (Maybe they make a matte version of these cards?)

Hopefully in a few weeks, I'll have a chance to get a card and try this out. I've written a color correction script to work with gray cards, but this is much more accurate than that! It seems as if it wouldn't be difficult to script these actions. I wish I had read this before I posted a final version of my preprocess script!

I've always wanted a way to script a more sophisticated color calibration. A lot of the archives I browse photograph color calibration cards, but I have never understood how they are used without blowing mega dollars on profiling software. Great to see a libre software approach! (It looks like Argyll is GPL and AGPL)

Thanks again!!!

User avatar
Misty
Posts: 481
Joined: 06 Nov 2009, 12:20
Number of books owned: 0
Location: Frozen Wasteland

Re: Automatic colour calibration using Argyll CMS

Post by Misty » 21 May 2013, 10:40

Yep, you're quite right about reflective material! given the fluorescent lighting, glare was something I'm concerned about. I'd asked Graeme about it; his response was that it's best to use a colour target that best resembles the kind of things you're scanning. Given that I'm not usually scanning glossy, photo-like stuff, this kind of IT8 chart is the wrong tool for the job.

In my case I'm using an IT8 chart simply because I already owned one for flatbeds, and it's decent for testing with. I wouldn't recommend buying one for this purpose though. Argyll supports a variety of charts for creating input profiles. I'm looking at buying a ColorChecker SG chart, which is widely available, and with a texture designed for camera scanning. It's expensive ($259.00), but for the stuff I'm doing it looks like it should work great.

There are some cheaper cards supported by Argyll, too.
The opinions expressed in this post are my own and do not necessarily represent those of the Canadian Museum for Human Rights.

User avatar
Misty
Posts: 481
Joined: 06 Nov 2009, 12:20
Number of books owned: 0
Location: Frozen Wasteland

Re: Automatic colour calibration using Argyll CMS

Post by Misty » 21 May 2013, 11:03

Something else I'd come across I should mention: because of the nature of input profiles, colour correction is really weird if you try to do it using the profile that Argyll produces.

What Argyll produces is an input profile - that is, a colour profile that attempts to reproduce the colours that can be represented by the input device, e.g. the camera in the specific shooting conditions under which the test chart was shot. The result is that the primaries aren't actually what you'd expect them to be, and applying colour curves might try to introduce colours that don't actually exist in that profile.

To do colour corrections, it's best to convert from the input profile to a working profile that can represent all the colours contained in your input profile. In my case I've been using ProPhotoRGB, but your results may vary depending on your camera/colour chart/ICC profile/usecase, &c.

Here's a practical example. I used the same photo and colour chart here, then applied a colour curve with a single 255, 113 point on it in Photoshop. Using the input profile, the result looked something like this (result converted to sRGB):
input_example_1.jpg
input_example_1.jpg (173.78 KiB) Viewed 8260 times
Whereas when I first converted to sRGB before performing the same colour curve, the result looks like this:
input_example_2.jpg
input_example_2.jpg (159.99 KiB) Viewed 8260 times
255,255,255 is simply not white in this colour space.
The opinions expressed in this post are my own and do not necessarily represent those of the Canadian Museum for Human Rights.

dpc
Posts: 314
Joined: 01 Apr 2011, 18:05
Number of books owned: 0
Location: Issaquah, WA

Re: Automatic colour calibration using Argyll CMS

Post by dpc » 21 May 2013, 18:07

How do you handle the uneven surface lighting of a diy bookscanner when calibrating with a color card?

For instance, would your color compensation values change if you were to flip that card upside down?

Should the intensity be normalized using a gray card (or even a blank book page) before applying these color adjustments?

User avatar
Misty
Posts: 481
Joined: 06 Nov 2009, 12:20
Number of books owned: 0
Location: Frozen Wasteland

Re: Automatic colour calibration using Argyll CMS

Post by Misty » 22 May 2013, 11:15

Good question. I'm fortunate enough to be dealing with even illumination right now, glare aside, but that could definitely be an issue.

Evening out the level of illumination over the frame before you do this would probably be a good idea.
The opinions expressed in this post are my own and do not necessarily represent those of the Canadian Museum for Human Rights.

User avatar
Misty
Posts: 481
Joined: 06 Nov 2009, 12:20
Number of books owned: 0
Location: Frozen Wasteland

Re: Automatic colour calibration using Argyll CMS

Post by Misty » 22 May 2013, 18:13

OK, I got the lighting under control - the problem with the test chart was 100% due to the overhead lights in the room. I had the rare opportunity to black out half the floor for the sake of scanning stuff today, so I'll have some beautiful sample photos to post tomorrow.
The opinions expressed in this post are my own and do not necessarily represent those of the Canadian Museum for Human Rights.

Post Reply