How and why does VueScan's TIFF and DNG output differ?

Question

The vuescan application supports saving scans DNG and TIFF format which can be processed and/or what it calls raw. I would like to understand the differences between the formats to help decide the best choice for archival storage.

I performed some tests with two 48bit sample scans taken at 2400dpi. The first summarised like this:

293936902 281M 2016-06-18-0001.dng   Zip
293918182 281M 2016-06-18-R-0001.dng Zip

293805386 281M 2016-06-18-0002.tif   None
293783510 281M 2016-06-18-R-0002.tif None

237436594 227M 2016-06-18-0004.tif   LZW   smallest
326009196 311M 2016-06-18-R-0004.tif LZW   largest

The first pair were saved as DNG, the one with -R- in the name is raw. The second pair are uncompressed TIF and the third is compressed TIF. The second sample follows a similar pattern:

raw.dng  124159636 119M ZIP 
save.dng 124159636 119M ZIP 

raw.tif  124051702 119M None
save.tif 124051702 119M None

raw.tif  154583376 101M LZW    smallest
save.tif 105012638 148M LZW    largest

Inspecting the files with Imagemagick showed the raw files to be darker (as expected for gamma 1) but Imagemagick reported the gamma as 0.454545 for all files. The differences that I identified were in what Imagemagick reported as the Channel statistics plus, in the DNG files, dng:ChannelMultipliers, dng:Green and dng:Temperature.

There is no option in VueScan to select the compression mechanism (i.e. ZIP OR LZW), only whether it is on or off, and that is only for TIFF; the DNG format is always ZIP compressed. Perhaps that is a requirement of DNG?

The DNG files being the same size as the uncompressed TIFF suggest to me that the ZIP compression is ineffective. But why is there such a huge size difference between the LZW compressed TIFF files when the uncompressed TIFF and DNG files are the same size ?

Considering the above, I would choose between the LZW compressed TIFF (because it is the smallest) or uncompressed raw TIFF unless there is a compelling reason to use DNG.

So I can see the formats have different outputs but I don't understand all of the differences, some of which are unexpected - such as the file sizes and the reported gamma values. Without understanding this it's difficult to select the most approproate file to archive. Hence the question:

How do these formats differ, especially considering an archival use-case and, with VueScan being the tool of choice, do these differences make VueScan's DNG or TIFF more appropriate for archiving?

(I have sought answers in the manual and bible prior to asking here.)

Additional information

Here are Lightroom histograms for four images:

Raw DNG Raw TIFF

Raw DNG Raw TIFF

Save DNG Save TIFF

Save DNG Save TIFF

All four histograms are different, which would indicate that is more than a simple format difference between TIFF and DNG. There is very little difference between the raw and save DNG files but the TIFFS are quite different, and the raw TIFF is significantly different to the others.

The save TIFF file is the only one with an embedded ICC profile and that is "sRGB IEC61966-2.1".

Below (on the left, the smaller image) is a snap of the VueScan preview, which I would say looks closest to the save TIFF file. The larger image is the save-tiff image with the scanner's calibrated ICC profile instead of the default sRGB.

Answer 1

The image pixel data contained in raw TIFF and DNG files produced by VueScan is identical. This can be proved by extracting and comparing that data. This can be done using Linux dcraw and imagemagick command-line tools:

$ diff -s \
 <(stream -map rgb -storage-type short raw.tif - | sha256sum) \
 <(dcraw -E -4 -c raw.dng | tail -n +4 | dd conv=swab 2> /dev/null | sha256sum)
Files /dev/fd/63 and /dev/fd/62 are identical

(I won't muddy this answer with a technical explanation - add a comment if that would be of interest here.)

The overall file content, however, is different. A DNG file is a TIFF file with a specific structure that includes a low-resolution preview and special metadata. Of particular note are the following tags:

• ColorMatrix1
• As Shot White XY
• Baseline Noise

These three tags are set by VueScan and used by a developer application (such as DarkTable) as an input colour profile (which DarkTable lists as embedded matrix). There are some other tags but they don't affect how the image is processed.

You can take a VueScan DNG and "re-scan" it into a raw TIFF and the pixel data in both files will be the same. You could alternatively use dcraw to do the same thing: $ dcraw -E -4 -T raw.dng

So the issues raised in the question are not so much due to the differences between the file contents but how Darktable (or whatever) processes the different files. You can make a DNG look like a TIFF by removing the abovementioned tags and selecting the sRGB colour space.

With regard to file sizes - I haven't looked at the effectiveness of compression algorithms because it would appear to be, as others have suggested, dependent on the distribution of the input data. But, comparing like-for-like, the image bitmap data content is identical.

What remains unclear is how VueScan computes ColorMatrix1, AsShotWhiteXY and BaselineNoise.

Answer 2

The difference might not be in the data contained in the comparable uncompressed 48-bit TIFF and DNG files, but rather in the way Imagemagick is interpreting that data to present it on your 24-bit screen.

I'm not sure how it works out with data from a scanner, but with raw data from a camera that includes proprietary information such as data from masked pixels gathered on the sensor, converting to DNG does "bake in" the black point and white point of an image file by throwing away the information from the masked pixels as well as the other proprietary data that some sensors collect.