7

The vuescan application supports saving scans DNG and TIFF format which can be processed and/or what it calls raw. I would like to understand the differences between the formats to help decide the best choice for archival storage.

I performed some tests with two 48bit sample scans taken at 2400dpi. The first summarised like this:

293936902 281M 2016-06-18-0001.dng   Zip
293918182 281M 2016-06-18-R-0001.dng Zip

293805386 281M 2016-06-18-0002.tif   None
293783510 281M 2016-06-18-R-0002.tif None

237436594 227M 2016-06-18-0004.tif   LZW   smallest
326009196 311M 2016-06-18-R-0004.tif LZW   largest

The first pair were saved as DNG, the one with -R- in the name is raw. The second pair are uncompressed TIF and the third is compressed TIF. The second sample follows a similar pattern:

raw.dng  124159636 119M ZIP 
save.dng 124159636 119M ZIP 

raw.tif  124051702 119M None
save.tif 124051702 119M None

raw.tif  154583376 101M LZW    smallest
save.tif 105012638 148M LZW    largest

Inspecting the files with Imagemagick showed the raw files to be darker (as expected for gamma 1) but Imagemagick reported the gamma as 0.454545 for all files. The differences that I identified were in what Imagemagick reported as the Channel statistics plus, in the DNG files, dng:ChannelMultipliers, dng:Green and dng:Temperature.

There is no option in VueScan to select the compression mechanism (i.e. ZIP OR LZW), only whether it is on or off, and that is only for TIFF; the DNG format is always ZIP compressed. Perhaps that is a requirement of DNG?

The DNG files being the same size as the uncompressed TIFF suggest to me that the ZIP compression is ineffective. But why is there such a huge size difference between the LZW compressed TIFF files when the uncompressed TIFF and DNG files are the same size ?

Considering the above, I would choose between the LZW compressed TIFF (because it is the smallest) or uncompressed raw TIFF unless there is a compelling reason to use DNG.

So I can see the formats have different outputs but I don't understand all of the differences, some of which are unexpected - such as the file sizes and the reported gamma values. Without understanding this it's difficult to select the most approproate file to archive. Hence the question:

How do these formats differ, especially considering an archival use-case and, with VueScan being the tool of choice, do these differences make VueScan's DNG or TIFF more appropriate for archiving?

(I have sought answers in the manual and bible prior to asking here.)

Additional information

Here are Lightroom histograms for four images:

Raw DNG Raw DNG Raw TIFF Raw TIFF

Raw DNG Raw DNG Raw TIFF Raw TIFF

Save DNG Save DNG Save TIFF Save TIFF

Save DNG Save DNG Save TIFF Save TIFF

All four histograms are different, which would indicate that is more than a simple format difference between TIFF and DNG. There is very little difference between the raw and save DNG files but the TIFFS are quite different, and the raw TIFF is significantly different to the others.

The save TIFF file is the only one with an embedded ICC profile and that is "sRGB IEC61966-2.1".

Below (on the left, the smaller image) is a snap of the VueScan preview, which I would say looks closest to the save TIFF file. The larger image is the save-tiff image with the scanner's calibrated ICC profile instead of the default sRGB.

Preview save tiff, scanner profile

  • 1
    Please ask only one question. – Olivier Jun 18 '16 at 21:43
  • Perhaps I worded it wrong but I meant it as one question because, I believe, VueScan's raw and its DNG implementation aren't necessarily the same as what one gets from camera raws. I asked specifically within the context of VueScan (had there been a VueScan tag I would have used it). Given VueScan's perhaps esoteric use of DNG I think the 'most appropriate format' is a factor of the format differences rather than subjective opinion. I've re-worded slightly. – starfry Jun 19 '16 at 8:37
  • Just to add, my thinking at the minute is to archive the processed save file because I plan to profile my scanner (I just need to buy a target) and I think its worth archiving the image taking into consideration the scanner profile. Also, the reading I have done leads me to believe that VueScan's use of DNG is not in line with the format's intended purpose - i.e. to carry unprocessed camera raw data rather than scanner rgb – starfry Jun 19 '16 at 8:42
  • I've just discovered here that the gamma reported by imagemagick is the amount of correction it expects the display to do, not what's in the image. I've only started looking into this detail but I think perhaps gamma is a latent characteristic of the image data (i.e. because correcting the image alters its pixel values). – starfry Jun 22 '16 at 9:43
1

The difference might not be in the data contained in the comparable uncompressed 48-bit TIFF and DNG files, but rather in the way Imagemagick is interpreting that data to present it on your 24-bit screen.

I'm not sure how it works out with data from a scanner, but with raw data from a camera that includes proprietary information such as data from masked pixels gathered on the sensor, converting to DNG does "bake in" the black point and white point of an image file by throwing away the information from the masked pixels as well as the other proprietary data that some sensors collect.

  • As far as I understand it, the "raw" scanner data is 48-bit rgb which, from what I've read, isn't normally what goes in a DNG (i.e. camera raw). – starfry Jun 20 '16 at 12:32
  • @starfry Yeah, but your monitor can likely only display at 24-bit rgb. Even if your monitor is capable, the application very likely assumes the monitor is limited to 24-bit and generates an image that is 24-bit. So the 48-bit data in the file is not all being displayed when you view it on the monitor, it has been converted to 24-bit. What the answer is suggesting is that the application you are using to view the files doesn't convert the 48-bit data from the DNG in the same way it converts the same 48-bit data from the TIFF. – Michael C Jun 20 '16 at 16:10
  • I know the monitor has a 24-bit display characteristic; as long as I view the images using the same software (i.e. gimp/photoshop/imagemagick) on the same hardware (i.e. monitor/video card) then I'm doing a fair comparison. As far as I know (I did check), the same rendering intent is used for both images. I think the difference is due to one being gamma corrected by VueScan to 2.2 before being saved (which is expected). This doesn't explain differences between DNG/TIFF or why their LZW-compressed sizes are so different. – starfry Jun 22 '16 at 9:52
  • I've updated the question to illustrate the differences seen in Lightroom. – starfry Jun 25 '16 at 11:04
  • @starfry Even when rendered by the same software, different processes may be used to render files in different formats. This might be true even when the two different formats are just "containers" for the same basic data. How that basic data is handled by the rendering application might be different for each due to the way the application has been coded. – Michael C Jun 25 '16 at 20:53
1

The image pixel data contained in raw TIFF and DNG files produced by VueScan is identical. This can be proved by extracting and comparing that data. This can be done using Linux dcraw and imagemagick command-line tools:

$ diff -s \
 <(stream -map rgb -storage-type short raw.tif - | sha256sum) \
 <(dcraw -E -4 -c raw.dng | tail -n +4 | dd conv=swab 2> /dev/null | sha256sum)
Files /dev/fd/63 and /dev/fd/62 are identical

(I won't muddy this answer with a technical explanation - add a comment if that would be of interest here.)

The overall file content, however, is different. A DNG file is a TIFF file with a specific structure that includes a low-resolution preview and special metadata. Of particular note are the following tags:

  • ColorMatrix1
  • As Shot White XY
  • Baseline Noise

These three tags are set by VueScan and used by a developer application (such as DarkTable) as an input colour profile (which DarkTable lists as embedded matrix). There are some other tags but they don't affect how the image is processed.

You can take a VueScan DNG and "re-scan" it into a raw TIFF and the pixel data in both files will be the same. You could alternatively use dcraw to do the same thing: $ dcraw -E -4 -T raw.dng

So the issues raised in the question are not so much due to the differences between the file contents but how Darktable (or whatever) processes the different files. You can make a DNG look like a TIFF by removing the abovementioned tags and selecting the sRGB colour space.

With regard to file sizes - I haven't looked at the effectiveness of compression algorithms because it would appear to be, as others have suggested, dependent on the distribution of the input data. But, comparing like-for-like, the image bitmap data content is identical.

What remains unclear is how VueScan computes ColorMatrix1, AsShotWhiteXY and BaselineNoise.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.