Why does Darktable export enormous JPEGs?

Question

I have a Nikon D3200 and just started shooting in Raw.

Raw files are 18-24mb, and if I convert them to JPEG using Nikon's ViewNX 2, I get 8-12mb files at the top quality and resolution using sRGB; the same size that I would get out-of-camera shooting in JPG with the maximum quality.

Instead using Darktable with the same options (quality 100 and sRGB at the maximum resolution), I get files of 25-30mb, which is a lot — even more than the original Raw.

Is there a reason for that? How can I change it? If it's not possible to change, which setting would you advise me to use in Darktable to get smaller files without loosing too much quality?

Answer 1

It seems that the large file size is expected for the settings you chose.

When I take a NEF image from a Nikon D5100 and convert it to JPEG using the sRGB colour space and quality 100, the result is about 18MB in size. Doing the same with dcraw and ImageMagick's convert also leads to a 18MB file.

JPEG 100 quality should probably be never needed. If you want lossless files, use TIFF. For display JPEG 100 is not needed, you can safely use something like 95 and not be able to tell the difference.

What JPEG compression to pick can be researched further and there are questions about this here, but picking a value in the 90s will significantly reduce the size and be fit for purpose for most applications.

Answer 2

Uncompressed RGB files (3 values per pixel) will be larger than your raws, as the raws contain a monochrome bitmap (1 value per pixel), and usually a downscaled, aggressively compressed preview that takes a fraction of the size, 400k for Canon 10MP cameras, and 1M for Nikon D5100 (I know these numbers because I used to read them out of the raw files and store them in a temporary file). Nikon NEFs store full res previews while Canon uses half res, hence the larger fraction.

So you can expect an uncompressed 16bit RGB "bitmap" to be approaching 3 times larger than the raw, and times 3 divided by 2 for 8bit uncompressed. A 16MP 8bit image is 48Mb uncompressed.

Now to the point, since we are talking about jpegs here. It is the uncompressed bitmap size that is the base size for the compression to work its way down from , not the raw size. This means that if you set the compression to be very gentle, it can easily still be larger than the raw.

The jpeg quality numbers are arbitrary - all you know for sure is that quality 100 is larger than 90, and 90 is larger than 80 in the same program. With the JPEGLIB that I use for my jpeg handling a 16MP image becomes 4Mb, and it quickly goes down to 1.7Mb at 95, at no perceivable quality hit. I normally keep it at 80, which is 950k. And flipping between the 4Mb and the 950k file at 100% zoom, I can see no difference. So it seems like overkill that your 100% is barely compressing at all.

So you should take an image with a lot of details and a clear sky, and save then at 100, 90,80,70,60,50, until you can see a quality hit. For your 24MP images you will probably hit that limit at around 1.5Mb, whichever the jpeg quality number that may be in your program.

Answer 3

The discrepancy in size between JPEGs encoded at "100% quality" in different encoders is likely to be a result of one of the following:

• Different chroma sub-sampling setting. Chroma sub-sampling in JPEGs reduces the spatial resolution of the chroma channels in return for reducing the file size with little noticeable different in quality, at least of most photographic material.

  Different JPEG encoders may adjust the chroma sub-sampling differently: some may adjust it along with the quality setting, so a higher quality will also use less sub-sampling. Some may use a different, separate control to the quality setting. And some may not allow any adjustment to this setting at all.

• Differences between how encoders interpret the quality setting. Different JPEG encoders will apply different meanings to the quality setting, so you can't compare quality settings between different encoders.

  For example, 80% quality in one encoder may choose similar average quantizer values to 92% quality in another encoder. The quality settings is only indirectly influencing the algorithm that the encoder uses to choose quantizer values for each block in the image. Under the hood, it's these quantizer values that actually affect the quality, and you can't usually adjust these directly.

  Even the "100%" quality setting can mean different things between encoders. In some it means to use the lowest quantizers available (preserving the most detail). In others it might not necessarily go that far, or it may (as mentioned above) affect other aspects of the encoding such as the chroma sub-sampling.

Note that the JPEG image will never be larger in size than the uncompressed bitmap image, for any normal image. However, comparing it to a RAW file can be a bit misleading since a RAW file would need several steps (including a step called demosaicing, which will increase size due to interpolation) to expand it out to a proper uncompressed bitmap image.

Answer 4

Different encoders use different values for the maximum quality. While JPEG is great at reducing file size by allowing images to be simplified in subtle ways, it's a one trick pony that doesn't look for patterns across the entire image that would allow for the size to be reduced losslessly.

The RAW file on the other hand is likely using a general purpose lossless compression that looks finds patterns that the JPEG encoder isn't finding. That, plus the overhead of the JPEG file format can result in a larger overall file than the original RAW potentially.

As far as different sizes for max quality between encoders, that's simply a result of the difference between values used when encoding "100%". It still isn't lossless and simplification still occurs, but the level of simplification can very from one program to another. Particularly at the higher qualities, relatively small amounts of quality gain can result in significantly more data being needed since you are adding on least significant bits of an already large number of values.