10

Is there some way of saving (basic) non-destructive RAW developing/adjustments that is not limited to a particular app ecosystem (eg Adobe)? Understandably some transformations are proprietary (eg. some cool denoise algorithm, or a spot removal tool) but others are rather generic that I think could easily be standardized (eg. crop, exposure, white balance) and be read by other applications.

At first I thought sidecar xmp files could be useful in that respect, since Camera RAW/Lightroom and darktable save their adjustments inside such files, but seems that adjustments created by Camera RAW/Lightroom can not be previewed/edited by darktable and vice versa.

Any solution that has worked for you?

This question is mostly about creating a futureproof way of storing the original camera data, but also the edit history (adjustments in a non-destructive way)

3

They're all proprietary. Each application uses their own algorithms to convert the raw data. To the best of my knowledge none of the adjustments made with one application will translate identically in another application unless both applications use the same raw conversion engine "under the hood" (e.g. Adobe Lightroom and Adobe Photoshop both use Adobe Camera Raw to do the actual raw conversion). So you have to choose only one application to do raw conversion.

If you then want to use other tools to further edit the image you must export the results of the raw conversion in a standardized raster image format that can be edited by the other tools. Probably the most common way to deal with this is to do the raw conversion and then export the images as 16-bit tiff files.

Of course converting to tiff locks in a lot of the decisions made in raw conversion: black point, white point, gamma correction, etc. and reduces the amount of further adjustment to things such as color temperature/white balance, etc. But the 16-bit tiff has a lot more information than, say, an 8-bit jpeg. The maximum number of gradations in an 8-bit color value are 256. The maximum number of gradations in a 16-bit color value are 65,536. In addition to the lower bit depth there is also the consideration of compression. So the tiff file will allow a lot more adjustment before things start to fall apart than a jpeg will. The biggest disadvantage of 16-bit tiff files is their size: A raw file from a 20MP camera will run around 24-30MB. The exact size depends on content. Most raw files are compressed using lossless compression so the more variation there is in the scene the larger the file size will be. The same 20MP image converted to a 16-bit tiff file will run around 100MB uncompressed.


From the comments:

After posting the question last night, I discovered that darktable supposedly (and to some extent) can read the adjustments inside adobe's xmp files - for what it's worth, here and here are the links. I haven't managed to make it work thus far though.

Even when the instructions can be read properly the question still remains, is darktable applying the exact same algorithm to get the exact same result? Or is it just using the darktable algorithms to approximate the results from a certain setting using ACR? My hunch is that you could get the exact same results by editing in darktable to begin with as you could get by translating the instructions from ACR using darktable. You just have to learn how to use the different GUI to get there.

Indeed @MichaelClark, one could use LR, Br or darktable without needing to use another! I was more interested in finding a workflow, that is futureproof - in the sense that I won't be locked down in one particular application/ecosystem/OS.

In that case always save your original raw files. That's about as future proof as it gets. People like to make a lot of noise about a "standard" raw format, but it's just a bunch of smoke and mirrors. DNG is no more future proof than .cr2 or .nef. And more applications let you convert .cr2 and .nef files than let you work with .dng files.

Nothing is future proof. Absolutely nothing.

  • 1
    Do you understand what black point is? All values in the sensor readout data beneath the selected black point are normalized at zero. There is no longer any distinction between those values that were once different. Thus, that information is irretrievably gone. Same thing with white point. All values above the white point are normalized at the maximum. Any distinction between them in the sensor readout is thrown away. – Michael C Jan 13 '17 at 7:47
  • 1
    Fully in agreement, but just some additional opinion - as I'm still pretty new to this & can't provide a full second answer, but I've found my 'best' method is to do my initial colour editing in Nikon's own software, then export tiff & do the rest in photoshop. Opening directly in Adobe is, imo, nowhere near as good a result. I do find, however, that the tiff file sizes are more like 180MB than 100, for a 24mp image – Tetsujin Jan 13 '17 at 8:17
  • 2
    Thank you @Michael Clark, and everybody for the comments! TIFF is what I would call a "baked" format. Beyond baked-in parameters like exposure/clipping/demosaicing and noise-reduction that depend a lot on the raw-developer-software interpretation, there are also more drastic transformations, like Masks, rotation and cropping that are being destructively applied. While I would definitely use tiff for archival purposes of a final photograph, I don't feel using it for anything else. – FotisK Jan 13 '17 at 13:37
  • 1
    @FotisK Even when the instructions can be read properly the question still remains, is darktable applying the exact same algorithm to get the exact same result? Or is it just using the darktable algorithms to approximate the results from a certain setting using ACR? – Michael C Jan 13 '17 at 17:02
  • 2
    @FotisK In that case always save your original raw files. That's about as future proof as it gets. People like to make a lot of noise about a "standard" raw format, but it's a bunch of smoke and mirrors. DNG is no more future proof than .cr2 or .nef. And more applications let you convert .cr2 and .nef files than let you work with .dng files. – Michael C Jan 14 '17 at 2:24
2

After a few days of research, I've concluded that at the time of writing there is no holy grail. But as for futureproofing I'd go the following route:

  • keep the original RAW file
  • work with Adobe products for the adjustments (either inside Lightroom or with sidecar xmp files produced by adobe camera RAW)

This probably is as futureproof and cross-platform as it gets. Which is, not a lot, but not totally bleak either.

As a second layer of future-proofing, I would retain

  • a 16-bit TIFF of the original RAW (in case a day comes that nobody wants to create a RAW decoder for my particular camera)
  • a high quality jpg export of every variant/version of a particular image (in case I need reference to reproduce this exact look)
  • and a 16bit TIFF of all the final products (eg. Edits that made it in print, exhibitions, or in general were used someplace beyond my light table)

DNG Files? No thanks

While I have not looked deeply into DNG files (so please correct me if I'm mistaken somewhere) I understand that

  1. they cannot replace RAW files, because

    a. there is a level of interpretation of the sensor data, which means they are already slightly away from the original RAW

    b. It's not one universal file format; it's a container. Which means that in the distant future someone will have to write software that reads the particular flavor of DNG that your camera files got converted into. Not much different fate to that of RAW files, is it?

  2. DNG files are probably not any better at future-proofing the adjustments:

    a. the xmp adjustments embedded in the DNG file, are not any more universal than the adjustments inside the sidecar xmp. They are still software-specific. On the contrary, I've seen a few 3rd party programs (see below) that read ACR/Lightroom adjustments from .xmp files, but I'm not aware of any that can read them out of DNG files.

Reasoning behind choosing Lightroom and Camera Raw

Futureproofing

Adobe products have become pretty much the industry standard and as such:

  1. Competitors will have to lure existing Adobe customers into their own ecosystems. A few are already attempting reproducing (subset of) the adjustments that ACR stores inside the .xmp files. For instance the open source project Darktable is claiming accurate reproduction of crop,rotation,flip and tags, and mostly accurate reproduction of exposure/blacks, grain, tone curve (only lightness supported), color zones local contrast - as seen here and here. Others imply that you can migrate whole lightroom libraries including the adjustments (Capture One Pro claims that it "allows import of other 3rd party application catalogs to make migration easier than ever, maintaining folder structures and most adjustments on your images" accompanied by a picture showing Aperture, Lightroom and Media Pro as seen here. On1 Photo RAW supposedly will be adding support for that in March 2017 - for more read here, here and the timeline here). While these attempts can never be 100% accurate, they are accurate enough to give a starting point for further image manipulation and/or at the very least underline an interest by 3rd parties to figuratively-speaking reverse engineer Lightroom's and ACRs ways of storing adjustments.
  2. For better or for worse, Adobe is rather successful so this means that there is no reason in the foreseeable future to go out of business
  3. Even then, due to the large Adobe user base, it's quite possible that future engineers will probably be interested in recovering Lightroom and ACR generated data.

Cross-platform

While adobe products do not run on Linux, they run on macOS, Windows, Android and iOS. As of July 2017 Lightroom mobile, supports raw editing (read here). For linux, one could use darkroom which however is a one way trip since Darkroom does not sync it's own adjustments back in adobe's .xmp format.

1

The DNG file format will allow saving adjustment information directly in the file, rather than a sidecar file, and the DNG format is a documented specification that is free for anyone to use. Some cameras support DNG natively as a RAW format, for others, you can use the free Adobe DNG converter. Within the Adobe ecosystem, images edited with the version of ACR provided in even an old version Photoshop Elements (7) will tranfer to the latest Lightroom (CC) with all the adjustments intact.

Unfortunately, I've tested DNG files edited with Lightroom, in Corel Aftershot, and Raw Therapee and neither appear to be able to recognise the adjustments, which is not surprising given that the adjustment parameters the different programs use are quite different, even if they do similar things.

I haven't tested On1 Photo RAW or Cature One Pro, but I'd imagine the results will be similar.

  • 2
    See the answers (and especially the comments to the answers) to Do I lose anything converting to DNG? for issues about whether or not information is lost in the conversion to DNG. Enlightening conversation. – scottbb Jan 13 '17 at 23:32

protected by Community Jan 18 '17 at 0:42

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.