2

When Lightroom saves metadata changes to a DNG file, does it:

A) Just rewrite the XMP/metadata portion of the DNG file to the existing file, i.e. a partial file update.

or

B) Overwrite the entire file to disk (in place).

or

C) Delete the existing file and write a whole new file to disk (with same filename obviously). (I'm not even sure if this is technically any different to B above)

In case it's relevant / makes a difference, I'm using Lightroom on Windows 8.1 64bit on a NTFS filesystem on a regular spinning HDD (not SSD). But also keen to know if the answer differs for other systems or SSDs.

Basically I'm considering using DNGs with the original RAW file embedded, so my DNG files would be about 30MB. Just wondering if that whole 30MB is going to be rewritten to disk every time I touch anything in Lightroom (and even when I don't actually change anything according to what some people have reported).

  • It seems a very good argument for having separate standalone XMP files, in which case, only the small XMP file gets rewritten with change, and the raw image file is never rewritten. I don't know if XMP is combined into the image file. It is technically possible to rewrite only the affected disk sectors, but would guess it is much easier that the entire file may be rewritten? – WayneF Sep 2 '16 at 17:10
  • As an aside I think that SSD's are now to the point that under normal usage you are not going to exhaust their write capability. That said, they do have different failure modes to spinning disks which means you should still have your backups. – Peter M Sep 2 '16 at 19:52
  • 1
    I had noticed abysmal performance when doing updates to a larger number of files using dng. Writing only to xmp was a huge improvement. – ths Sep 2 '16 at 20:23
1

If the metadata is integrated into the main file then the main file is generally rewritten by the majority of operating systems/file systems in most computing systems used by photographers. I'm not aware of any current scenario which, even if a file system allows for the possibility of partial rewriting of files, actually occurs in such a way when using Lightroom.

If the metadata is stored in a separate sidecar file then only the sidecar file needs to be rewritten. With Lightroom the user has the option to use separate XMP sidecar files for storing metadata and the editing steps taken when working with an image file. The user also has the option to include the metadata inside the image file itself. There are advantages and disadvantages to either choice.

There are file systems that handle changes to files in a way that doesn't rewrite an entire file each time a file is changed, but they're not in very common usage by consumers who use their home computers to store photographs, or even by most professional photographers. If you use file systems such as ZFS or ReFS (that even theoretically allow for the possibility of partial rewriting of files) for storing photos you are in a very small minority.

As the number of users adopting Apple's new APFS (through upcoming hardware replacement) grows that may change over time in the future, and applications such as Lightroom might leverage such capability in the future. As of now most photographers aren't using such a file system on their computers, and even those who are don't gain anything with regard to partial file writing if the application, such as Lightroom, doesn't use the capability.

  • 2
    I haven't tried it with APFS yet (I avoid beta OSes), and I don't use Lightroom on Windows Server (the only way to try ReFS) but I can tell you that your statement about ZFS is wrong, and I doubt it's correct for APFS or ReFS. I ran Lightroom CC 2015.mumble on OS X 10.11 with O3X ZFS a few months ago under truss and watched it write out the entire file on every metadata update. I think you're getting confused by the CoW feature of these file systems, which merely avoids a complete file rewrite when the app touches part of the file only; if the app doesn't play ball, the FS can't save you. – Warren Young Sep 3 '16 at 4:50
  • I'm not sure the second paragraph implies at all what any particular application does, only that some of the cutting edge file systems allow for the possibility of leveraging such capabilities in the future that may change the current reality which is the primary gist of the answer.. – Michael C Sep 3 '16 at 18:05
  • @WarrenYoung Your comments to this answer seem to be arguing the exact opposite of your comments here: photo.stackexchange.com/a/79564/15871 – Michael C Sep 3 '16 at 18:13
  • Re: conflicting comments, this is exactly what I meant when I said that I think you're confused. My comment on your other answer is only addressing the data safety issue you brought up; CoW doesn't affect the application's I/O behavior, it only tells you what the filesystem does on disk as a result of the application's I/O. Most any tutorial on ZFS will explain this. Here's a quick summary of ZFS. – Warren Young Sep 4 '16 at 4:18
  • 1
    Re: My comment above about the "entire file" being rewritten, I've just repeated my tests, and this time I get different results. I suspect I was just sloppy in my original test, rather than the behavior of Lightroom changing since the last test. The main thrust of my comment above stands, however, which is that the results will be independent of the filesystem. CoW doesn't silently change application behavior to turn whole-file rewrites into partial-file rewrites. It only affects the safety of doing so. – Warren Young Sep 4 '16 at 4:22
0

Lightroom does not rewrite the entire DNG file when saving metadata to the file, either via Cmd/Ctrl-S or when you have the "Automatically write changes into XMP" catalog setting enabled.

I verified this by monitoring Lightroom CC 2016.6.1 on OS X 10.11.6 under dtruss, then wrote a Perl script to analyze the collected data. The comments in that script explain the method, but if you only want to see a typical set of results, here you go:

Data I/O for file _1020153.RW2:
    256K read
Data I/O for file _1020153.xmp:
    6.87K read
    3.59K written
Data I/O for file _1020176.dng:
    3.38M read
    6.52K written

The two raw files named above were captured on the same day by the same camera, and the I/O comes from applying the same keyword to each file. The only difference is that one is the camera's native raw file (Panasonic RW2) and the other was converted by Lightroom to DNG before the test. The RW2 file is 18.7 MiB and the DNG file is 13.4 MiB. (The size difference is because DNG uses better lossless compression than is typical for native camera raw formats.)

As you can see, although neither update wrote out nearly enough data to rewrite the entire file, the DNG update involved about 13× as much reading and nearly 2× as much writing.

Also of interest is that Lightroom does more than just write to the XMP file in the native camera raw case. It also reads a substantial amount of data from the raw file, probably to assure itself that the raw file hasn't changed out from under it before it goes writing to the companion XMP file.

I would not take this single test result as representative if I were you, however. Try it on your own files. I wouldn't be surprised if you got different results.

Keep in mind that the full cost of these I/Os is only paid the first time you work on a given set of files in a given Lightroom session. If you apply a bunch of keywords in succession, the data reads in particular will be nearly instant, since the file contents will still be in your OS's buffer cache.

dtruss is not available for Windows, but there are equivalent alternatives. Adapting the Perl analysis script to work with the output of such tools should be straightforward.

  • The difference in compression methods is the major contributing factor, but the size difference may also be partly due to the way Adobe products, including the DNG convertor, discard information in raw files that Adobe products don't use. – Michael C Sep 4 '16 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.