Does Lightroom actually rewrite the *ENTIRE* DNG file on every metadata change?

Question

When Lightroom saves metadata changes to a DNG file, does it:

A) Just rewrite the XMP/metadata portion of the DNG file to the existing file, i.e. a partial file update.

or

B) Overwrite the entire file to disk (in place).

or

C) Delete the existing file and write a whole new file to disk (with same filename obviously). (I'm not even sure if this is technically any different to B above)

In case it's relevant / makes a difference, I'm using Lightroom on Windows 8.1 64bit on a NTFS filesystem on a regular spinning HDD (not SSD). But also keen to know if the answer differs for other systems or SSDs.

Basically I'm considering using DNGs with the original RAW file embedded, so my DNG files would be about 30MB. Just wondering if that whole 30MB is going to be rewritten to disk every time I touch anything in Lightroom (and even when I don't actually change anything according to what some people have reported).

Answer 1

If the metadata is integrated into the main file then the main file is generally rewritten by the majority of operating systems/file systems in most computing systems used by photographers. I'm not aware of any current scenario which, even if a file system allows for the possibility of partial rewriting of files, actually occurs in such a way when using Lightroom.

If the metadata is stored in a separate sidecar file then only the sidecar file needs to be rewritten. With Lightroom the user has the option to use separate XMP sidecar files for storing metadata and the editing steps taken when working with an image file. The user also has the option to include the metadata inside the image file itself. There are advantages and disadvantages to either choice.

There are file systems that handle changes to files in a way that doesn't rewrite an entire file each time a file is changed, but they're not in very common usage by consumers who use their home computers to store photographs, or even by most professional photographers. If you use file systems such as ZFS or ReFS (that even theoretically allow for the possibility of partial rewriting of files) for storing photos you are in a very small minority.

As the number of users adopting Apple's new APFS (through upcoming hardware replacement) grows that may change over time in the future, and applications such as Lightroom might leverage such capability in the future. As of now most photographers aren't using such a file system on their computers, and even those who are don't gain anything with regard to partial file writing if the application, such as Lightroom, doesn't use the capability.

Answer 2

Lightroom does not rewrite the entire DNG file when saving metadata to the file, either via Cmd/Ctrl-S or when you have the "Automatically write changes into XMP" catalog setting enabled.

I verified this by monitoring Lightroom CC 2016.6.1 on OS X 10.11.6 under dtruss, then wrote a Perl script to analyze the collected data. The comments in that script explain the method, but if you only want to see a typical set of results, here you go:

Data I/O for file _1020153.RW2:
    256K read
Data I/O for file _1020153.xmp:
    6.87K read
    3.59K written
Data I/O for file _1020176.dng:
    3.38M read
    6.52K written

The two raw files named above were captured on the same day by the same camera, and the I/O comes from applying the same keyword to each file. The only difference is that one is the camera's native raw file (Panasonic RW2) and the other was converted by Lightroom to DNG before the test. The RW2 file is 18.7 MiB and the DNG file is 13.4 MiB. (The size difference is because DNG uses better lossless compression than is typical for native camera raw formats.)

As you can see, although neither update wrote out nearly enough data to rewrite the entire file, the DNG update involved about 13× as much reading and nearly 2× as much writing.

Also of interest is that Lightroom does more than just write to the XMP file in the native camera raw case. It also reads a substantial amount of data from the raw file, probably to assure itself that the raw file hasn't changed out from under it before it goes writing to the companion XMP file.

I would not take this single test result as representative if I were you, however. Try it on your own files. I wouldn't be surprised if you got different results.

Keep in mind that the full cost of these I/Os is only paid the first time you work on a given set of files in a given Lightroom session. If you apply a bunch of keywords in succession, the data reads in particular will be nearly instant, since the file contents will still be in your OS's buffer cache.

dtruss is not available for Windows, but there are equivalent alternatives. Adapting the Perl analysis script to work with the output of such tools should be straightforward.