How do I batch apply sidecar files to the corresponding photos by filename in darktable?

Question

I have pretty much the opposite question to this one.

I recently needed to upgrade my computer. On my new computer, I still have all of the documents from my old computer, copied over in the same directory structure. This means that within my darktable working location, I have all of my original raw files, and all of my sidecar files (thankfully). Unfortunately, the fresh version of darktable doesn't automatically know that, for example, IMG_1234.CR2.xmp should be applied to IMG_1234.CR2. So none of my previous developing and editing work shows up anymore.

Obviously, I could manually apply each sidecar file to its corresponding image file to get my edits back, but across thousands of photos, that would be tedious and error prone. Is there a way that I can automate this by filename across my whole photo collection from within darktable?

---

What I tried

For what it's worth, I tried a couple things based on reading the online manual, but it didn't quite do it. From section 2.2.7 Sidecar files:

Once an image has been imported into darktable the database entries take precedence over the XMP file. Subsequent changes to the XMP file by any other software are not visible to darktable – any changes get overwritten the next time darktable synchronizes the file. This behavior can be changed in the preferences dialog (see Section 8.2, “Core options”). On request darktable looks for updated XMP files at startup and offers a choice whether to update the database or overwrite the XMP file.

... and from section 8.2 Core options:

LOOK FOR UPDATED XMP FILES ON STARTUP

Check file modification times of all XMP files on startup to find out if any got updated in the meantime by some other software. If updated XMP files are found a menu opens for the user to decide which of the XMP files to be reloaded – replacing darktable's database entries by the XMP file contents – and which of the XMP files to be overwritten by darktable's database. Activating this option also causes darktable to check for text sidecar files that have been added after import time – see option “overlay txt sidecar over zoomed images” in Section 8.1, “GUI options” (default off).

I tried selecting the "LOOK FOR UPDATED XMP FILES ON STARTUP" option, then restarting darktable, but this only worked for one photo. I'm not sure, but I think it's because the sidecar files are all from before the photos were imported into the new install, so darktable ignores them because it thinks they're old.

---

Kludgy last ditch option

I'd prefer to find some way native to darktable to get this done, but failing all else, I might try the following from my base darktable working directory (I'm in a Linux environment):

for i in $(find . -name "*.xmp")
do touch $i
done

After this, I'd try the "LOOK FOR UPDATED XMP FILES ON STARTUP" option again. This approach makes me vaguely nervous.

Answer 1

I think your suspicion that the date of the sidecar files plays a role is correct. The manual states (bolding mine):

Look for updated xmp files on startup

Check file modification times of all XMP files on startup to find out if any got updated in the meantime by some other software. If updated XMP files are found a menu opens for the user to decide which of the XMP files to be reloaded – replacing darktable's database entries by the XMP file contents – and which of the XMP files to be overwritten by darktable's database. Activating this option also causes darktable to check for text sidecar files that have been added after import time – see option “overlay txt sidecar over zoomed images” in Section 8.1, “GUI options” (default off).

So it seems you need to have XMP files that have a modification time that is later than the last database modification time.

Roughly the steps I would have in mind would be:

1. Create a copy of +/- 100 of your photos and XMP files into a separate folder
2. Import the copies into darktable
3. Set the "Look for updated xmp files on startup" options in darktable.
4. Close darktable.
5. Loop over your XMP files and touch them to update the modification time only (hence the -m option):

for file in $(find . -name "*.xmp")
do
    touch -m $file
done

6. Open darktable. I expect it now to replace the database entries by the XMP file contents.

Answer 2

Reimporting pictures will apply existing sidecar files.

Conversely, replacing the newly created database with the previous database is another option when file locations have not changed.

Darktable stores all your edits in the sidecar files. It stores metadata about the sidecar files in an SQLite database.

When Darktable opens it uses the SQLite database to generate the light table view. As Darktable runs it uses the SQLite database for sorting and searching. It updates the SQLite database when ever images are imported.

When upgrading computers or reorganizing the file system, I prefer to rebuild the SQLIte database by reimporting everything into a new empty database. This purges metadata about pictures in locations that no longer exist or that are irrelevant to my needs (for example duplicates in backup locations).