How to reorganize photos along with sidecar files?

Question

I'm using darktable 3.6.0 and am still struggling to find a good workflow. I originally started out with a folder structure where my photos were grouped by "event", but this proved limiting. Instead, I want to switch to a folder structure by date, like yyyy/mm/dd/*.CR2.

To use this new structure during import from camera, I'm using gphoto2 from the command line:

gphoto2 --get-all-files --skip-existing --filename '%Y/%m/%d/%f.%C'

Subsequently I import the new photos into darktable. That all works fine.

But how do I convert my existing library to this structure, while preserving the sidecar files (*.XMP) containing my edits and metadata?

I tried to use the new copy & import feature in darktable, again with $(YEAR)/$(MONTH)/$(DAY) folder naming pattern, but this has two problems. One, sidecar files are not imported but instead created anew. And two, everything ends up in a folder with today's date.

Since gphoto2 apparently knows how to read metadata, I tried to use it to import from a folder rather than a camera, but it doesn't seem to have an option for that. (Am I overlooking something?) But in any case, it would probably ignore sidecar files.

I could come up with a shell script using exiftool to read metadata and move files around accordingly, but surely there is a simpler way? In Lightroom this would have been trivial...

Answer 1

Here's the shell script I came up with:

#!/bin/bash

set -e

in_dir=$1

for f in $(find $in_dir -iname \*.cr2 -or -iname \*.jpg | sort); do
  out_dir=$(exiftool -DateTimeOriginal $f | awk '{print $4}' | sed 's#:#/#g')
  mkdir -p $out_dir
  mv -v $f $out_dir/$(basename $f)
  mv -v $f.xmp $out_dir/$(basename $f).xmp || true
done

rmdir $in_dir

It's invoked one directory at a time, and will crash if it encounters anything unusual. It also doesn't work if there are spaces in any filename. So it's not great, but it solves my problem for now... better answers are still welcome, if not for me, then for the next person who comes along.