1
\$\begingroup\$

I'm using darktable-cli to export photos from my library. I started to get an error with the latest version(2.6.1). It's a segmentation fault with error 

[init] the database lock file contains a pid that seems to be alive in your system: 16768
[init] database is locked, probably another process is already using it

I'm always getting this error if darktable is running. And if darktable is not running, most of the times. I have a script running 4 instances darktable-cli in parallel. As I mentioned I didn't have this error with the previous versions.

I don't know how darktable works internally, but I feel like there shouldn't be a need for library for the cli. Is there a need for it?

As a workaroung I tried changing the library for the darktable calls. But --library option is ignored. Darktable still worked with the default library. The command is:

darktable-cli raw-file xmp-file jpg-file --width 1920 --height 1080 --core --conf plugins/imageio/format/jpeg/quality=50 --conf plugins/imageio/storage/disk/overwrite=true --library /tmp/darktable.db

Is this command argument correct?

Improve this question
\$\endgroup\$
0

3 Answers 3

Reset to default
4
\$\begingroup\$

I'm not using Darktable myself, so I can't check it myself but this might be a re-occurrence of a bug that has been fixed two years ago. You might file a bug-report with them and while it's been discussed/fixed, you can revert back to the previous version to keep your stuff functional.

Edit: Checking the documentation you might try to call darktable-cli explicitly setting the library to :memory:, so that it looks something like this:

darktable-cli [...] --library ':memory:'
Improve this answer
\$\endgroup\$
3
  • \$\begingroup\$ --conf --library ':memory:' didn't work neither \$\endgroup\$
    – rosencreuz
    Mar 25, 2019 at 6:30
  • \$\begingroup\$ @rosencreuz Shouldn't that be --core --library ':memory:'? What's the error message? \$\endgroup\$
    – Lothar
    Mar 25, 2019 at 7:45
  • \$\begingroup\$ --core --library ':memory:' also doesn't work. Same error. Very long error message starting with [init] the database lock file contains a pid that seems to be alive in your system: 1915 [init] database is locked, probably another process is already using it and last line is segmentation fault (core dumped) darktable-cli test.jpg --width 1920 --height 1080 --core. \$\endgroup\$
    – rosencreuz
    Mar 26, 2019 at 17:30
3
\$\begingroup\$ 
darktable-cli $src $xmp $export [options] --core --library ':memory:' --configdir '/dev/shm/$uniquevalue'

It seems that there's also a data.db in the --configdir parameter (default: $HOME/.config/darktable) which also gets locked by a darktable session. This probably holds user preferences and styles, but if an xmp sidecar is specified then it's not important. (One could probably copy the data.db into a unique instance if necessary.)

Here's a snippet using GNU parallel:

parallel "darktable-cli {} somesettings.xmp export/{/} [opts] --core --library ':memory:' --configdir '/dev/shm/{/.}/' ; rm -rf /dev/shm/{/.}" ::: *.jpg

Darktable will create a new data.db in a dir with the source filename exclusively for that darktable-cli session and remove it after exporting.

Notes for this example:

  • My source images are jpgs. Change the *.jpg to a glob matching your sources (or pipe it to parallel).
  • {/.} strips out dirpath and extensions from files, so that can be used to specify output file type
  • I'm creating the temporary configdir in memory (/dev/shm) for speed; each instance is ~152KB
  • --configdir ':memory:' creates a literal :memory: directory, so doesn't have the same magic as --library :)

Hope this helps, I feel like this is just a workaround and that darktable-cli should probably not lock or open the data.db as read-only.

Improve this answer
\$\endgroup\$
3
  • \$\begingroup\$ Some questions: (1) what does {/} without the . do? (2) what does the ::: do? \$\endgroup\$
    – juhist
    Jul 7, 2019 at 6:49
  • \$\begingroup\$ {/} Basename of input line. ::: arguments Use arguments from the command line instead of stdin (standard input). And I want to thank you @anomalyconcept for the elegant workaround, I have a script to parallel process raw files from sd card (copy, backup to external storage, develop using darktable, then create smaller "web" versions) and that darktable bug/feature(?) introduced recently forced my parallel invocation to have --max-procs 1 for a while. Was checking again to see if they fixed it and found this answer. Already using it while the bug is fixed, back to usual performance. \$\endgroup\$
    – Jorge Nerín
    Aug 3, 2019 at 18:08
  • \$\begingroup\$ Thank you! The dedicated --configdir directory was the right trick.With my 12-thread CPU, exporting 36 images that take 84 s. single-threaded are done in 45 s. with 4 threads. \$\endgroup\$
    – miguev
    Oct 13, 2019 at 11:02
2
\$\begingroup\$

Unfortunately, you can't run multiple instances of darktable at a time.

If your script is running 4 instances of darktable-cli in parallel, there's your error. You need to serialize your script to run at most one darktable-cli at a time. Having some kind of file locking mechanism that waits for the old darktable-cli to exit is needed. An example of such a program is flock.

For example, flock /some/where/somefile.txt darktable-cli raw-file xmp-file jpg-file --width 1920 --height 1080 --core --conf plugins/imageio/format/jpeg/quality=50 --conf plugins/imageio/storage/disk/overwrite=true --library /tmp/darktable.db ought to work.

Improve this answer
\$\endgroup\$
1
  • 1
    \$\begingroup\$ Fortunately, you can run multiple instances of darktable-cli at a time! Just need to use a dedicated --configdir directory for each instance, and them clean-up later. This allows me to run 4 instances of darktable-cli in parallel, get the most out of my 12-thread CPU and but export times to about half. Exporting 36 images that take 84 s. (single-threaded) are done in 53 s. with 2 threads, 45 s. with 4 threads, 53 s. with 6 threads ─ there's a balance to be found depending on your CPU and I/O bandwidth. \$\endgroup\$
    – miguev
    Oct 13, 2019 at 10:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.