0
\$\begingroup\$

I'm running out of space on my local machine so I've mounted a Google Cloud Storage bucket to my local machine using gcsfuse. I am now exporting my library from Lightroom Classic as JPEG files to the mounted GCS bucket, but the process is extremely slow, much slower than it should be if one is strictly accounting for encoding speed and time to upload the files to GCS.

I believe what is happening is that Lightroom is incrementally writing the encoded files to the filesystem. This is fine on a normal system, but with gcsfuse, a file modification includes reading the entire file from GCS, editing the file and writing the entire thing back to GCS. This means that each incremental write is taking orders of magnitudes more time than it would otherwise.

I've come up with a temp solution of exporting 50 files at a time to a local directory and then doing a mass move to the directory that consists of the mounted GCS bucket, but I'd like to make it so that I can export my entire library in one go from Lightroom to the mounted GCS bucket. To do this I need to change Lightroom to only write JPEG files after they've been entirely encoded, is there a way to do this?

\$\endgroup\$
8
  • \$\begingroup\$ But what? Seems like the question is unfinished. \$\endgroup\$
    – mattdm
    Aug 20, 2019 at 19:56
  • \$\begingroup\$ sorry about that, the question got cut off for some reason. It's all there now \$\endgroup\$ Aug 20, 2019 at 20:19
  • 1
    \$\begingroup\$ why do you believe lightroom is doing incremental writes? what evidence do you have? Is your lightroom catalog being backed up because that would do it lol \$\endgroup\$ Aug 20, 2019 at 20:36
  • \$\begingroup\$ When I do an ls -l of the mounted partition it shows the files being slowly increased in size as if they are being written incrementally. I also am able to show that exporting to a local directory and then moving those local files to the network-mounted partition is net faster than just exporting directly to the externally mounted partition. This makes me believe that the files are being written incrementally. \$\endgroup\$ Aug 20, 2019 at 21:45
  • 1
    \$\begingroup\$ I'm voting to close this question as off-topic because it only nominally involves photographs as computer files and doesn't have much of anything to do with photography itself. \$\endgroup\$
    – Michael C
    Aug 20, 2019 at 23:26

2 Answers 2

4
\$\begingroup\$

It seems extremely unlikely that Lightroom would include a setting (whether obvious or secret) to deal with this obscure situation.

I think you're in a situation where the old joke applies:

Doctor! It hurts when I do this!

Well, don't do that.

You have a workaround. I'd focus on making that as painless as possible.

\$\endgroup\$
3
  • \$\begingroup\$ Fair enough - it just makes it significantly more involved on my half as I need to intervene every 20 minutes or so to automate the move to the mounted partition and then start the next export, I'll just deal with that for now. You're right that it would bee very odd for adobe to account for this edge case. \$\endgroup\$ Aug 20, 2019 at 21:47
  • \$\begingroup\$ Why intervene manually? You could definitely script something, but that'd be pretty far off topic for this site. \$\endgroup\$
    – mattdm
    Aug 20, 2019 at 22:19
  • \$\begingroup\$ Oh, I see your other answer now. Yeah, something like that. :) \$\endgroup\$
    – mattdm
    Aug 20, 2019 at 22:20
0
\$\begingroup\$

For anyone who wants to know how I made the workaround less involved, I wrote a python script to automatically monitor the local directory that Lightroom Classic is exporting to, and move all files (except the most recent which is possibly still being written) to the network drive as they are exported. This allows me to do a giant, unsupervised, export.

This script runs in the directory that the files are being exported to, and the files are being moved to the mounted directory which is at "/Users/me/gcs-buckets/my-bucket"

#!/usr/bin/env python

import os
import sys
import shutil
import time

def main():
    while True:
        files = os.listdir(".")
        files = filter(lambda fn: len(os.path.splitext(fn)) == 2 and os.path.splitext(fn)[1] == '.jpg', files)
        # files are of the format "jmt2019_part1-476.jpg", parse into
        # (476, "jmt2019_part1-476.jpg") so we can sort by the integer
        file_pairs = [(int(os.path.splitext(f)[0].split("-")[1]), f) for f in files]
        file_pairs = sorted(file_pairs, key=lambda t: t[0])
        #print file_pairs
        for _, f in file_pairs[:-1]:
            # for all but the most recent file as it's likely being written still
            new_path = os.path.join("/Users/me/gcs-buckets/my-bucket", f)
            print "moving file", f, "to new path at", new_path
            before = time.time()
            shutil.move(f, new_path)
            after = time.time() - before
            print "moved file", f, "to remote storage in", after, "seconds!"
        print "waiting 10 seconds!"
        time.sleep(10)

if __name__ == '__main__':
    sys.exit(not main())
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.