Force Lightroom Classic to write entire image at once during export? (Exporting to network-mounted GCS bucket)

Question

I'm running out of space on my local machine so I've mounted a Google Cloud Storage bucket to my local machine using gcsfuse. I am now exporting my library from Lightroom Classic as JPEG files to the mounted GCS bucket, but the process is extremely slow, much slower than it should be if one is strictly accounting for encoding speed and time to upload the files to GCS.

I believe what is happening is that Lightroom is incrementally writing the encoded files to the filesystem. This is fine on a normal system, but with gcsfuse, a file modification includes reading the entire file from GCS, editing the file and writing the entire thing back to GCS. This means that each incremental write is taking orders of magnitudes more time than it would otherwise.

I've come up with a temp solution of exporting 50 files at a time to a local directory and then doing a mass move to the directory that consists of the mounted GCS bucket, but I'd like to make it so that I can export my entire library in one go from Lightroom to the mounted GCS bucket. To do this I need to change Lightroom to only write JPEG files after they've been entirely encoded, is there a way to do this?

Answer 1

It seems extremely unlikely that Lightroom would include a setting (whether obvious or secret) to deal with this obscure situation.

I think you're in a situation where the old joke applies:

Doctor! It hurts when I do this!

Well, don't do that.

You have a workaround. I'd focus on making that as painless as possible.

Answer 2

For anyone who wants to know how I made the workaround less involved, I wrote a python script to automatically monitor the local directory that Lightroom Classic is exporting to, and move all files (except the most recent which is possibly still being written) to the network drive as they are exported. This allows me to do a giant, unsupervised, export.

This script runs in the directory that the files are being exported to, and the files are being moved to the mounted directory which is at "/Users/me/gcs-buckets/my-bucket"

#!/usr/bin/env python

import os
import sys
import shutil
import time

def main():
    while True:
        files = os.listdir(".")
        files = filter(lambda fn: len(os.path.splitext(fn)) == 2 and os.path.splitext(fn)[1] == '.jpg', files)
        # files are of the format "jmt2019_part1-476.jpg", parse into
        # (476, "jmt2019_part1-476.jpg") so we can sort by the integer
        file_pairs = [(int(os.path.splitext(f)[0].split("-")[1]), f) for f in files]
        file_pairs = sorted(file_pairs, key=lambda t: t[0])
        #print file_pairs
        for _, f in file_pairs[:-1]:
            # for all but the most recent file as it's likely being written still
            new_path = os.path.join("/Users/me/gcs-buckets/my-bucket", f)
            print "moving file", f, "to new path at", new_path
            before = time.time()
            shutil.move(f, new_path)
            after = time.time() - before
            print "moved file", f, "to remote storage in", after, "seconds!"
        print "waiting 10 seconds!"
        time.sleep(10)

if __name__ == '__main__':
    sys.exit(not main())