The Sleeping Giant's Sea Lion

by Jakub

submit your photo


Hall of Fame
View past winners from this year

Please participate in Meta
and help us grow.

Take the 2-minute tour ×
Photography Stack Exchange is a question and answer site for professional, enthusiast and amateur photographers. It's 100% free, no registration required.

What is the best way to synchronize Adobe Lightroom databases between two computers (a desktop and a laptop)?

I'd like to be able to edit photos on either machine, and preferably not copy the whole 3+ gb database on each sync.

share|improve this question
4  
Are you asking about Lightroom or Aperture? –  Fredrik Mörk Jul 28 '10 at 6:58
    
I've doing this for a while now, without the drawbacks listed below. I'll talk about this tool, but a quick question first: is your LR catalog 3GB (i.e. the .lrcat file itself, and how many photos is that?), or did you add the size of your preview folder in this estimate, or more catalogs, or even your photos? Thanks. –  sebastien.b Sep 1 '10 at 14:18
    
@sebastien.b The 3GB size is for everything in Lightroom data folder, including the previews (but not my photos). Can I get away with just copying the .lrcat file? I thought data about edited images would be in the .lrdata (previews) folder. –  dbkk Sep 5 '10 at 17:23
    
@dkk OK, thanks. Yes, I've a solution for you, I've been using it for 5 years on small/large files, and for 2 years since I started using LR. Hard to install, but a breeze to use once it's set up, I'll post details soon. No, edited images are not in the previews folder. –  sebastien.b Sep 7 '10 at 22:40
1  
The link above covers most, if not all of this. If you have additional questions that the previous question doesn't answer, you can open up a new question to address those. The ones on software development are probably off topic here though. Thanks for the question though and welcome to photo stack exchange! On a related note, the way that I sync is much more simple. I write my changes to XMP sidecar files on my laptop, and just transfer those to my desktop at home. –  dpollitt Feb 29 '12 at 14:25

7 Answers 7

I've been synchronizing collections spanning up to 60GB and ten thousands of small and large files for years now. This includes my LR catalogs and pictures since I started photography. I'll be honest though, this is not a tool a lot of people use, it is not trivial to setup but it will solve your problem. Once it is up and running, it is very easy to work with and it will perform its duty quickly and effortlessly. As a software engineer, this is one of the top 3 pieces of software I use daily and can't live without.

The tool is Unison File Synchronizer.

Unison

Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.

  • It's free, run on both Windows and Unix, and even works across platforms.
  • It can deal with updates to both replicas of a distributed directory structure.
  • Updates that do not conflict are propagated automatically.
  • Conflicting updates are detected and displayed (more about that below).
  • Unison is resilient to failure. It is careful to leave the replicas and its own private structures in a sensible state at all times.
  • It works between any pair of machines connected to the internet. It is careful with network bandwidth. Transfers of small updates to large files are optimized using a compression protocol similar to rsync.

Large files

This later point is relevant to your problem. If a large file has been updated (say, your LR catalog), it will not copy the whole file over. It will instead transmit a smart, compressed subset (a delta) representing the difference between both files. For example, if I modify the Artist name on a MP3 or add some keywords to a JPEG file, it will not send the whole song or photo over, but only a very small portion of it. Granted, bigger changes happen in your LR catalog, but never quite big enough that they require to send the whole file over. This makes sense because your catalog is a SQLite database (a SQL db on disk); for performance reasons it is in the database engine's best interest not to shuffle every bits around.

Renaming

Unison has no problem renaming files and directories. Files are not identified by names actually, but by a unique signature. This is very convenient.

Compared to Microsoft Live Sync

Similar to Microsoft Live Sync, Unison does not keep an online copy of your files (thank God), which means both machines need to be connected to the internet.

Unlike Microsoft Live Sync (from what I understand), it does display potential conflicts and its user interface will let you decide which replica should "win". That is, if you made changes on both sides, it will let you mark if the 'left' side should have priority over the 'right' side (see screenshot below), or vice versa. You can do so file by file, replica by replica, on a subset of files, etc. You can even, for text files, have Unison launch a tool that will let you resolve the conflict manually at the contents level (say, using WinMerge).

Thumbnails and Previews

Now what about thumbnails and previews? Do not synchronize them. You do not need to, I never did, and LR is smart enough to regenerate a preview if it's missing. This has never been a problem for me in 2 years.

Besides your RAW and JPEG files of course, which piece of data should you synchronize to make sure you can work with LR on both sides? There are 3 components to consider, to my knowledge:

  • Your LR catalog file (Lightroom 3 Catalog.lrcat), which is where LR stores its metadata, keywords, rating, labels and develop settings. Remember, LR is non-destructive, it doesn't touch your original files but store every single change you made to a photo in its catalog instead (unless you instruct LR to dump these settings back to your XMP/DNG/PSD files, but that's another story and not a problem anyway). Your catalog is the important file to sync.
  • Your Previews directory (Lightroom 3 Catalog Previews.lrdata), which is where LR stores intermediate representation of your photos for faster viewing (1:1, low res, thumbnails, etc). Do not synchronize this folder.
  • Your Adobe Camera RAW Cache (ACR Cache). Every time you open an image in Camera Raw the full resolution of the image must be loaded into Camera Raw. As you can imagine, this can be pretty processor intensive. The Camera Raw cache will cache recently opened images to make re-opening them faster. This cache is actually used by LR and Photoshop if you have both. Do not synchronize this folder either, its content will be regenerated automatically.

You can change the default location of your catalog, previews and Camera RAW Cache from the Preferences. This is how I moved my catalog to a specific user directory (my HOME directory, with personal data), and this is the very directory I sync. You don't have to do that, but it makes my life easier.

Installation

As I mentioned earlier, installing Unison is tricky, check the user manual online, and comment if you are still in the blue. It does involve running a SSH server on one of the computer (if you sync between two), traditionally using Cygwin or a subset of it. This is the tricky part, but SSH is an open, robust, safe and secure solution. I trust this set up more than running Microsoft Live Sync or Dropbox, which are essentially client/servers too.

Once your SSH server is running, you are pretty much all set. You only need to write a small text file called a 'profile' that will tell Unison which folders need to be sync'ed. I realize it's not very user friendly, but it does have a powerful syntax that lets you ignore files using regular expressions or simple patterns. Here is what the top of a profile looks like, for instance:

root = d:\users\barre
root = ssh://barre@foo.net/d:\mystuff

The first line specifies where my local replica is, and the second where my remote replica can be found, on the foo.net (fictitious) machine at home for example, in the mystuff directory. This is even simpler if you stay within a local network. I use Unison to sync between 4 machines at 3 different sites.

ignore = Path {Pictures/Lightroom/Lightroom 3 Catalog Previews.lrdata}
ignore = Regex Pictures/Lightroom/.*\.lock
ignore = Regex Pictures/Lightroom/.*-journal

These 3 lines instruct Unison to ignore my LR thumbnails/previews, as well as any temporary lock/journal files found in the Pictures/Lightroom sub-directory (relative to the root directory specified above).

And that's about it.

Running Unison

The first time you run Unison takes quite a bit as it computes hash signatures for all the files under its new jurisdiction. This is key to help computing differences quickly. The next time you run it, it's astonishingly fast. The user interface is a bit rough because Unison is an exotic beast written in OCaml + GTK. Just pick your profile from a list and Unison will check your local replica for changes. At the same time it will call itself through SSH on the other replica to compute remote changes. Wait a few seconds and it will display a list of changes and potential conflicts to help you reconcile the two, if needed. If there is nothing to resolve, just click "Go" and sync happens automagically.

In this screenshot below I’m synchronizing my local machine (first column), with my remote machine named KANEDA. The second column, in between, is what Unison thinks should be done.

alt text

As you can tell:

  • I deleted one file on my local machine (First Friday…jpg), and the change will be propagated from left to right (the arrow), thus deleting the file on the remote.
  • I changed both my LR catalog and a DNG on the remote. The arrow is pointing from right to left, i.e. these changes will be propagated to my local replica.
  • I changed my .emacs.el file on both replicas, in a way that can’t be merged. Unison tells me at the bottom when the modification happened on both sides, the files size, etc. The question mark in the middle denotes that it’s up to me to make a choice. This line is selected, so I can just press the left or right arrow key to decide which side should win. I can even override any other row.

This situation doesn’t happen very often to me anymore, 99% of the time I don’t have any choice to make, I just review the situation quickly, press “Go”, and Unison synchronizes both replicas.

In Conclusion

I've used this tool for years and I swear by it. I synchronize and backup music, photos, documents and personal data using Unison, a task that would be impossible to do efficiently manually. It supports very large collections and thousands and thousands of file. It handles renaming, smart sync of large files and works quickly. It never corrupted a single one of my files.

This is still a file-level synchronization tool. It is not aware of the subtleties of your catalog, only Lightroom does. If you made changes to your catalog on both sides, you will have to make a choice between the two. At least the whole file is not copied over, and the user interface makes it easy to tell Unison which one should win. As somebody pointed out in this post, a bit of discipline should take care of this problem. Your catalog is one complex file. Even if a general-purpose tool used high-level SQL commands to inspect your catalog’s database and try to synchronize it with another, I still wouldn’t trust it over Lightroom.

Good luck.

share|improve this answer
1  
This is an awesome answer. If setting up SSH wasn't so daunting to me I would try this. –  dpollitt Jul 28 '11 at 15:08
1  
I use Windows software called Allway Sync, which is also a file-level synchronization tool that warns you and allows you to "pick the winner" when conflicts occur. I don't use it for Lightroom, but I think it would work very similarly. The down side is that isn't free if you use it heavily (and syncing raw files would probably trigger the "heavy" alarm). allwaysync.com –  coneslayer Feb 15 '12 at 20:09
    
@sebastien.b - what do you do as your harddrive runs out of space? presumably you are not able to keep every raw file you've ever taken on your harddrive. Do you move them somewhere when you don't need them anymore? –  Tom Dec 27 '12 at 23:00
    
Is Unison available for Macs? I have the latest version of OSX and I'd like to be able to keep my Lightroom db file in sync between my iMac and MacBook Air. Thanks! –  user24360 Nov 25 '13 at 17:53
    
@MarkSchwartz Yes it is, I use it all the time to sync between my Mac on my PCs. –  sebastien.b Nov 27 '13 at 16:08

If we remove Lightroom from the equation, we reduce the question down to this:

  • How can I synchronize changes to large files between multiple (2 in this case) computers, without just copying the whole file each time

Now, let's say you're 100% disciplined. You only ever make changes to the file on one computer at a time, and you allow the synchronization software run and do its job before you change the file on any of the other computers.

Warning, read further below for a BIG caveat

In this case, there are many software programs that can do what you want, but since any software that runs on only 1 of the computers will have to read the file on any of the other computers to work out what changed, the only viable solution is one that has software running on both ends, ie. both machines.

Software like this includes:

  • DropBox
  • Microsoft Live Sync

and there are many other. Basically you install the software on all the machines involved, set up synchronization, which usually means registering the folder on one of the machines as "I want to synchronize this folder", and then visiting all the other machines and mapping the synchronized folder (that the software knows about) to a folder on disk. After that, it's all automatic. Changes on one computer gets propagated to the others.

Microsoft Live Sync does not keep an online copy of your files, which means synchronization only happens when computers are online and can talk to each other, whereas DropBox keeps an online copy so you can update on one machine, let DropBox synchronize, turn the machine off, and then turn on another to get the changes downloaded.

Now, the problem here, and you'll find this holds true for all file-level synchronization tools, is that at some point, your disciplination skills waver, and you end up having changes on more than one computer. There is no way for the above type of software to work out how to merge these files. Microsoft Live Sync will flag it as a conflict, and then go about making sure all unique copies of the files are available on all the machines. This essentially means one of them will be renamed, and then both will be downloaded to the machine they're not on.

I don't know how DropBox handles conflicts, but I can tell you it won't be able to merge.

So now you have 2 copies of your catalog.

It gets worse. The Lightroom catalog also contains many smaller files, like thumbnails and such, and what you might end up with is that changes to those on multiple computers gets cross-synchronized to other computers, but end up not corresponding to the catalog they belong to.

Example:

  • Catalog + smaller files A, B and C on machine 1 is changed
  • Catalog + smaller files D, E and F on machine 2 is changed

After synchronization:

  • Conflict on catalog files
  • A, B and C copied to machine 2, but does not correspond to the catalog any more (remember, conflicting catalog changes)
  • D, E and F copied to machine 1, and does correspond to the catalog (at least, one of them)

So let me give you what I consider is a good advice: Don't use file-level synchronization tools for this. It might work in the beginning, but at some point you're left with a restore from backup and hours (or days) worth of lost time.

If, on the other hand, you can find true Lightroom catalog synchronization tools (I've looked but haven't found any yet), that should work. The lightroom catalog is (or at least was in earlier versions) an SQLite database file, so replicating the individual changes to the contents is perhaps doable, but hard, and requires precise tools that understands the database type and most likely the content.

Other ways you can do what you want:

  • Use RDP or similar to work from one machine, but keep everything on just 1
  • ... suggestions?
share|improve this answer
1  
Very good points. I regularly use file-level sync tools, but had a vague feeling they're inadequate in this case. That's why I specifically asked about Lightroom. –  dbkk Jul 29 '10 at 2:08

You could try a different strategy: I use a small large capacity (1TB) USB3 drive and take that wherever I go. These are smaller then a wallet nowadays and quite fast at 7200RPM and USB3. I use Cobian backup freeware to make sure the contents are always backed on my primary workstation and then everytime I plug it back in. You can take the backup a step further and use an online backup service. Personally I use Egnyte for work ($50/ year for 1TB) where I have a use a similar setup for my GIS data. (Egnyte is a File Sharing service with backup capabilities) It takes a long time to run the backup the first time but it's fairly quick (and seamless) thereafter.

share|improve this answer
    
That's actually a really good solution, and is so much easier than syncing. Don't know why I didn't think of it - I may end up exploring that as an option... –  eouw0o83hf Feb 29 '12 at 19:16
    
It works for me. I do a lot of travel for work and I need to bring massive amounts of data everywhere I go. Disciplined backup strategy is essential with this approach. –  Jakub Feb 29 '12 at 19:18

I'm currently developing a tool to synchronize catalogs between computers especially when one is a PC and the other one is a Mac. It currently focuses on the hardest part: making a catalog created/modified on one computer usable on the other despite the differences in layouts. Although work in progress, I use it for about six months with success. So you may want to have a look to LRSync

share|improve this answer

I am not sure if you are talking about Adobe Lightroom or Apple Aperture. Assuming you are talking about Lightroom, since you used that in the title and in the tags of your question, I would recommend using Microsoft Live Sync.

Live Sync Info

Live Sync is a system that allows you to share folders between your computers through the Microsoft "cloud" network. Synchronization is automatic, lightweight, and allows nearly unlimited files. The current "beta" version of Microsoft Live Wave 4 supports the following:

  • Up to 200 synchronized folders
  • Up to 100,000 files per folder
  • A maximum size of 40Gb per file

Thats a grand total of 800,000,000 Gb, or 800 petabytes of data. When syncing, Live Sync only syncs the differences between your shared folders. If you added 100 photos and edited 50 more, it will sync only those photos and the related lightroom data, along with the lighroom collection database. Only modified parts of files, along with patching details, will be transferred, making sure syncs are as quick as possible.

Possible Drawbacks

Live Sync may not be the right tool for the job. If you need to sync considerable amounts of data (many gigs) very quickly, Live Sync may be too slow. Syncing huge amounts of new data on a regular basis will overload your allotted bandwidth that you have to use with Live Sync, so your syncs can take a very long time. If you only need to sync a few hundred megs here and there throughout each day, it will serve you well. If you need to sync many gigs every day, it may not perform well enough to be useful.

Portable Drive Solution

If you do need to sync multiple gigs per day, then a better solution than Live Sync would be a portable external hard drive. To keep things quick, I would try to get a portable drive that supports E-SATA for maximum speed. Just move your lightroom collection and your photography to the portable drive, and set up both your desktop and your laptop to use the portable drive as your collection storage area. The drawback with this solution is you need to be able to physically move a hard drive between the two systems, which may not be ideal. A nice, high-speed, high-density drive like a Hitachi or a Seagate Barracuda will keep things fast.

share|improve this answer
    
FYI Live Sync is now Live Mesh and has totally different stats e.g. 25GB total, 50MB per file, etc. –  glenneroo Jun 18 '11 at 11:43
    
Despite the name change, the stats are still listed as pretty much the same (more, even). If you click the "Sync folders between computers" on the Windows Live Mesh 2011 download page, it says 200 folders, 100k files per folder, 50Gb per file: explore.live.com/windows-live-mesh-sync-p2p-using –  jrista Jun 18 '11 at 17:27
    
That link doesn't say anything and i can't find any info on their site. But i'll believe you... though i could swear i read other stats on one of Microsoft's pages at some point. shrug –  glenneroo Jun 21 '11 at 19:29
    
From that link (still seems to work for me): "With Windows Live Mesh, you can sync up to 200 folders, each up to 50 GB in size and containing up to 100,000 files." –  jrista Jun 21 '11 at 23:33
    
nevermind i'm an idiot this whole conversation can be deleted now ;) –  glenneroo Jun 22 '11 at 13:49

You can always "export as catalog" from your laptop and then "import from catalog" it into your workstation later if you want to consolidate everything there.

Both options are under the file menu.

share|improve this answer
3  
Does this preserve your edit history on the photos? I can't remember the exact reason that I didn't go with this option, but I'm fairly certain that it wasn't going to work for me for one reason or another. –  eouw0o83hf Feb 29 '12 at 20:50
    
That's a good point, I don't know from the top of my head, but since LR is a non destructive editor, the stack of changes have to be stored internally anyway and I would seem silly that this is not exposed as the changes history. I may be wrong, though. –  guioconnor Mar 21 '12 at 2:12

Although the other answers here are very good, they all have synchronization problems (which they address in some detail).

Another approach (really easy to use, but a bit tricky to setup) would be to buy a NAS (network attached storage) which is effectively a small pre-built and configured server that you attach to your main gateway/router - usually with an Ethernet cable.

You then access it as a network drive on your local system and you also (usually) get a static IP address and URL for it (usually by using a service like dyndns.com).

Once you or your local geek get this setup, you put your LightRoom and any other files on it. You then access this one copy from your home system, your notebook, or from any computer on the Internet (with encryption, password security, etc.).

This way you have just one set of data so there are no synchronization issues at all (as long as two people don't try to update it at the same time). You can access it securely from anywhere.

Since it's not stored on the computer you're working on, you have more options for running backups of the data without loading down the computer you're working on. In fact, most NASs that support more than one disk drive will allow you to set them up as RAID 1, 5, or 10 or even (much more simply) to just to use the two (or more) drives separately as mirrors of your data so that you end up with multiple copies of your data on different drives more or less automatically. You can think of it as your own personal cloud storage that no one else has access to (unless you explicitly give it to them).

This takes a bit of expertise to setup, but once it's done, it's very easy and flexible to use - for any data that you need to access from more than one computer or even just for data that you want to backup, but still have access to.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.