What is the best way to synchronize Adobe Lightroom databases between two computers (a desktop and a laptop)?
I'd like to be able to edit photos on either machine, and preferably not copy the whole 3+ gb database on each sync.
|show 1 more comment|
I've been synchronizing collections spanning up to 60GB and ten thousands of small and large files for years now. This includes my LR catalogs and pictures since I started photography. I'll be honest though, this is not a tool a lot of people use, it is not trivial to setup but it will solve your problem. Once it is up and running, it is very easy to work with and it will perform its duty quickly and effortlessly. As a software engineer, this is one of the top 3 pieces of software I use daily and can't live without.
The tool is Unison File Synchronizer.
Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.
This later point is relevant to your problem. If a large file has been updated (say, your LR catalog), it will not copy the whole file over. It will instead transmit a smart, compressed subset (a delta) representing the difference between both files. For example, if I modify the Artist name on a MP3 or add some keywords to a JPEG file, it will not send the whole song or photo over, but only a very small portion of it. Granted, bigger changes happen in your LR catalog, but never quite big enough that they require to send the whole file over. This makes sense because your catalog is a SQLite database (a SQL db on disk); for performance reasons it is in the database engine's best interest not to shuffle every bits around.
Unison has no problem renaming files and directories. Files are not identified by names actually, but by a unique signature. This is very convenient.
Compared to Microsoft Live Sync
Similar to Microsoft Live Sync, Unison does not keep an online copy of your files (thank God), which means both machines need to be connected to the internet.
Unlike Microsoft Live Sync (from what I understand), it does display potential conflicts and its user interface will let you decide which replica should "win". That is, if you made changes on both sides, it will let you mark if the 'left' side should have priority over the 'right' side (see screenshot below), or vice versa. You can do so file by file, replica by replica, on a subset of files, etc. You can even, for text files, have Unison launch a tool that will let you resolve the conflict manually at the contents level (say, using WinMerge).
Thumbnails and Previews
Now what about thumbnails and previews? Do not synchronize them. You do not need to, I never did, and LR is smart enough to regenerate a preview if it's missing. This has never been a problem for me in 2 years.
Besides your RAW and JPEG files of course, which piece of data should you synchronize to make sure you can work with LR on both sides? There are 3 components to consider, to my knowledge:
You can change the default location of your catalog, previews and Camera RAW Cache from the Preferences. This is how I moved my catalog to a specific user directory (my HOME directory, with personal data), and this is the very directory I sync. You don't have to do that, but it makes my life easier.
As I mentioned earlier, installing Unison is tricky, check the user manual online, and comment if you are still in the blue. It does involve running a SSH server on one of the computer (if you sync between two), traditionally using Cygwin or a subset of it. This is the tricky part, but SSH is an open, robust, safe and secure solution. I trust this set up more than running Microsoft Live Sync or Dropbox, which are essentially client/servers too.
Once your SSH server is running, you are pretty much all set. You only need to write a small text file called a 'profile' that will tell Unison which folders need to be sync'ed. I realize it's not very user friendly, but it does have a powerful syntax that lets you ignore files using regular expressions or simple patterns. Here is what the top of a profile looks like, for instance:
The first line specifies where my local replica is, and the second where my remote replica can be found, on the foo.net (fictitious) machine at home for example, in the
These 3 lines instruct Unison to ignore my LR thumbnails/previews, as well as any temporary lock/journal files found in the Pictures/Lightroom sub-directory (relative to the root directory specified above).
And that's about it.
The first time you run Unison takes quite a bit as it computes hash signatures for all the files under its new jurisdiction. This is key to help computing differences quickly. The next time you run it, it's astonishingly fast. The user interface is a bit rough because Unison is an exotic beast written in OCaml + GTK. Just pick your profile from a list and Unison will check your local replica for changes. At the same time it will call itself through SSH on the other replica to compute remote changes. Wait a few seconds and it will display a list of changes and potential conflicts to help you reconcile the two, if needed. If there is nothing to resolve, just click "Go" and sync happens automagically.
In this screenshot below I’m synchronizing my local machine (first column), with my remote machine named KANEDA. The second column, in between, is what Unison thinks should be done.
As you can tell:
This situation doesn’t happen very often to me anymore, 99% of the time I don’t have any choice to make, I just review the situation quickly, press “Go”, and Unison synchronizes both replicas.
I've used this tool for years and I swear by it. I synchronize and backup music, photos, documents and personal data using Unison, a task that would be impossible to do efficiently manually. It supports very large collections and thousands and thousands of file. It handles renaming, smart sync of large files and works quickly. It never corrupted a single one of my files.
This is still a file-level synchronization tool. It is not aware of the subtleties of your catalog, only Lightroom does. If you made changes to your catalog on both sides, you will have to make a choice between the two. At least the whole file is not copied over, and the user interface makes it easy to tell Unison which one should win. As somebody pointed out in this post, a bit of discipline should take care of this problem. Your catalog is one complex file. Even if a general-purpose tool used high-level SQL commands to inspect your catalog’s database and try to synchronize it with another, I still wouldn’t trust it over Lightroom.
If we remove Lightroom from the equation, we reduce the question down to this:
Now, let's say you're 100% disciplined. You only ever make changes to the file on one computer at a time, and you allow the synchronization software run and do its job before you change the file on any of the other computers.
Warning, read further below for a BIG caveat
In this case, there are many software programs that can do what you want, but since any software that runs on only 1 of the computers will have to read the file on any of the other computers to work out what changed, the only viable solution is one that has software running on both ends, ie. both machines.
Software like this includes:
and there are many other. Basically you install the software on all the machines involved, set up synchronization, which usually means registering the folder on one of the machines as "I want to synchronize this folder", and then visiting all the other machines and mapping the synchronized folder (that the software knows about) to a folder on disk. After that, it's all automatic. Changes on one computer gets propagated to the others.
Microsoft Live Sync does not keep an online copy of your files, which means synchronization only happens when computers are online and can talk to each other, whereas DropBox keeps an online copy so you can update on one machine, let DropBox synchronize, turn the machine off, and then turn on another to get the changes downloaded.
Now, the problem here, and you'll find this holds true for all file-level synchronization tools, is that at some point, your disciplination skills waver, and you end up having changes on more than one computer. There is no way for the above type of software to work out how to merge these files. Microsoft Live Sync will flag it as a conflict, and then go about making sure all unique copies of the files are available on all the machines. This essentially means one of them will be renamed, and then both will be downloaded to the machine they're not on.
I don't know how DropBox handles conflicts, but I can tell you it won't be able to merge.
So now you have 2 copies of your catalog.
It gets worse. The Lightroom catalog also contains many smaller files, like thumbnails and such, and what you might end up with is that changes to those on multiple computers gets cross-synchronized to other computers, but end up not corresponding to the catalog they belong to.
So let me give you what I consider is a good advice: Don't use file-level synchronization tools for this. It might work in the beginning, but at some point you're left with a restore from backup and hours (or days) worth of lost time.
If, on the other hand, you can find true Lightroom catalog synchronization tools (I've looked but haven't found any yet), that should work. The lightroom catalog is (or at least was in earlier versions) an SQLite database file, so replicating the individual changes to the contents is perhaps doable, but hard, and requires precise tools that understands the database type and most likely the content.
Other ways you can do what you want:
You could try a different strategy: I use a small large capacity (1TB) USB3 drive and take that wherever I go. These are smaller then a wallet nowadays and quite fast at 7200RPM and USB3. I use Cobian backup freeware to make sure the contents are always backed on my primary workstation and then everytime I plug it back in. You can take the backup a step further and use an online backup service. Personally I use Egnyte for work ($50/ year for 1TB) where I have a use a similar setup for my GIS data. (Egnyte is a File Sharing service with backup capabilities) It takes a long time to run the backup the first time but it's fairly quick (and seamless) thereafter.
I'm currently developing a tool to synchronize catalogs between computers especially when one is a PC and the other one is a Mac. It currently focuses on the hardest part: making a catalog created/modified on one computer usable on the other despite the differences in layouts. Although work in progress, I use it for about six months with success. So you may want to have a look to LRSync
I am not sure if you are talking about Adobe Lightroom or Apple Aperture. Assuming you are talking about Lightroom, since you used that in the title and in the tags of your question, I would recommend using Microsoft Live Sync.
Live Sync Info
Live Sync is a system that allows you to share folders between your computers through the Microsoft "cloud" network. Synchronization is automatic, lightweight, and allows nearly unlimited files. The current "beta" version of Microsoft Live Wave 4 supports the following:
Thats a grand total of 800,000,000 Gb, or 800 petabytes of data. When syncing, Live Sync only syncs the differences between your shared folders. If you added 100 photos and edited 50 more, it will sync only those photos and the related lightroom data, along with the lighroom collection database. Only modified parts of files, along with patching details, will be transferred, making sure syncs are as quick as possible.
Live Sync may not be the right tool for the job. If you need to sync considerable amounts of data (many gigs) very quickly, Live Sync may be too slow. Syncing huge amounts of new data on a regular basis will overload your allotted bandwidth that you have to use with Live Sync, so your syncs can take a very long time. If you only need to sync a few hundred megs here and there throughout each day, it will serve you well. If you need to sync many gigs every day, it may not perform well enough to be useful.
Portable Drive Solution
If you do need to sync multiple gigs per day, then a better solution than Live Sync would be a portable external hard drive. To keep things quick, I would try to get a portable drive that supports E-SATA for maximum speed. Just move your lightroom collection and your photography to the portable drive, and set up both your desktop and your laptop to use the portable drive as your collection storage area. The drawback with this solution is you need to be able to physically move a hard drive between the two systems, which may not be ideal. A nice, high-speed, high-density drive like a Hitachi or a Seagate Barracuda will keep things fast.
You can always "export as catalog" from your laptop and then "import from catalog" it into your workstation later if you want to consolidate everything there.
Both options are under the file menu.
Although the other answers here are very good, they all have synchronization problems (which they address in some detail).
Another approach (really easy to use, but a bit tricky to setup) would be to buy a NAS (network attached storage) which is effectively a small pre-built and configured server that you attach to your main gateway/router - usually with an Ethernet cable.
You then access it as a network drive on your local system and you also (usually) get a static IP address and URL for it (usually by using a service like dyndns.com).
Once you or your local geek get this setup, you put your LightRoom and any other files on it. You then access this one copy from your home system, your notebook, or from any computer on the Internet (with encryption, password security, etc.).
This way you have just one set of data so there are no synchronization issues at all (as long as two people don't try to update it at the same time). You can access it securely from anywhere.
Since it's not stored on the computer you're working on, you have more options for running backups of the data without loading down the computer you're working on. In fact, most NASs that support more than one disk drive will allow you to set them up as RAID 1, 5, or 10 or even (much more simply) to just to use the two (or more) drives separately as mirrors of your data so that you end up with multiple copies of your data on different drives more or less automatically. You can think of it as your own personal cloud storage that no one else has access to (unless you explicitly give it to them).
This takes a bit of expertise to setup, but once it's done, it's very easy and flexible to use - for any data that you need to access from more than one computer or even just for data that you want to backup, but still have access to.