20

I read here that Lightroom V3 Catalogs can handle lots of images, but the largest database anyone mentioned was 60k. Meh! ;)

I currently have over 300k RAW + 200k JPEGs (from days long gone) clocking in at around 3 TB and its growing rapidly. Everything is sorted into 14 catalogs (e.g. Finished, Light Graffiti, Photoshoots, Work, etc.) because way back when I switched from RawShooter to V2, I found that large-ish Catalogs were wayyyy slow, even on a quad-core with 6 GB of RAM.

I would love to go back to the simpler days of 1 catalog, but I'm scared it's gonna get messy again... not to mention the import will take bloody forever ;)

Does anyone have any real-world experience working with MASSIVE catalogs (> 500k) and can tell me what I can expect performance-wise? Are some operations meh, speedy or time for a coffee-break?

NOTE: I'm particularly interested in Windows 7 experiences, as it seems to me that LR generally runs faster on Macs.

| improve this question | |
  • 1
    request for clarification: You're asking about Windows 7, because LR runs faster on Macs? I guess that just means you don't want Mac answers, because they may not reflect your reality? In case you mean anything different, I thought I'd ask, here. :) – lindes Dec 14 '10 at 8:31
  • Because I believe it runs faster on Macs. It might still be interesting to know about Mac experiences (for other curious people) but the person sitting next to me is on a Mac and LR runs seemingly faster there. – glenneroo Dec 14 '10 at 14:51
  • Hmmm, I found this question because I was worried that my catalog had grown to 83GB and 66K items (mixture of RAW, DNG and JPEG). – Paul Hadfield Dec 17 '12 at 11:45
17

I was told hundreds of thousands by an Adobe employee working on Lightroom. So, less than a million.

The Lightroom 3 FAQ seems to support this:

How many photos can I have in a catalog? There is no specific maximum number of photos you can store in a Lightroom catalog. Your computer might run out of address space for your photos between 100,000 and 1,000,000 photos.

Source: http://kb2.adobe.com/cps/333/333736.html

PS: Delete is your friend. Everyone says storage is cheap but the cost of managing storage is not. Personally I delete 7 out of every 8 photos I shoot and my photo collection is nicer and more manageable because of that.

| improve this answer | |
  • 2
    +1 Good point about volume, though I've recovered some shots that I thought were toast before. I'm not entirely sure what Adobe means about "address space" for this... I'm going to assume that it isn't an integer because that massively exceeds 1,000,000 on even a 32 bit platform. Quite obscure there. – John Cavan Dec 10 '10 at 3:19
  • 2
    @John: the address space they mention is how much storage is indexable in the database file. Different images require different amounts of space, but when they run out of space in the file then they can't add any more images (or expand the storage needed for an image already there). Based on the limits of the database engine they use, I believe this is 4GB. If that's the case, then based on the size of my catalogs relative to the number of images they have... I'd expect the limit to be well above 1M images. But they probably set that limit for "worst case". – cabbey Dec 17 '10 at 22:07
  • +1 for deleting. I'll go through my catalog every 2 months and delete anything that I can't save and I'm not 100% on, frankly if it's on my drive and I've not shared it or printed it, I'm probably not going to look at it again. – Nicholas Smith Sep 17 '11 at 11:48
11
+50

I've got a lightroom instance with 211,489 images in it.

It's certainly a bit slower than sub 10K collections, but it is usable. It does take a long time to completely start, though, since LR seems to either insist on making sure all the library photo files are actually there when it loads the libraries, or doing some sort of library scanning something. Either way, it takes a few minutes until all the catalog is available.

The GUI is responsive while it's scanning the libraries, so you can work while it's counting the files. Unless you want to work on a file it hasn't scanned yet, anyways.

While this is off-topic a bit, the best software I've found for managing enormous catalogs is Picasa, of all things.

Picasa manages a 600K image collection I have without any appreciable slowdown on start. It also seems to do dynamic loading of everything, so you get a low-res thumbnail almost instantaneously, which then improves as it loads the actual file.

-

What I can say, about every platform I have tried, is that importing always seems to take forever. importing 100K+ images is 10 Hours +, if not days. I would strongly recommend breaking the import up into sections, so if something dies/your computer accidentally goes to sleep/whatever, you don't lose all your progress.

| improve this answer | |
  • 1
    Are you sure about that bit about LR making sure photo files exist? I have archived some of my photos that were in my libraries, and LR never seemed to slow down on startup. It would also show all the thumbnails generated for the missing libraries, and only check for existence once you actually tried to work with a missing image. – jrista♦ Dec 15 '10 at 4:40
  • 1
    Well, it may just be loading the libraries. It certainly takes a few minutes, and the subfolders in the catalog come up one-by-one (the image count gets populated, I mean). – Fake Name Dec 15 '10 at 4:42
  • 1
    @jrista - If you look carefully, even before you try to use an image, Lightroom makes a little icon appear on the upper-right corner of missing images when you are in the grid-view within the library module. I am not sure if it does this on folder open or on startup but I do know that it does something to each folder at startup and it does at least recompute the image count each time. Seems a little silly but it is clearly visible in the interface if you have the left folder pane opened. – Itai Dec 15 '10 at 16:47
  • Interesting. Are you on Windows or Mac? And what do you mean by "a long time to completely start" ? 30 seconds? 5 minutes? – glenneroo Dec 16 '10 at 0:36
  • 1
    As to windows vs Mac, Yes. Startup time is generally a minute or two, it varies too. – Fake Name Dec 16 '10 at 5:21
7

Apparently, Adobe uses a SQLite database for the catalogs and that means it is subject to the limits of the database. Depending on how Lightroom uses the database, it could run up against these limits relatively quickly, it's not Oracle after all... :) So, yeah, big catalogs could suffer performance hits if your searches don't hit any indexes and, since I'm not sure how the database is indexed, I couldn't say how likely the miss is and what the frequency of full table scans is required as a result.

Anyways, I expect choices such as this were originally based on general experience of film photography, for which I expect that zero people have reached multiples of a hundred thousand images on. However, digital is a different beast. Net effect, I think the smaller the catalog, the better Lightroom will perform, so if you can divide up your collection to hit 100,000 or so, it would probably be better.

Anyways, if you're a techie and want to play, here's a starting point.

| improve this answer | |
  • ooooh that could be fun! hmmm but it says it works with catalog version 1.4.1. Will have to play around I guess ;) – glenneroo Dec 10 '10 at 3:53
  • I doubt they've mucked with the underlying DB structure of the database, so you're probably good to go. I don't use LR, I use Photoshop with Bridge, I catalog my own way (on the file system and on backups), so that's probably all I can add. – John Cavan Dec 10 '10 at 4:13
  • None of the limits in the page you linked to refer to running limits; only software and database design limits. So they do not describe the total number of possible entries in the database, for example. – Kendall Helmstetter Gelner Dec 16 '10 at 5:44
  • @Kendall - The last one does, it's the page limit. There are probably a few other impacting behaviours such as overflowing the size of a page (I'd suspect it's 8k), and such. As you push to those limits, which are soft, you probably start to effect performance notably. – John Cavan Dec 16 '10 at 11:36
  • 2
    The statement "it could run up against these limits relatively quickly" is not substantiated. Really, the place SQLite has typically been weak has been in concurrency, not capacity. AFAIK, SQLite runs equally well on Windows and MacOS so that shouldn't be an issue. It's important to remember that Lightroom is way more than just the database that structures its catalog. If you need one stop shopping, then LR is the place to go. If you like to mix and match, there are plenty of DAM tools around. Like Extensis Portfolio. (and I'm out of space!) – Steve Ross Sep 17 '11 at 5:50
4

I will add in here, my wife's system has around 400k images at least 100k are raw. The system was a little sluggish with her Q6600/4gb system but on her new AMD 1100t 6core/8gigs the catalog is just as snappy as a brand new database. I still haven't moved her onto SSD(for OS and Database files) so I am looking forward to seeing how that improves it.

I would like to add that checking the option to render previews during import was key to allowing my wife to flick through 1000 image imports with zero delay on raw preview rendering.

| improve this answer | |
1

There is a fully featured trial version available on Adobe's website. I think you should just download it and test your library on your own computer to see the real performance.

| improve this answer | |
  • Thanks, but I already own it and I have actually tried a few times to do a full import but I never get past several hours before I need my computer for some other project. It's sad because the import consumes almost all available resources on my system (4x 3.4ghz, 6gb of ram) leaving almost nothing left for me to even run a Remote Desktop session to work - I'm guessing it will take around 24 hours. – glenneroo Dec 16 '10 at 13:45
  • Try Picasa. It does incremental imports - You can close it while importing, and it will pickup where it left off when you open it again. – Fake Name Dec 20 '10 at 9:06
  • 2
    I'm sorry but Lightroom vs Picasa? Maybe for editing JPEGs but for RAWs forget about it! – glenneroo Jan 16 '11 at 23:43
1

There is no specific limit, the limit is somewhere between 100K and 1 million. But in general you're going to have poorer performance the more photos you have, and it's going to be a non-linear decrease, ie, adding the 100,001th photo is going to make more of a dent in performance than the 1001th photo.

I imagine that alot of it's going to depend on how good of a machine you have. If you have a top notch machine, it's probably going to work better than an older machine might.

Maybe you could try to do the importing at night or some other time you're not going to use your machine for many hours?

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.