Can anyone recommend a good open-source photo management platforms for power users?

Question

Short version of the question:

Does anyone know of any good open-source photo management/editing suites, a la Aperture or Lightroom?

I'd want it to run on MacOS X, by the way, though options that are (more or less) cross-platform would certainly be welcome, as long as MacOS X is one of the supported platforms.

I know there is some stuff out there, but so far, I haven't run into anything that makes me particularly happy. (Though I admit, I've only glanced at some of the available options, and probably done less than that, for others.)

Going into a lot more detail (warning: the rest of this post is going to be long. Feel free to skim -- I've made some things bold, to help with that)...

There are a bunch of things I'd like to see in such a program. (Some of these may be "in your dreams" type features, but hey, that's in part what this post is about -- finding the software package I've been dreaming of. Which Aperture and Lightroom get kind of close to, but not quite there, for various reasons.) (This post was inspired in part by a question about Lightroom, which seems to highlight a potentially-missing feature.) Such features might include (and this is only a subset, I'm sure):

• It needs to be fast -- Aperture and Lightroom do a decent job (usually) at doing things quickly. This would need to at least get close to their numbers, and preferably beat them.

• Scriptability -- It'd be really nice to be able to write little scripts to query a set of photos in various ways, and then act upon them -- whether that's to make adjustments, or to do a bulk export, or automatic additions of tags, or whatever. This is really my #1 requirement, I think -- I'm particular about certain things, and currently have scripts that I run pre-import and post-export from Aperture or Lightroom. It'd be nice to have those things integrated in. To define what I'm looking for further, I'd like the ability to do things like:

  • mangle filenames during import, based on camera metadata. (e.g., change [card]/DCIM/123CANON/IMG_4567.CR2, shot on my 30D, into something like [datastore]/2010/11/2010-11-30-some_shoot/my30d-123-4567.CR2, where some_shoot is something I'm prompted to type in during import, and the rest is figured out from the metadata and/or original filename.)

  • take that some_shoot and also automatically apply EXIF and/or IPTC data during the import based on it -- and/or other things I'm prompted for (where I can configure what things I want to be prompted for) or have configured (e.g. auto-adding copyright statements, etc.)

  • automatic importing -- doing all the above as soon as I insert a card, or, at my preference (in a setting somewhere), upon a single button-press or whatever.

  • selecting images with arbitrary queries -- something SQL-Like, perhaps? Though also different than that -- being able to create, say, a variable that's a collection of images, from which you can make further selections or take other actions. Maybe something like (arbitrarily using ruby-like syntax for my pseudocode):

    lowlight = library.search(:iso => 100,
                              :exposure => '< 1/4',
                              :aperture => '> f/16')
    

    after which I could then do:

    thefunstuff = lowlight.search(:rating => '> 3', # 3 stars or better
                      # must have all of these tags:
                      :tags => [ 'beach', 'california' ],
                      # and any one or more of these:
                      :any_tag => [ 'light painting', 'LEDs', 'fire poi' ])
    

    after which I could then do:

    thefunstuff.add_tag('light painting') # make sure all have this tag
    thefunstuff.export_to_flickr(:find_set => 'Low Light',
      :create_set => 'Light Painting on California Beaches')
    

  • changing settings -- whether I'm working on the current_photo, or thefunstuff from above, having the ability to change various settings -- whether it's adjust_exposure(+0.1), or set_whitebalance(5000, -3) # kelvin, tint, or photoB.exposure = photoA.exposure or even:

    thephotosIwanttweaked.set(photoB.get_settings(:exposure,
      :whitebalance, :iptc => { :tags, :copyright })
    

    where thephotosIwanttweaked is a variable containing a collection of photos previously obtained -- perhaps with a query as shown above, or perhaps via GUI-based selection (click an image, shift-click to select several more, then say thephotosIwanttweaked = gui.currently_selected_photos or some such)

• Keyboard-based interaction mode -- As a programmer in a "past life" (surely obvious from the above), I find that I tend to like to keep my hands on the keyboard a lot of times. GUI and mouse-based (or tablet-based, or what have you) interaction are quite useful when manipulating images, and I want that to exist, too. I just find that typing "select all" at a prompt, or hitting "command-A" on my keyboard, or the like, is far quicker and easier (especially for some kinds of things) than doing it by the GUI. (See the section above about selecting images with arbitrary queries, for example.) Lately, I've been starting to use emacs for things (after switching from vim -- editor wars aren't allowed here, right? Oh, few of you even know what I'm talking about, huh?). Having the ability to have actual emacs be part of this, and/or to have emacs able to talk to it via an API, would be way cool, in my book. (Of course, this would presumably mean that there'd also/instead be an elisp way to interact with this, rather than ruby, but whatever. Or maybe a new language is invented, specific to the purpose.

• Extensibility -- this thing should have a nice API for writing anything from RAW import tools to fast image editing plugins to exporters for your favorite website. Maybe face detection and such, too?

• Real-time GUI manipulations -- much like the UIs in Aperture or Lightroom. Along with all the above, the standard GUI-based manipulation strikes me as quite important, too -- having real-time (or close to) feedback when making visual changes is key to visual things.

• Ability to identify objects/people in photos -- One thing that I think is lacking in Aperture's new face detection stuff, and which could have been really helpful for me recently, is a way to identify ("tag", whatever) people or objects within a photo. Example scenario: I'm shooting a sporting event, and I want to go through and quickly and easily identify which players are in each photo. I imagine me as a human doing a lot of the work on this, though automatic detection would be nifty, too... but the thing that I see as being different from existing UIs is a way to basically select a region of the photo that represents a particular player, and then do so for other players as well, and then go through in a second pass and try to tie them together (with the computer perhaps helping, along the way). So like, maybe I select a player in one photo, and I don't know who they are yet, because their number is obscured... but later, I select what ends up being the same person in another photo, where their number is visible, and then, because of attire or whatever other distinguishing feature there might be, I'm able to tie the two together. But I still don't know their name, necessarily -- but perhaps I have a roster, and that can get looked up. This could also be useful in a variety of other situations, I imagine -- a studio shoot where you want to identify which props were used in which shots, say, so that you can later ask for a photo that includes the such-and-such prop. Stuff like that. Developing a good UI for this would likely be an interesting challenge, but I think I could imagine how it could be done that could make sense.

• Photo and/or meta-data manipulation on multiple devices -- Maybe the RAW files only exist on one device, or maybe they're on a network drive and can be accessed from multiple computers. But what if, also, previews and metadata were uploaded automatically to a web server somewhere, so that you could get access to them on your smart phone, say, and do ratings, tagging, and the like. The data would get synced up (somehow), and could also potentially be shared to different people -- perhaps (if, say, this was being used at some sort of company) your event coordinator is better at doing the identification tasks, and your photographer is better at post-processing the image itself, and your graphic designer wants input on things, as well. If all those people could access the same images, that could be really really useful. (This could also apply to a photo business, with assistants and such.)

Anyway, hopefully that gets the general flavor across of the kinds of things I'd like to do and see, though I'm sure I've only scratched the surface on what's possible, and that even a subset of this stuff would be useful to me. Does anyone know of anything like this?

Alternately, would anyone be interested in possibly starting work on such a beast? I'd need a lot more experience with GUI programming, graphics manipulation, and the like -- not to mention more time and energy to work on this -- before I'd be able to do anything that even begins to be useful on my own... but I think if I had some people to work with, we might be able, together, to do something really really cool.

I could imagine forming a company around it, too -- there might well be some hardware that could be useful to integrate with it, which could be the money-making piece. Or it could all just be done as volunteer-done open-source software. Either way.

OK, I'm done rambling now. I'm very curious to see what sorts of responses this question will bring. :)

Answer 1

Have you taken a look at Darktable? It seems to be open-source and Linux-only.

Feature description

It needs to be fast

It is quite quick and supports computations on the GPU using OpenCL.

Scriptability

It supports scripting using LUA. This is a pretty recent feature (as of '15), so the API is still small.

Keyboard-based interaction mode

You can do quite a few things using the keyboard. You have to check this yourself.

Extensibility

If you can't do it with Lua, you can hack the C code. Well...

Real-time GUI manipulations

Most filters are reasonably quick, so I think yes.

Ability to identify objects/people in photos

Nope

Photo and/or meta-data manipulation on multiple devices

It supports making temporary local copies of images on a remote drive for manipulation (and presumably back-syncing). I'm not using it this way, so I cannot comment much.

Friendly workflow

It doesn't touch the files but stores everything in sidecar files. This approach is extremely great and makes DT compatible with other programs (at least on the common meta-data properties). While it has a database to speed things up, the sidecar files allow you to copy photos around while retaining the manipulations you made with Darktable.

Answer 2

Wow, that's a long list.

I think we can all be pretty safe in saying that there sure isn't anything like that now.

One of the most interesting project that shares a lot of your goals is F-Spot -- you might want to look at getting involved there. It's also worth noting that photo management features are on the "long-term roadmap" for the excellent raw converter RawTherapee, so that might an interesting place to direct some energy.

Answer 3

Yes, it's safe to say there is no such thing (Update: look at that, there might be, but with the level of performance you are asking?). Even Gimp is not remotely close to Photoshop for advanced users, and the list of features you describe would put such an app in the "advanced" category. This is not true for all apps, Firefox was born after Internet Explorer and is definitely a competitor feature-wise. Photoshop is a monster though, and Adobe has tons of resources to throw at PS and Lightroom to make them the fantastic pieces of software engineering that they are.

Moreover, the open source movement has Unix roots, and one old precept of that philosophy is "Small is beautiful. Make each program do one thing well". The link you provided exemplifies it well. There is a ton of small tools that you can use and that can form a chain to get you there, but this is not what makes Lightroom or Aperture attractive. The key to these applications is that they were designed by photographers for photographers as one coherent workflow. This lets you do 90% of simple to semi-advanced photo tasks from one single app using the same user interface. This saves people so much time!

What I'm getting at is that this week you could get Lightroom for $150. This is really cheap for the time it will save you and the power behind that tool. If you are shooting RAW, this is hard to live without. This needs to be put in perspective too: it's half the price of my cheapest lens, and I have 3 lenses next to me that are almost 10 times the cost.

I work in the open source business, and this how I've made a living for more than 10 years. We create advanced software (scientific viz, biomedical, supercomputing), but not out of thin air, we have to fund them, we have to pay salaries. The community helps but when it gets pretty sophisticated like this the learning curve can be steep. I don't see any competitor to Lightroom showing up anytime soon just from volunteers working on their free time, with that level of performance.

Your first request, performance, is key for me here, because if it takes just 30% or 50% more time for me to process one photo, multiply that by 15,000+ pictures a year and the time I wasted completely justify buying a fast commercial app. I've seen open source apps focus on features, some on optimization/performance, rarely both (and I don't blame them). The race for Javascript performance in Firefox is pretty recent, for example. I pick (and support) open source software first, but when I need to get the job done and a tool has a clear lead, I (or my company) will put money to buy a reasonably priced shareware or commercial app. Pragmatism I guess.

Now here is something I want to ask you honestly. How far did you use Lightroom? It seems you did, but I'm just asking. You mention that it is not up to what you want, but I'm a bit puzzled here:

• It needs to be fast: LR is multi-threaded and uses multiple cores. I throw really big files at it (21MP) and I'm actually amazed its new denoise engine can work so fast. Advanced memory management, multi-threading, image processing, signal processing, you would need some serious fellows to beat that. I'm not saying it can't be faster (it's not GPU accelerated, to my knowledge), but if your app just chokes and page swap after 6MP, I've a problem.
• Scriptability, Extensibility: the plugin framework in Lightroom is entirely scriptable. Plugins are scripts (it uses LUA).
• Mangle filenames during important, based on camera metadata: LR does a lot of that. I have a precious preset that helps a lot, you can rename files at import by combining patterns and keywords that are replaced by metadata (unfortunately I don't see how to extract a substring of the original filename). See my answer to another question for more info.
• Automatically apply EXIF and/or IPTC data during the import automatic importing: again, LR can do that (I apply a ton of info at import time, automatically).
• Selecting images with arbitrary queries: you pretty much described Smart Collections in LR. Lot to love here. See my answer to another question for more info.
• Keyboard-based interaction mode: there is a huge list of shortcuts in LR, it's actually a bit overwhelming. For even more flexibility, check PADDY for Lightroom: free Key mapping, external keyboards, midi controllers, macros.
• Ability to identify objects/people in photos: not in LR, but in Aperture I think.
• Photo and/or meta-data manipulation on multiple devices: I think the new "Publish" services in LR3 would let you do that (i.e. they are not just "upload", they are more "hey, did that photo change, if yes then I need to push/publish it to other services somewhere").

Answer 4

Not a complete answer, but Exiftool is a perl library/command line tool photo metadata reader/writer. Theoretically some of what you want could be scripted with it, especially auto-tagging images and setting file names and directories.

Answer 5

Here is my import-and-mangle script; it's nothing special, but perhaps useful to look at. The mangling could be extended to include whatever exiftool can get its hands on.

Answer 6

Darktable (which you can download for OSX here) is probably the software that comes closest to what you are looking for. The workflow still has some rough edges, compared to Lightroom or Aperture, but quite usable especially if you are looking for a scripting interface.

Answer 7

I would add DigiKam (KDE photograph management and editing tool) and LightZone (editing tool) to the list to examine.