Saving metadata in images vs. in an external database

Question

In my search for a perfect photo management application (I know :) I've been doing a bit of comparison between the few applications that seem attractive to me feature / cost wise. One of the more important differences is that some applications save metadata into the images (overwriting the originals in a way) and some save it into their databases.

Although I prefer to keep the originals always untouched, I can see how the approach of having all metadata with the image itself could prove beneficial.

Are there any non obvious advantages/disadvantages to one approach or the other, and what is generally considered best practice (aware, that this is somewhat subjective, so if you could give a reason or two)?

Answer 1

If at least some cataloging information is written to the image, then you can reconnect a file to your database. In principle this can be a single unique ID.

This saves you from:

• You moved or renamed an image file.

If you can write more info into the file -- keywords, captions -- then you are saved from:

• Your database is corrupted.
• You upgraded your computer and your database program doesn't work there.

A third option is to write metadata to sidecar files. Typically these files have the same name as the primary file, but a different suffix.

• This makes all of your data recoverable if your database crashes.

Downsides of storing data in the image:

Writing to the original files can corrupt the file. Most RAW formats are well understood enough now to at least identify and replace strings of metadata with the same length string. If you tell your camera to put the copyright string

Copyright 2018 J. Random Shutterbug Image XXXX-XXXX-XXXX-XXXX-XXXX-XXXX  

Then as long as the DAM keeps that string the same length you are golden.

Keeping all metadata (or as much as you can) in the original images makes for very slow access. Your program has to read at least the first few blocks of every image.

• Writing data back is time consuming.

• Some file formats don't have any metadata capability.

• Some file formats (Photoshop PSD) are noted for mangling metadata.

• A glitch during the write process can corrupt the image file. The alternative, writing a new file, then replacing the old file requires that the entire file be both read and written, rather than just a chunk of it. This has serious performance issues.

Downsides of Databases

Databases are fast, but they are blobby, and you are writing into the middle of blobs of data. If the implementation of the database is solid, there isn't much to worry about. But hard disks have errors, and a single error can make a database partially or fully unusable. Good database design has redundancy built in so that you can repair/rebuild.

Databases are frequently proprietary. Data may be compressed for speed. Getting your data out may be tricky. (Problem for people using aperture)

Databases frequently are optimized in different ways. In general robustness is gained at the cost of performance and complexity. One compromise is to write all changes first to a transaction file (fast...) and then a background process does the database update in the background.

Downsides of Sidecars

You have to read a zillion files at startup.

If you do a batch change (Add the keyword "Italy" to all 3000 of your summer holiday trip shots) the catalog program has open, modify and write back 3000 files.

If you rename a file, and don't rename the sidecar file too, your meta data is no longer connected to your image.

Best practice

Opinion only here: Sorry.

• You want a unique asset tag that resides in the image. This can be an actual tag like the copyright one mentioned above, or it can be a derived tag from information in the image. This could be the EXIF time stamp (Not unique -- multiple shots per second, multiple cameras.) If your program reads makernotes, the best one is Camera model + Camera serial number + timestamp + hundredths of a second.
• You want a database for speed. It, of course has the unique ID
• You want sidecars for rebuilding your database, and for data portability. They have the unique ID.

If the database crashes, it can be rebuild from the sidecars.

If a sidecar is corrupted, it can be rebuilt from the database.

If an image is renamed the ID can be used to reconnect it to the sidecar, and to fix the database.

To make this work, you have to use a lot of timestamps. If the sidecar is more recent than the latest time stamp in the database record, then the sidecar is the authoritative record.

Given the relatively fragile nature of raw files, best practice is a system that only writes zero or once to the Raw file. The Exif

Sidecars don't need to be updated in real time. The slick way to do this would be that whenever the database makes a change to a record:

• Make a new record that duplicates the old record in the database.
• Make the change in the new record.
• New record is flagged, "not written to sidecar"
• Old record is marked "obsolete"
• Another thread writes the sidecar files out, writing out the new one, then deleting the old one (or renaming the new one to the old one's name).
• Periodically you run a cleanup on the database removing obsolete records older than X days. This gives you the ability to rollback changes.

This is not complete: It doesn't address the issue of non-destructive edits. Many programs now allow the creation of multiple images from the same master file, and do not create a new bitmap, but rather a file with a series of instructions for how to make the image from the master. AFAIK all such methods are proprietary. This results in a quandary as the apps that do a good job of tracking metadata may not be able to deal with the non-destructive edits. This can be critical if you crop a person out of an image.

The workaround is that you always write out a new bitmap image from a serious edit. Ideally you have a script that looks for new NDEs and writes out an image based on this, copying the metadata from the master and at some point bringing it up for review for mods to the metadata.

Answer 2

This seems to be a highly polarizing thing. While I would never choose a software that modifies my images in any way, I know people who would not choose one that would not store the metadata in files!

The issue is that if the metadata is external than files are not touched. On my system, images are mounted on a read-only partition so I guarantee that no software can change them which has a few advantages which are important to me:

• The files never change: This means the file-date is always the same and reflect the time of capture. The OS can be used to find and sort the files just as it would any other files.
• The files never change: Backups continue incrementally and do not duplicate or invalidate any file. Each file occurs only once in the 3rd level backup which is on optical-disks.
• The files never change: There is no risk of inadvertent corruption, damage or overwriting due to errors in the imaging application. Corruption and loss are obviously still possible which is why I have 3 levels of backups.

The advantage cited for having metadata in the file is that these two cannot be separated and they always travel together when copying or moving files.

Answer 3

A big problem with storing Metadata in images is that formats like JPEG EXIF have proprietary metadata, called "makernotes" produced by camera makers, editing any EXIF data can result in the complete loss of the proprietary metatdata.

For example, using recent versions of Picasa to assign a title to an image results in the loss of all Nikon's proprietary data about the lens used and the camera settings (exposure compensation used etc). Older versions of Picasa didn't have this problem (presumably they used a different code-base for this feature). This is an example of how some workflow that seems to work now, might have highly undesirable consequences in a later version of the software you use.

Answer 4

If you'll carefully select a right DAM solution that will not damage your existing metadata while updating your images (as RedGrittyBrick said) you will have more benefits with saving your metadata to images:

• You can easily recover all your image descriptions (and hundreds of hours of your hard work) from metadata in case of your database failure. Just answer to a question: whether you are ready to start annotate your image collection again.
• You can embed your copyrights to images so others can not illegally them
• You are not tight with your DAM solution and can easily migrate to another DAM solution later. So at least consider to using a DAM solution that allows you to store metadata in images if you'll need it at some day.
• You can exchange information between two applications on the metadata level: for example use Publishing, Editing and your DAM tool.

Of course, the above said is true if your DAM solution is a mature product with a proven track of customers, and your metadata will be correctly written into the images according to XMP/MWG specification.

And of course you need to backup your image originals and arrange an automatic, daily backups of your database.

Avoid DAM solutions with the following issues:

• Limited support of metadata standards. Random support for XMP or Native Format Specific metadata.
• Incorrectly written metadata that can’t be read in the same way it was written. For example, very few products can save/read correctly hierarchical keywords and split the location into Region\Country\State\City\Location according to IPTC\XMP\MWG specifications.
• Limited succes in writing metadata for various formats, including Camera RAW, PNG or PDF.