If at least some cataloging information is written to the image, then you can reconnect a file to your database. In principle this can be a single unique ID.
This saves you from:
- You moved or renamed an image file.
If you can write more info into the file -- keywords, captions -- then you are saved from:
- Your database is corrupted.
- You upgraded your computer and your database program doesn't work there.
A third option is to write metadata to sidecar files. Typically these files have the same name as the primary file, but a different suffix.
- This makes all of your data recoverable if your database crashes.
Downsides of storing data in the image:
Writing to the original files can corrupt the file. Most RAW formats are well understood enough now to at least identify and replace strings of metadata with the same length string. If you tell your camera to put the copyright string
Copyright 2018 J. Random Shutterbug Image XXXX-XXXX-XXXX-XXXX-XXXX-XXXX
Then as long as the DAM keeps that string the same length you are golden.
Keeping all metadata (or as much as you can) in the original images makes for very slow access. Your program has to read at least the first few blocks of every image.
Writing data back is time consuming.
Some file formats don't have any metadata capability.
Some file formats (Photoshop PSD) are noted for mangling metadata.
A glitch during the write process can corrupt the image file. The alternative, writing a new file, then replacing the old file requires that the entire file be both read and written, rather than just a chunk of it. This has serious performance issues.
Downsides of Databases
Databases are fast, but they are blobby, and you are writing into the middle of blobs of data. If the implementation of the database is solid, there isn't much to worry about. But hard disks have errors, and a single error can make a database partially or fully unusable. Good database design has redundancy built in so that you can repair/rebuild.
Databases are frequently proprietary. Data may be compressed for speed. Getting your data out may be tricky. (Problem for people using aperture)
Databases frequently are optimized in different ways. In general robustness is gained at the cost of performance and complexity. One compromise is to write all changes first to a transaction file (fast...) and then a background process does the database update in the background.
Downsides of Sidecars
You have to read a zillion files at startup.
If you do a batch change (Add the keyword "Italy" to all 3000 of your summer holiday trip shots) the catalog program has open, modify and write back 3000 files.
If you rename a file, and don't rename the sidecar file too, your meta data is no longer connected to your image.
Opinion only here: Sorry.
- You want a unique asset tag that resides in the image. This can be an actual tag like the copyright one mentioned above, or it can be a derived tag from information in the image. This could be the EXIF time stamp (Not unique -- multiple shots per second, multiple cameras.) If your program reads makernotes, the best one is Camera model + Camera serial number + timestamp + hundredths of a second.
- You want a database for speed. It, of course has the unique ID
- You want sidecars for rebuilding your database, and for data portability. They have the unique ID.
If the database crashes, it can be rebuild from the sidecars.
If a sidecar is corrupted, it can be rebuilt from the database.
If an image is renamed the ID can be used to reconnect it to the sidecar, and to fix the database.
To make this work, you have to use a lot of timestamps. If the sidecar is more recent than the latest time stamp in the database record, then the sidecar is the authoritative record.
Given the relatively fragile nature of raw files, best practice is a system that only writes zero or once to the Raw file. The Exif
Sidecars don't need to be updated in real time. The slick way to do this would be that whenever the database makes a change to a record:
- Make a new record that duplicates the old record in the database.
- Make the change in the new record.
- New record is flagged, "not written to sidecar"
- Old record is marked "obsolete"
- Another thread writes the sidecar files out, writing out the new one, then deleting the old one (or renaming the new one to the old one's name).
- Periodically you run a cleanup on the database removing obsolete records older than X days. This gives you the ability to rollback changes.
This is not complete: It doesn't address the issue of non-destructive edits. Many programs now allow the creation of multiple images from the same master file, and do not create a new bitmap, but rather a file with a series of instructions for how to make the image from the master. AFAIK all such methods are proprietary. This results in a quandary as the apps that do a good job of tracking metadata may not be able to deal with the non-destructive edits. This can be critical if you crop a person out of an image.
The workaround is that you always write out a new bitmap image from a serious edit. Ideally you have a script that looks for new NDEs and writes out an image based on this, copying the metadata from the master and at some point bringing it up for review for mods to the metadata.