What did pros do when their photos storage getting huge? [duplicate]

Question

I have been shooting digital for more that 10 years, now my photos are getting very huge. it was not a problem since I have a huge storage. But these huge photos began to bother me whenever something goes wrong with my system or physical failure in my hard disk also it bother me when I transfer my Photos from PC to another for upgrading or data retrieving. Some photos carry my best memories I want to keep. First I backup all my photos into DVD's and it is getting huge too and hard to search with it for a particular photo. also I brought external Hard drives but I think my photos are not safe too.

How the pros deal with that problem "today"?

I know my question seem to be a duplicate question mentioned in ( What method is best to take backups of your digital photos? )

The best answer tells "General rule: the more copies you have, the better." which mean "Copies of more and more Hard disks" which is not a good solution those days. the question and its answer are outdated, its from more than 3 years ago!! and certainly today there are some new technologies and web services that didn't exists in 2010. for example it didn't mention cloud backup and storage services and the (NAS) and the backup arrays. and now in 2015 no body depend on Hard disks only.

Answer 1

Chase Jarvis has written about how he manages his backups. you can find it here: http://blog.chasejarvis.com/blog/2010/06/workflow-and-backup-for-photo-video/

Doing backups to DVD's works for smaller libraries, but as you've found, as it grows, it becomes impossible to maintain reliably. There are also issues with how long DVDs last before you risk starting to lose bits and possibly have them become unreliable, unless you use the more expensive archival disks. (see here to start: http://photo.net/digital-darkroom-forum/00W6MR but there's lots of discussion on this all over the net)

The beginning answer is to use three disks: Your data disk, a disk you back up your data disk to, and a second backup disk that you store offsite somewhere and rotate with your backup disk on a regular basis. Backing up disk to disk is the only real solution today for large data sets, and the only way to make sure your backups are reliable is to retire disks before they get too old (and fail), and to have a copy somewhere offsite for those catastrophic failures (like theft or the building burning down).

As your data size keeps growing, even managing disks becomes a pain. when I hit that point I went with a NAS (Network Attached Storage). Some info on that here: http://www.chuqui.com/consider-upgrading-home-network-nas/

And for general "how the heck do I set up a reliable backup?" you can start here: http://www.chuqui.com/want-know-backups-2013-edition/

Bottom line: multiple copies of the data you want to protect. At least one copy offline and offsite. Make sure you do backups regularly. the best practices that I've found is to use multiple drives, not trying to manage with DVDs or other optical media. It's less hassle, more reliable and most cost effective over time.

(cloud backups? I keep exploring them, I keep deciding it's not worth the hassle/cost yet. But it's coming. Your mileage my vary, but also take time to think about how you'll restore from a cloud backup and how long it'll take to replace a catastrophic, complete loss). Most cloud backups don't handle that as well as I want yet.

Answer 2

My personal method of storage for the past 6 years has been the following; A raid 5 storage array. This requires an enclosure or dedicated standalone Server with a minimum of 3 hard drives where a certain percentage of each hard drive contains the parity for the other hard drives.

If any one of the hard drives fail, the system will continue working, you simply replace the broken hard drive and the new hard drive will auto build itself and bring the array back to a 3 hard drive configuration.

Most units will allow for more than 3 hard drives providing huge scalability for the future increase in storage requirements. The array in most cases will also allow for redundancies. Simply put, you can add extra drives in the enclosure that will only become operational once there is a hard drive failure as it will simply rebuild itself to form the complete storage space.

When the unit is running close to capacity, the user can add further drives. You can buy these in iSCSI or Fibre making the performance ideal for a realtime photo editing session.

To have such a configuration, unless you are IT savvy, you will need the help of a specialist as there are requirements for the understanding of Server Software, plus such a system is not cheap and will require the purchase of a high end PC or a basic server. The user may also wish to have his own personal DR site and have a second such system elsewhere mirroring the changes in realtime, or go for the below,

The next step to this; I recently took out a contract with an online rackspace for cloud storage. For a minimal fee, they back this array of mine as and when required by me. I can set it, real time, hourly, or weekly, it is very flexible.

Although, currently such a system runs in the Terabyte category, it has the potential to exceed well into the Petabyte range and serve for many years to come.

ADDITIONAL NOTES

On its own, a Raid 5 solution cannot be considered a true backup solution, and is only as secure as the premises it resides in and therefore is vulnerable and open to fires, theft and other disasters, thus the reasons mentioned above for a second unit elsewhere or cloud backup.

In the past, such storage units were backed up periodically onto Magnetic Tapes and stored in fireproof safes, a method which is still being used today by some banks and government establishments.

I am aware of photographers who use Mag Media, IE LTO Tapes.

What a Raid 5 Storage Array provides, is a very safe way of storing data locally. rather than have a 2 drives mirrored, it spreads this over multiple drives, with each drive having just enough Parity information about the other drives so that if one drive fails, the array continues to work and the faulty drive can easily be replaced. the more drives; the more secure. However, what you see on your PC/Mac, is one volume. Raid 5 has been the chosen method of storage throughout the industry for many years.

However, for the sake of Disaster Recover, it is imperative that this is backed up, but as Data has grown, so has the backup times and it is no longer acceptable to have long downtimes or systems running slow when being backed up. this has given rise to Realtime backup and SAN Storage and very quickly has become the De facto choice for all professionals.

These are basically a large Library of drives in a Raid configuration, to form one single volume(Disk) and replicated at another location to provide a proper Dr solution. As the users work, their work is always opened and closed within the SAN and nothing is ever stored locally on the Desktop. This means that there is no need for a timed backup as everything is always stored on multiple drives and replicated at another site providing a true real-time backup.

Most Professional agencies who require the Pro Photographer to submit their photos at the end of any session, usually have such a system of storage as they are the Copyright holders.

As for the photographer who works from his own studio, we need a similar system, but one which is cost effective but fast as we also need to do editing and with cheaper SAN servers not always being able to work at block level, brings the need for a more localised storage system, and therefore, the most secure of such systems is a Raid 5 Storage Array which allows for striping and mirroring.

As a standalone, it is the most secure storage method available, but for a true backup, you either need to replicate the storage to another site, or utilise the cloud to do auto backups.

Please note, a Raid 5 is only as fast as the slowest Drive installed.

As a final Note

This Raid 5 Storage method with online cloud back is one that I have chosen and is only one possible solution to the problem of ever growing data. If my data was growing at a slower rate, I may have considered using Raid 1+0 which theoretically would have been faster, but not as scalable as a Raid 5. My Decision was based on long term growth and ease of scalability.

In no uncertain terms, am I promoting or selling such a solution but I have encouraged a lot of my fellow photographers who have also benefitted from the same solution. some of us have even considered sharing our storage in a consolidated closed Network thus eliminated the need for cloud altogether.

Answer 3

There are all sorts of professionals, so they do all sorts of things, but in general, losing photo archives is a business risk, and so should be treated as any such thing. Depending on the scale of your business, a commercial off-site backup contract is probably appropriate. Smaller operations — and particularly semi-pros, where photography isn't really the primary income source — might go with something DYI.

From your description, it sounds like you don't have business concerns. That's not necessarily bad, just different. The best enthusiast solutions don't necessarily look at all like scaled-down versions of professional IT. In fact, for personal use, there are great options not available to professionals — for example, Amazon Prime Photos allows unlimited storage of RAW and JPEG files, but explicitly forbids use in conjunction with any photography business.

Answer 4

Currently, buying sale-price bare SATA drives (plus a case to store them in) is cheaper than anything else. DVD media of good provanance is much more expensive and horribly too small!

HDD is also by far more robust than optical media. Plus, being one piece to mount rather than thousands, it is practical to scan for errors on the backed up data, as an automated job.

Store a drive in a Pelikan case off site. My sister lost a lot of photos etc. In a house fire! My off-site backup protects against that.

As for simply having enough storage, I noticed in 2000 when consumer digital cameras were just becoming useful, that HDD size was increasing faster than camera resolution. Thus, I concluded that I don't have to search through removed media for photos, but can practically keep them on active storage.

For working pros, removed HDD is fine as it is easy to reconcile a job to a date and specific drive to mount. Home use is less structured, with the desire to flip through pages looking for something visually.

So a larger answer is: first, define your needs. How structured, how do you use the archives? What is the data space size of your archive now and moving forward? How valuable is it, and how much budget and effort is it worth spending on it? The answers are different for paid gigs, experiments/class exercises, grandkids' early years, and travel to well-trod tourist locations.

With those in mind first, you can then weigh available options.

Answer 5

Always remember the 3-2-1 rule of backups: 3 copies (including the original), in 2 formats, with at least 1 copy stored off-site. This means you're really looking at making two backups, or using a backup solution that stores multiple generations in the cloud.

You can rent cloud storage from Amazon starting at $.01/GB/mo. They handle backups and redundancy, and you can automatically mirror your photos with software you can get for free + Amazon's software + software already on your system using the instructions in the link. The link also has suggestions for when a turnkey service like CrashPlan can make more sense. The Glacier and CrashPlan options both handle both the off-site requirement and the 2nd format requirement, but for CrashPlan you may still want to look at another option for the third copy requirement (Glacier already handles redundancy - CrashPlan might, but it's not clear from their marketing material). This could be as simple as rsync to an external drive.

Answer 6

And to extend the @mattdm answer:

1. Some delete old photos. This is bad, very bad. You never know when you will need this info

2. Some remove already created and provided to customer jpg files. But keep all raw, psd, xmp files to be able to reproduce the end result. This is not bad, but you loose time if you need to rebuild good amount of jpg from above files

3. Some keep all. All the raw, psd and so on plus all the jpg files. This is for backup purpose because you never know when your storage will get broken and you will loose files

For the purpose of storage you can use a lot of different strategies. Currently price of GB memory is so small so there is no sense to delete any file. And to create better workflow and archiving strategy you can use so named layered storage:

Keep working files on fast device as

1. SSD disk. Better on two of them and use some sync program to keep copies in sync
2. Move finished projects to external storage. Better on two of them. And keep one of these storage offside
3. Move old project to cloud. This can be counted as third copy, but you never know

P.S. And to answer to your last question: Pros extend storage :)

Answer 7

Use the cloud.

For 20 years I did all the multiple backups, dozens of DVD's, etc. locally. I now find using Dropbox much more practical. I have 7000 photos that take up 20 or 30 GB.

I was a little worried about eventually hitting the 100 GB limit with videos and the like... but that just got solved by my $7 a month plan now giving me 1000GB. Thank You Dropbox!

I have Dropbox linked to several computers so this also ensure multiple backups locally just by syncing them.

Time is money and I was spending (on average) more than an hour a month organizing and making backups. My time is worth more than $7 a hour a month.

Answer 8

Pre the advent of digital photos, professionals stores photos as negatives and prints and often prints were stored at the same premises. There were instances where a photographer lost his life's work in a fire - it happened to a friend of mine.

The only secure storage was to lock the negatives up in a vault.

You have much better options now days for storing digital images.