7

I have been shooting digital for more that 10 years, now my photos are getting very huge. it was not a problem since I have a huge storage. But these huge photos began to bother me whenever something goes wrong with my system or physical failure in my hard disk also it bother me when I transfer my Photos from PC to another for upgrading or data retrieving. Some photos carry my best memories I want to keep. First I backup all my photos into DVD's and it is getting huge too and hard to search with it for a particular photo. also I brought external Hard drives but I think my photos are not safe too.

How the pros deal with that problem "today"?

I know my question seem to be a duplicate question mentioned in ( What method is best to take backups of your digital photos? )

The best answer tells "General rule: the more copies you have, the better." which mean "Copies of more and more Hard disks" which is not a good solution those days. the question and its answer are outdated, its from more than 3 years ago!! and certainly today there are some new technologies and web services that didn't exists in 2010. for example it didn't mention cloud backup and storage services and the (NAS) and the backup arrays. and now in 2015 no body depend on Hard disks only.

  • This isn't really a photography-specific problem; pretty much anyone with information they value has it. – Blrfl Jan 4 '15 at 20:46
  • 1
    I'm voting to close this. I think there are two potential questions: how to set up a good personal backup solution for a serious photography hobby; and what to actually do for a professional photography business. This question is couched in terms of the latter but really is asking about the former. That, in turn, ends up with answers that aren't practically useful for either case. – mattdm Jan 4 '15 at 21:02
  • 2
    @mattdm, disagree on the close, although there really are two questions here: "how do I manage all of these photos on my computer" and "how do I back them up and keep them safe". I don't think there are really good answers to both of those in his context on site right now. – chuqui Jan 4 '15 at 21:44
  • 4
    Define "Huge": 10GB? 1,000GB? 10,000GB? Petabyte? – Joel Coehoorn Jan 5 '15 at 1:12
  • 8
    They buy more space. – Michael C Jan 5 '15 at 4:25
8

Chase Jarvis has written about how he manages his backups. you can find it here: http://blog.chasejarvis.com/blog/2010/06/workflow-and-backup-for-photo-video/

Doing backups to DVD's works for smaller libraries, but as you've found, as it grows, it becomes impossible to maintain reliably. There are also issues with how long DVDs last before you risk starting to lose bits and possibly have them become unreliable, unless you use the more expensive archival disks. (see here to start: http://photo.net/digital-darkroom-forum/00W6MR but there's lots of discussion on this all over the net)

The beginning answer is to use three disks: Your data disk, a disk you back up your data disk to, and a second backup disk that you store offsite somewhere and rotate with your backup disk on a regular basis. Backing up disk to disk is the only real solution today for large data sets, and the only way to make sure your backups are reliable is to retire disks before they get too old (and fail), and to have a copy somewhere offsite for those catastrophic failures (like theft or the building burning down).

As your data size keeps growing, even managing disks becomes a pain. when I hit that point I went with a NAS (Network Attached Storage). Some info on that here: http://www.chuqui.com/consider-upgrading-home-network-nas/

And for general "how the heck do I set up a reliable backup?" you can start here: http://www.chuqui.com/want-know-backups-2013-edition/

Bottom line: multiple copies of the data you want to protect. At least one copy offline and offsite. Make sure you do backups regularly. the best practices that I've found is to use multiple drives, not trying to manage with DVDs or other optical media. It's less hassle, more reliable and most cost effective over time.

(cloud backups? I keep exploring them, I keep deciding it's not worth the hassle/cost yet. But it's coming. Your mileage my vary, but also take time to think about how you'll restore from a cloud backup and how long it'll take to replace a catastrophic, complete loss). Most cloud backups don't handle that as well as I want yet.

  • 1
    Regarding cloud backups, have you looked into CrashPlan? I've been using them for years and they're ridiculously cheap for what you get. I'm storing over 2TB with them all for their usual $6/month and they haven't complained yet. – fluffy Jan 5 '15 at 3:45
  • I can't vouch for CrashPlan, but I do note that a) they have both personal and business plans, and b) they are one of the providers that addresses chuqui's concern about disaster recovery with a "we'll ship you a hard drive" option. (And likewise you can send in a hard drive to get started rather than slowly uploading terrabytes.) Definitely something to look for in any cloud-based backup provider. – mattdm Jan 5 '15 at 5:19
  • 1
    I know people really happy with CrashPlan. also Backblaze. For me it's more complicated since I have to back up my NAS, and support for either is at best a nasty hack... – chuqui Jan 5 '15 at 7:15
  • Please don't forget that the NAS is no substitute for the extra remote backup (ie, a backup in a separate location than the one your system lives in. For example at your parent (or sibling)'s house, etc) – Olivier Dulac Jan 5 '15 at 11:55
  • For me (NAS) is a new idea. I have to stop a while thinking about all the valuable links that you mentioned. It will let me busy for a while :) – hsawires Jan 5 '15 at 14:06
4

My personal method of storage for the past 6 years has been the following; A raid 5 storage array. This requires an enclosure or dedicated standalone Server with a minimum of 3 hard drives where a certain percentage of each hard drive contains the parity for the other hard drives.

If any one of the hard drives fail, the system will continue working, you simply replace the broken hard drive and the new hard drive will auto build itself and bring the array back to a 3 hard drive configuration.

Most units will allow for more than 3 hard drives providing huge scalability for the future increase in storage requirements. The array in most cases will also allow for redundancies. Simply put, you can add extra drives in the enclosure that will only become operational once there is a hard drive failure as it will simply rebuild itself to form the complete storage space.

When the unit is running close to capacity, the user can add further drives. You can buy these in iSCSI or Fibre making the performance ideal for a realtime photo editing session.

To have such a configuration, unless you are IT savvy, you will need the help of a specialist as there are requirements for the understanding of Server Software, plus such a system is not cheap and will require the purchase of a high end PC or a basic server. The user may also wish to have his own personal DR site and have a second such system elsewhere mirroring the changes in realtime, or go for the below,

The next step to this; I recently took out a contract with an online rackspace for cloud storage. For a minimal fee, they back this array of mine as and when required by me. I can set it, real time, hourly, or weekly, it is very flexible.

Although, currently such a system runs in the Terabyte category, it has the potential to exceed well into the Petabyte range and serve for many years to come.

ADDITIONAL NOTES

On its own, a Raid 5 solution cannot be considered a true backup solution, and is only as secure as the premises it resides in and therefore is vulnerable and open to fires, theft and other disasters, thus the reasons mentioned above for a second unit elsewhere or cloud backup.

In the past, such storage units were backed up periodically onto Magnetic Tapes and stored in fireproof safes, a method which is still being used today by some banks and government establishments.

I am aware of photographers who use Mag Media, IE LTO Tapes.

What a Raid 5 Storage Array provides, is a very safe way of storing data locally. rather than have a 2 drives mirrored, it spreads this over multiple drives, with each drive having just enough Parity information about the other drives so that if one drive fails, the array continues to work and the faulty drive can easily be replaced. the more drives; the more secure. However, what you see on your PC/Mac, is one volume. Raid 5 has been the chosen method of storage throughout the industry for many years.

However, for the sake of Disaster Recover, it is imperative that this is backed up, but as Data has grown, so has the backup times and it is no longer acceptable to have long downtimes or systems running slow when being backed up. this has given rise to Realtime backup and SAN Storage and very quickly has become the De facto choice for all professionals.

These are basically a large Library of drives in a Raid configuration, to form one single volume(Disk) and replicated at another location to provide a proper Dr solution. As the users work, their work is always opened and closed within the SAN and nothing is ever stored locally on the Desktop. This means that there is no need for a timed backup as everything is always stored on multiple drives and replicated at another site providing a true real-time backup.

Most Professional agencies who require the Pro Photographer to submit their photos at the end of any session, usually have such a system of storage as they are the Copyright holders.

As for the photographer who works from his own studio, we need a similar system, but one which is cost effective but fast as we also need to do editing and with cheaper SAN servers not always being able to work at block level, brings the need for a more localised storage system, and therefore, the most secure of such systems is a Raid 5 Storage Array which allows for striping and mirroring.

As a standalone, it is the most secure storage method available, but for a true backup, you either need to replicate the storage to another site, or utilise the cloud to do auto backups.

Please note, a Raid 5 is only as fast as the slowest Drive installed.

As a final Note

This Raid 5 Storage method with online cloud back is one that I have chosen and is only one possible solution to the problem of ever growing data. If my data was growing at a slower rate, I may have considered using Raid 1+0 which theoretically would have been faster, but not as scalable as a Raid 5. My Decision was based on long term growth and ease of scalability.

In no uncertain terms, am I promoting or selling such a solution but I have encouraged a lot of my fellow photographers who have also benefitted from the same solution. some of us have even considered sharing our storage in a consolidated closed Network thus eliminated the need for cloud altogether.

  • Sorry, but I don't really see how RAID5 helps here, since you can't treat it as a proper backup method. And I doubt the asker actually needs the improved availability RAID5 offers in case of a drive failure. – mkataja Jan 5 '15 at 6:55
  • part of the problem is the drive failure, and what @anqphotography offer is a solution of local backup. and I think it is part of the solution building a local backup raid 5 array. at least when a drive fails it will be easy replacing it with another drive, without any worry about data transfer. – hsawires Jan 5 '15 at 7:05
  • 1
    If you decide to go with RAID5 just make sure to have some real backup method as well, or preferably two (one local, one off-site). RAID is not a backup solution. Of course it all depends on how much you value your data. – mkataja Jan 5 '15 at 7:29
  • 2
    RAID is not backup. RAID is adding redundancy to reduce the chance you need a backup at some point, but it doesn't remove the need for backups. – chuqui Jan 5 '15 at 8:16
  • 1
    None of them will have readable information. But in one case (single failure) you loose for RAID1 50% of the disks for RAID5 only 20% of the disks (from example above). Which of them is more durable? – Romeo Ninov Jan 5 '15 at 12:13
3

There are all sorts of professionals, so they do all sorts of things, but in general, losing photo archives is a business risk, and so should be treated as any such thing. Depending on the scale of your business, a commercial off-site backup contract is probably appropriate. Smaller operations — and particularly semi-pros, where photography isn't really the primary income source — might go with something DYI.

From your description, it sounds like you don't have business concerns. That's not necessarily bad, just different. The best enthusiast solutions don't necessarily look at all like scaled-down versions of professional IT. In fact, for personal use, there are great options not available to professionals — for example, Amazon Prime Photos allows unlimited storage of RAW and JPEG files, but explicitly forbids use in conjunction with any photography business.

  • if the photographer is pro he or she will use pro solution, so do not discard so easy the IT pro solutions. And it not depend if it is individual or company. It depend of amount and value of the information you need to store! – Romeo Ninov Jan 4 '15 at 20:33
  • @romeo I'm not discarding those solutions, just suggesting that they are likely to be irrelevant to non-businesses. And I strongly disagree that commercial or not does not matter. A commercial photography business may have legal and contractual obligations that just don't come into the picture, and on the other side, may not place any personal value on any of it. This, combined with an income stream linked to the data, means that "pro" solutions aren't necessarily of relevance to noncommercial photographers. (And we have plenty of other questions already about those, by the way.) – mattdm Jan 4 '15 at 20:41
  • you mention individual. And individual can be Pro, or just enthusiast. And beside legal part memories of one amateur can have bigger value than work items for pro :) – Romeo Ninov Jan 4 '15 at 20:49
  • 1
    That's exactly the point. If you are a business, a solution which covers liability (perhaps by delegating it to a third party) can be just fine. You might cover some of the risk by insuring against the monetary value of the loss — in the event of a disaster, cash will do. But for your literally priceless baby photos, that sort of solution isn't helpful. – mattdm Jan 4 '15 at 20:58
  • 1
    @JoelCoehoorn Yeah, that's ultimately why I decided to vote to close the question after providing a partial answer. The full answer is too broad to be a good Stack Exchange question. – mattdm Jan 5 '15 at 1:26
3

Currently, buying sale-price bare SATA drives (plus a case to store them in) is cheaper than anything else. DVD media of good provanance is much more expensive and horribly too small!

HDD is also by far more robust than optical media. Plus, being one piece to mount rather than thousands, it is practical to scan for errors on the backed up data, as an automated job.

Store a drive in a Pelikan case off site. My sister lost a lot of photos etc. In a house fire! My off-site backup protects against that.

As for simply having enough storage, I noticed in 2000 when consumer digital cameras were just becoming useful, that HDD size was increasing faster than camera resolution. Thus, I concluded that I don't have to search through removed media for photos, but can practically keep them on active storage.

For working pros, removed HDD is fine as it is easy to reconcile a job to a date and specific drive to mount. Home use is less structured, with the desire to flip through pages looking for something visually.

So a larger answer is: first, define your needs. How structured, how do you use the archives? What is the data space size of your archive now and moving forward? How valuable is it, and how much budget and effort is it worth spending on it? The answers are different for paid gigs, experiments/class exercises, grandkids' early years, and travel to well-trod tourist locations.

With those in mind first, you can then weigh available options.

  • 1
    "HDD is also by far more robust than optical media"? I do not think it means what you think it means... I've dropped dvd disks out of windows before and they still worked fine, I don't know of many hard drives that would survive as well. And Blu-ray disks can hold 25GB, 50GB, 100GB, or 128GB – Xen2050 Jan 5 '15 at 7:34
  • OK, they have different strengths. HDD have been known to be recoverable (by a pro) after being in house fires. Likewise for dropping: won't knock the bits off the platter. The platters will be recoverable from anything that doesn't breach the case or some really ridiculous shock (which I suppose would flatten the case anyway), super magnets (static field) or degaussing (don't know how resistant or what strength magnet is needed; only that military had trouble reliably zapping them in case of a plane being captured). – JDługosz Jan 5 '15 at 12:28
  • 1
    50GB: so it would take 60 discs to match the 3TB HDD I last bought (for under a hundred dollars). Manually swapping them and tying up the computer for 20 minutes write + 20 verify = 40 hour work week. I've never seen higher capacity discs at useful price, but I suppose they are coming. – JDługosz Jan 5 '15 at 12:32
  • These are for backups, recorded cd's & dvd's have a "predicted lifetime" of 10-20-100 years. I'd rather not spend a fortune on HD data recovery when it gets bumped & stops reading – Xen2050 Jan 5 '15 at 12:49
  • I've never had a harddisk suffer from dye-decomposition (fade), or trashed by a fingerprint, or suffer from a case of only being readable in the specific drive that created it (all problems I have had with optical media) and given that the cost for any decent full backup set to harddisk is the same price or cheaper than optical media, you can afford to treat HD's as a write-once media that takes up less space. – James Snell Jan 5 '15 at 20:51
3

Always remember the 3-2-1 rule of backups: 3 copies (including the original), in 2 formats, with at least 1 copy stored off-site. This means you're really looking at making two backups, or using a backup solution that stores multiple generations in the cloud.

You can rent cloud storage from Amazon starting at $.01/GB/mo. They handle backups and redundancy, and you can automatically mirror your photos with software you can get for free + Amazon's software + software already on your system using the instructions in the link. The link also has suggestions for when a turnkey service like CrashPlan can make more sense. The Glacier and CrashPlan options both handle both the off-site requirement and the 2nd format requirement, but for CrashPlan you may still want to look at another option for the third copy requirement (Glacier already handles redundancy - CrashPlan might, but it's not clear from their marketing material). This could be as simple as rsync to an external drive.

  • The problem is that when your data set starts being counted in Terabytes, how long does it take to do backups, and worse, get it restored? – chuqui Jan 5 '15 at 2:38
  • @chuqui I believe it only backs up the differential: new photos added since the last run, so taking backups does not take long. Restoring can take a while, but you hope to new need to do that. – Joel Coehoorn Jan 5 '15 at 2:42
  • @chuqui: I have multiple TB on CrashPlan, and once everything is backed up, everything is incremental. The kind of storage used by CP and its ilk makes periodic full backups unnecessary. (Unprofitable, too.) – Blrfl Jan 5 '15 at 3:56
  • @birfl oh, you can definitely do it, but how fast is your network? and if you ever need to do a 100% restore, how long will it take to come back over the network? days? weeks? is that really acceptable? not for me. (some services now will ship you a USB of your data, which is a good alternative). For me, with my network, none of this works yet. – chuqui Jan 5 '15 at 7:13
  • 1
    @chuqui: If you have a short enough RTO, you should be charging your clients enough that you can afford storage that's reliable and protected enough to avert most disasters and meet that figure for those that happen. The reality is that most of us aren't in that league. 100 GB comes back over a 10 Mbps link in less than a day. Selective restoration of what you need to work on right now will get you back in business even faster, and the rest can be filled in over time or by having the backup service ship you a drive. – Blrfl Jan 5 '15 at 19:07
2

And to extend the @mattdm answer:

  1. Some delete old photos. This is bad, very bad. You never know when you will need this info

  2. Some remove already created and provided to customer jpg files. But keep all raw, psd, xmp files to be able to reproduce the end result. This is not bad, but you loose time if you need to rebuild good amount of jpg from above files

  3. Some keep all. All the raw, psd and so on plus all the jpg files. This is for backup purpose because you never know when your storage will get broken and you will loose files

For the purpose of storage you can use a lot of different strategies. Currently price of GB memory is so small so there is no sense to delete any file. And to create better workflow and archiving strategy you can use so named layered storage:

Keep working files on fast device as

  1. SSD disk. Better on two of them and use some sync program to keep copies in sync
  2. Move finished projects to external storage. Better on two of them. And keep one of these storage offside
  3. Move old project to cloud. This can be counted as third copy, but you never know

P.S. And to answer to your last question: Pros extend storage :)

  • 1
    Even full-res customer jpgs won't save enough storage to make a difference if you're keeping raw originals and lossless final versions, never mind the in between stuff. – Chris H Jan 5 '15 at 12:38
  • 1
    I do not know what is your understanding of full-res jpg's, but mine are between 50 and 100% of size of my raw files. So if I delete them the total size will differ a lot – Romeo Ninov Jan 5 '15 at 12:53
  • So assuming you don't store any intermediate files you're reducing the total size by 1/3 to 1/2 - that doesn't appreciably change the type of backup solution you need (i.e. it might mean buying a larger hard drive, but doesn't switch you from hard drive to everything on a handful of DVDs; it might mean paying an extra $1 per month for cloud storage, but doesn't mean you'll get away with a free account.) – Chris H Jan 5 '15 at 19:32
  • 1
    @ChrisH, as IT pro I will choose technical solution :) one NAS (with RAID5 and spare disk support) will be OK for my needs :) But talking about layered storage this NAS will be just one intermediate layer. And some cloud storage as (almost) cold backup will be fine – Romeo Ninov Jan 5 '15 at 19:49
2

Use the cloud.

For 20 years I did all the multiple backups, dozens of DVD's, etc. locally. I now find using Dropbox much more practical. I have 7000 photos that take up 20 or 30 GB.

I was a little worried about eventually hitting the 100 GB limit with videos and the like... but that just got solved by my $7 a month plan now giving me 1000GB. Thank You Dropbox!

I have Dropbox linked to several computers so this also ensure multiple backups locally just by syncing them.

Time is money and I was spending (on average) more than an hour a month organizing and making backups. My time is worth more than $7 a hour a month.

  • Yeah, but... anything you store in the cloud, even with good encryption will be eventually be easy to de-crypt, as computing power grows and algorithms are being cracked. Cloud is good for transient backups (for increased availability) of data that expires. The cloud is more a dynamic environment than an archival environment. I suggest only archiving data that you don't mind getting public eventually. These are e.g. install disc backups, large databases and multimedia that is available as of now on the internet. – TFuto Jan 5 '15 at 12:37
  • 1
    Using dropbox you have to be careful - deletion of a file can also be synced. You can get round this by booting a synced machine with no network connection but it's a real risk. – Chris H Jan 5 '15 at 12:40
  • For sure cloud backup is one of the solutions that must not be neglected. but I think choosing one of the cloud services is a problem by itself. Regarding all security issues. I have to choose the most efficient service, shall I host my photos in a regular cloud service like Dropbox that may host documents, images ...etc. or better cloud system that is designed to host photos only as an (image service) like Flickr for example. both serveries have the ability to show photos. while one is better and bigger than the other when it comes with photos. I may prefer Flickr. – hsawires Jan 5 '15 at 14:01
  • 1
    I also recommend using the cloud for backups. My preferred setup is to have a full computer backup on a NAS or attached hard disk (this is something that your operating system can do for you) and then have everything synced to the cloud as well. I discovered (over the years) that backups have to be automatic -- if you end up having to do them yourself regularly then you will fall behind and end up in trouble. Of all the cloud solutions my favourite is OneDrive. It has clients available for pretty much all devices and, with a £8/mo Office subscription, has 'unlimited' space. – Richiban Jan 5 '15 at 15:30
  • Note that security and encryption are valid needs but are not mentioned in the question title or details and might best be answered by an answer that requires them. – Michael Durrant Jan 5 '15 at 22:18
2

Pre the advent of digital photos, professionals stores photos as negatives and prints and often prints were stored at the same premises. There were instances where a photographer lost his life's work in a fire - it happened to a friend of mine.

The only secure storage was to lock the negatives up in a vault.

You have much better options now days for storing digital images.

Not the answer you're looking for? Browse other questions tagged or ask your own question.