1

TL;DR: I am amateur photographer and looking for advice on storing my photographs locally (not on cloud/online) for long-term.

Background

I started clicking photos in .JPG initially but realized it's limitation because I do lot of post processing. Then I started taking photos in RAW. I am taking photos in RAW for a 4 years now. It resulting into huge amount of photographs (around 2 TB). Looking at rate at which I am taking photos, soon I will be out of space.

Current State

I am currently using WD My cloud 4TB which gives me enough space plus continuous backup from my laptop. Plus I have 2TB WD my passport ultra hard disk which I use for daily transfer and moving files from one workspace to another. I use Adobe Lightroom 5 to make collections and creating libraries. However looking at current rate of capturing pictures, I will be out of space after year and half :(

Specific Question

How should I manage this space crunch efficiently as well as economically ?

Current options

  • One option is to convert RAW files to high quality JPG (will give me additional year before space crunch)
  • Delete trial shots and unwanted pictures (Less preferred by me, but it's last resort)

P.S: Points worth mentioning, I am not a professional photographer (at least not on my list of career options yet :P). I do lot of timelapse photography also (but I convert these images into JPG)

  • Main difference is I am not professional, I don't think I can afford options mentioned there like RAID :( – Dexter Oct 15 '15 at 8:03
  • 2
    A RAID costs about 300 dollars over the price of the HDDs, check Synology 4-bays or HP microserver gen. 8, basic model.with less than 700 dollars you get additional 4 TB (mirrored) with space for additional 2 disks. – FarO Oct 15 '15 at 11:37
2

As I think you realise, you've got three options:

  1. Reduce the number of photos you store (i.e. delete some photos)
  2. Reduce the size of each photo (i.e. convert from RAW to JPEG)
  3. Buy more storage

The balance between 1 and 2 can depend a lot on what sort of photography you're doing: the photojournalist's rule is "never delete a photo, because you never know when it might turn out to be interesting" - there are some famous stories about photos of Bill Clinton and Monica Lewinsky taken before the scandal broke. On the other hand, if you've got 10 shots of a baseball player during a swing where 1 of them has the ball on the bat and the other 9 don't, you're never going to do anything with those 9 shots.

Buying more storage obviously costs money, although you can consider "tiered storage" - have multiple copies of the better photos (e.g. on a RAID array) but only single copies of lower quality photos (e.g. on a single hard drive).

1

If buying additional space is expensive for you, consider the following actions:

  • Obtain a good photo manager
  • Remove blurry, duplicate and bad pictures
  • Sort your photos by importance
  • Convert less important pictures to high-res JPEG or/and burn it to DVDs

Of course this will require a lot of time investments, but remember that a picture that you can't find is as useless as the one that was lost.

0

Since you're doing a lot of timelapse photography, it is worthwhile to do think about lossless compression algorithms for the stacks belonging to the same timelapse scenes. I'm not sure if programs that do this already exist. If not then it shouldn't be all that difficult to develop your own algorithm. You can e.g. think of using the dcraw to convert the raw files to "totally raw" tiff files which contain only the raw sensor data without any further processing like demosaicing being done. Then you can remap the files in each stack by aligning them (you then need to split the files by collecting together the pixels that belong to the same color components) and adjust the brightness so that each remapped stack is highly compressible. If the remapping is done in a reversible way, you can recreate the original stacks from the compressed remapped stacks.

How best to do the compression will require some research. A simple method is to subtract from each remapped stack the first image in the stack, make the gray values of each image positive by adding the negative of the smallest negative gray value (since most raw files are 14 bits or less you have enough room to do this without exceeding the 16 bit range) and then compressing the image as a 16 bit image using PNG compression.

  • Can you explain little bit about dcraw ? That looks interesting. – Dexter Oct 16 '15 at 11:20
  • 1
    Re: your suggestion to use PNG. PNG has a nice feature for long-term archival storage. If you store two copies of a PNG, and both suffer a small amount of bit rot after a few decades, a good PNG can be pieced together from the good parts of two degraded ones, as long as the same IDAT chunk isn't broken in both copies. Each chunk has a "CRC" (checksum) so you will be able to identify which chunks have gone bad. – Glenn Randers-Pehrson Oct 17 '15 at 18:56
  • Noise is probably the main enemy of the frame-to-frame compression scheme you suggest – mattdm Oct 18 '15 at 12:40
0

My options are

1) Convert the raw images to DNG ones. They are still in raw format but use less space, and can be previewed more easily.

2) Do take decisions.

  • While taking pictures.

We offen take too many duplicated images just becouse, and also becouse we know that our brand new card can storage tons of photos.

This can force us to better think (or feel) our shoot. "Is my horizon horizontal?" Oh I dont know... Ill take another one" Think of that from the beguinning.

  • Does this situation really need a raw image?

Is it just an informal reunion where you just need a selfie or is it a outstaunding landscape, or a portrait for a client.

I started taking pictures in raw for compositing them later for a 360 view... I never use them.

  • Deleting bad photos.

This is a really tricky business. You can prepare a temporal folder and move the bad photos there. You now can take a second look to confirm all that is really for the trash.

Prepare your recicled bin in case you need to recover a photo. Never delete them from the memory card.

3) Have a good category system. Not only by date. It is more usefull by topic and after by date.

4) In my opinion now days is better to storage files on an external drive rather than DVD.

But for really special photos. The studio photos of your baby, your son's wedding. Make an aditional backup on a good quality DVD. A bad quality one can be errased just exposing them into a sun light becouse cheap ones are dye based.

5) Print some photos. Yes you can have a 100 year old paper photo. I am not sure about a 100 year old hard drive or even a dvd yet.

6) If you do some little comercial work, save a small fee for storage. You will be out of space in a year... start saving 10 bucks a month.

0

Here are a few things I do:

1) archive 2) rationalize 3) de-duplicate 4) worry less

Archive: So by this I mean the following: I archive photos that are old or that I no longer use/edit/look at etc. I do this two ways: 1) I move old images to a portable hard drive, and store it in my safe. I also publish photos on Flickr so that acts as a archival storage as well (Google Photos is good at this as well, but it compresses images). This way my main storage (NAS) and the related Lightroom databases are more manageable. I can't remember the last time I looked at those images from 2005, but I do know where they are.

Rationalize: I periodically go thru my folders to remove images that are bad, out of focus, etc. I have gotten in the habit of using the 'x' key in Lightroom, which tags those images as 'ready for deletion', and then later you can ctrl+backspace to permanently remove them.

De-duplicate: this isn't exactly what it sounds like, because I am not removing duplicate photos, but I am removing JPGs. Since I have the RAW files, and the Lightroom database, I simply do not need the JPG images, as they can be created at any time. So, any JPG I have created with Lightroom are deleted, and duplicates created by the camera are deleted as well. The only JPG I save are those that are original, when I did not shoot RAW.

Worry-less: Finally, I now worry less about it all, because I do one thing that you removed from consideration at the beginning: I backup to the cloud. Everything I have mentioned here, all images, including rejected, deleted, duplicates...all of it, is backed up to Crashplan. If anything goes missing, I know there is a copy in Crashplan. I even backup the NAS. No, I don't like paying for it, but at $60, its fantastic peace of mind: I know that everything is safe, even it the house burns down. Consider it.

0

Some things of interest:

best value in storage media

Bare drives, 3.5-inch SATA drives on special sale, are the best value. Much cheaper than another travel drive. Get a dock and pick up a new full-warranty but sale-price drive when the need arises. A fancy dock will set you back $30, and a cheap cable-only set with separate power brick might be found for $12.

Use a bare drive for backups, including off-site backups and overflow storage.

reduce stored files

You can delete duds, and long bursts of useless photos. Mark them as uninteresting, and months later drop them if you still feel that way.

The DNG has a compressed option. You can compress the great quantity of uninteresting shots without deleting them, "just in case". This will preserve more shadow detail and range, and not ruin them like converting to jpeg. I'm not sure about reducing resolution.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.