1

I've been shooting RAW+JPEG for years, but about the only time I use the RAW files is to play around with RAW processing techniques. For everything else, I use the JPEGs. I have several multi-TB hard drives that are running out of space. Instead of buying yet another hard drive, I'm considering running the following command:

# enable double asterisk for recursing directories
shopt -s globstar

# old command
# \rm -r **/*.raw **/*.arw **/*.crw **/*.cr2 **/*.raf **/*.nef **/*.pp3

# only delete files with an associated jpg
for i in **/*.jpg ; do
   f = "${i%.jpg}"
   for e in raw arw crw cr2 raf nef pp3 xml ; do
      rm "$f.$e" "$f.$e.pp3" "$f.$e.xml"
   done
done

Although I expect to feel an initial sense of loss, I likely won't miss them in the long run because I don't use them. I've already deleted some older raws, and so far the only noticeable repercussion has been increased disk space.

What are the pros and cons? Anything I'm missing? Perhaps this is something other packrats might consider trying?

  • 1
    Anything I'm missing? In the code? In the sense of "nooooo, don't delete all your RAWs!"? Generally, I think that this is a topic somewhere between "What are the pros and cons when shooting in RAW vs JPEG?" and one's own opinion on "Do I need my RAWs any more or is space more valuable?" Personally, I'm in the process of transforming all of my non-critical RAWs (from holidays, work,...) to JPEGs - and then deleting them. – flolilo Apr 22 at 9:57
  • I don't know. If I did, I wouldn't have missed it. – xiota Apr 22 at 9:59
  • cyberduck + backblaze b2. That way if you want to re-edit some raw using some new method/software, you just get it from there. If you never use it, you barely pay for it. 100gb per month of upload (every month) will cost you about 40$/year, with the advantage of data integrity (local HD -> bit rot in a few years) – Fábio Dias Apr 22 at 14:53
  • @ths This question exaggerates what I'm thinking about doing. Though I might feel like... "to digital hades with it" ... My approach will be more moderate. – xiota Apr 22 at 20:52
  • I don't recognize what the double astrisk ** is supposed to do in your example? Normally * is any set of characters, so what is that twice? If you're determined to do this, I'd suggest moving "mv" the files to another media first then delete from there just incase there's an Oops. – user10216038 Apr 26 at 20:09
5

Pros: You gain some space.

Cons: You won't ever be able to change your mind.

A 4TB drive is about $£€70 these days.
Buy one, stack all your old data on it & put it on a shelf. If the drive does die eventually, then at that point you'll probably regret not having a backup of it - but as you were going to throw them away beforehand, you ought to get 5 - 10 years of 'I wonder what happened to that pic... ahh, there it is' out of it, for something like the price of a month's TV/Internet access.

Personally, I'm not a fan of rm -r ... too many horror stories along the lines of "Oops, wrong path!". I'd be more inclined to find an alternative way to get them into the Trash if you're going to do it, then you've got an undo.

  • "I wonder what happened to that pic... ahh, there ['s the JPEG]." – xiota Apr 22 at 10:28
  • Now that I'm thinking about it, the -r is superfluous because of the **. – xiota Apr 22 at 10:37
  • @xiota If you're really planning to do it, check whether the extension is CR2 or cr2... My environment has uppercase file names, including the extension. – juhist Apr 22 at 10:40
  • @juhist - I batch rename everything to lowercase. I'll probably start clearing out raw files, but not all at once. Like, who needs raw files of bird and flower photos? – xiota Apr 22 at 10:45
5

The biggest thing that is missing from a work flow of "Just blast it all away from the command line" is that of conscious control. And deleting your raw files to save space might leave you missing the forest for the trees...

  • Are you actually deleting what you think you are deleting?
  • Are you actually saving the files you really want to keep anyway?

First step to dealing with file-pack-rat-syndrome is to begin dealing with it before it happens on new files:

  • Cull early, cull often.
  • Use a modern image library management system, such as lightroom, and rate, flag, and keyword images early on. From here you can develop a workflow scheme that helps you clearly decide which raw files you wish to keep.
  • Flag images with little to no value early on for deletion. [Ideally before you even bother keywording them.] If you have ten near duplicates of the same image with "slightly different lighting", then do you really want all ten of them?

Pick the best, trash the rest.

The important thing to remember is that images that you have no idea exist, or an easy way to find if you want them, have little to no value: Either ensure you establish a value in their storage. Keywording is your friend.

In my mind culling and maintaining a collection is an ongoing process. As new images are added, or skills, methods, or tastes change, then older images may become obsolete. Culling those of "Poor technical merit" is easy to do at time of ingest, and some artistic culling is also good to do at the time. But returning to the images for further review weeks, months, and years later is worth the effort to avoid an ever ballooning collection.

  • If you build your personal collection to the point that you will never revisit 99.9% of the images in your life, then why are you hanging on to them and cluttering your drive? If a photo is not a "Good image" in your mind, or at least a "Good example of your past artistic views", then it is worth asking why is it being carted around as part of your collection?

As you put strong systems in place to keep from rapidly expanding your new images, you can begin addressing your old images in smaller bite sized chunks with confidence.

  • Importing a new batch of images from the weekend? Process and cull one of your archive folders while you're at it.
  • Have some spare time? Make a cup of tea or coffee, and sit down to run through old data to cull and keyword.

The important thing is to make deletion of files an active thought process, and massage random unsorted images into a usable collection of images, rather than gathering terabytes of meaningless data that holds little chance of ever actually being used at any point in the future.

Sorting and culling doesn't have to be a massively long process with minutes or hours of debating on each and every photo, it is perfectly fine to scroll through a folder from an afternoon of shooting and declare none of them are worth bothering with, but the active review of your collection is what improves it as a collection.

Avoid "Blind deletions" unless you're critically pressed for space, and avoid keeping even just the .jpegs "just because...". [

Blindly deleting all of your raw image data for the sake of storing thousands more .jpeg files that you'll never actually use or care about anyway hasn't actually put you any further ahead in the grand scheme of things.

The road out of file-pack-ratted-ness is not one of merely 'saving space', but one of addressing what the space you're using is actually being used for!

  • 1
    Ditto, but don't cull too early. After a couple of weeks, the picture looks like having being taken by another person ("why did I took this?). Culling is much more efficient then. – xenoid Apr 22 at 17:28
  • @xenoid - Do you feel like the answer could be improved with expansion on "Cull early, cull often"? My view is that culling is an ungoing process, like trimming a flowering hedge - In the beginning there are obvious cuts to be made, but more is done over time to keep a collection in good shape. – TheLuckless Apr 22 at 17:47
  • In practice you won't cull something you have put on an archive disk. I do keep some pictures "by topic" (animals, planes...) so I can delete old pics if get something better later but even this has its limits. – xenoid Apr 22 at 17:53
  • @xenoid - Edited the middle to expand on my point. Do you feel that makes my long term anti-pack-rat point of view a little more clear and useful? – TheLuckless Apr 22 at 19:45
  • "As ... tastes change, then older images may become obsolete". Careful with that one: fashion goes in cycles which are much shorter than an average person's lifetime, and personal tastes can do the same. (I've elided the rest of that sentence because I agree with the other subpoints). – Peter Taylor Apr 28 at 7:59
2

It's tempting to say that this is the equivalent, for a film photographer (like me) of destroying your negatives and keeping only the prints. This is something I would never do, because I can make more prints from the negatives, but without the negatives I'm pretty much stuck.

But that's wrong. If I destroy my negatives the only way I can make copies of pictures I have is by some horrible process of making photographs of the prints and printing from those. Those second-generation prints will not be as good as the first-generation ones, especially if I intentionally printed the first-generation print very dark or otherwise deliberately lost information that was in the negative. On the other hand, if you have a JPEG, you can make endless perfect copies of it, and if you are happy with that JPEG as an image then you never need to worry about the raw file. Even if you're not happy with it you can manipulate it to some extent.

So I think that no-one can answer this question but you: are you happy with the JPEGs? Do you think you will still be happy with them in ten years? Is there any possibility anyone else will care that you've destroyed these irreplaceable things once you are some famous dead photographer, or will it just make you more famous because of the mythical lost raw files? These are not questions anyone can answer for you, unfortunately.

I would think hard before doing it however.


How to do this on a Unixoid system

If you are making huge changes like this, then you typically want to do it in a very careful way. A good approach is to do a dry run which simply lists all the files you want to delete, and then do a real run which deletes them. You also need to do things in a way which is very robust against peculiar filenames. For the sake of argument let's say you want to:

  • delete raw files under /myphotos;
  • delete raw files with names like *.dng & *.DNG.

Then here is a process which should be reasonably safe.

  1. Be in some safe working directory.
  2. Make a list of the files you will delete: find /myphotos \( -name '*.dng' -o -name '*.DNG' \) -print > files-to-delete.
  3. With a text editor look carefully through files-to-delete & make sure it is what you actually want to delete.
  4. Delete the files (this step is irrevocable): find /myphotos \( -name '*.dng' -o -name '*.DNG' \) -print0 | xargs -0 rm. Here the -print0 and xargs -0 will ensure that files with odd characters in their names are handled properly.
  5. Check you have done what you thought: run the find command from (3) again and there should be no output.

Note that I am not responsible for the safety of this: read the manuals & check carefully before making wholesale, irrevocable, changes like this.

  • More likely, would remain forever unknown because accidentally typed rm -r **/* .raw... – xiota Apr 22 at 10:40
  • Bad example. A better analogy might be what do I do with two negatives of every shot. One negative yields a great print with no burning, dodging, flashing, push processing, breathing on parts of the print to bring out detail, etc. The second negative needs burning, dodging, long exposures with printing contrast filters and multi-grade paper, etc. Do you keep both negatives of every shot now? – Stan Apr 22 at 14:33
  • @Stan In a world where people don't generally make prints but look at things on screens, the JPEG is the nearest thing there is to a print. (Also, I bet that if people do make physical prints from image files they still have to adjust them since paper is so different than a screen). – tfb Apr 22 at 14:51
  • Not every negative is worth keeping. When I still shot film, there were lots of negatives I threw away. With digital, there are lots of photos I trash both RAW and JPG. – xiota Apr 22 at 19:00
  • 2
    Where once a well crafted photo was valued like a well written poem, song, or short story, today it's seen as just more noise among endless tweets and snapchats. – Michael C Apr 22 at 22:47
1

My personal way of doing this:

I shoot RAW only - it is a bad habit, but since it gives me some peace of mind, I really do not care too much about changing it.

My library has three main folders (in descending file count): private (for holidays, pets, ...), professional (for work that is/was worth money), and projects (for larger, planned, [non-commercial] projects).

One day, I realized that I had roughly 500 GiB in RAWs on my hard drive. I looked through all photos and came to the conclusion that neither pet and holiday photos nor professional press photos need to be RAW - a decent JPEG will do for all of them (and is far more accessible).

Ever since then, I took Capture One (now: DigiKam) and went through all of the pictures. I delete whatever I do not like completely - and "develop" whatever is good enough to keep. Only pictures that in my opinion could be developed further artistically will be kept as RAW (without JPEG, typically). Whenever I feel the need to procrastinate, I try to go through a few of my photos.


If you are absolutely certain that you do not want any of the RAWs...go for it. Just nuke them - if there is a JPEG for every RAW and the JPEG is all you want, you will not lose a thing.

This raises a potential problem: What if a photo does not have a (working) JPEG? If you just delete all RAWs indiscriminately, there is a certain risk that you lose some important files.

Personally, this is why I do this on a picture-by-picture basis. It costs a lot of time, but then again, I get to appreciate almost forgotten treasures - and I can get rid of some of the worst photos.

0

No, don't just go deleting a huge amount of RAW files. If you haven't categorized the shots, purchase a large HDD and store the RAW files there. In the long term, you should consider categorizing your shots better, if not done already.

If you indeed have categorized the shots, I would keep the RAW files for those shots that you either:

  1. find extremely valuable
  2. find somewhat valuable and need some substantial processing done for the image apart from simple cropping

I find it hard to believe that you have that many shots that fulfill either of these two criteria. So, the space savings for shots not belonging to either category (1) or category (2) are going to be pretty significant.

In my case, I have found that quite many shots actually belong to category (2). The camera doesn't always choose the correct white balance and exposure. The exposure is probably the most typical adjustment I make for the RAW files.

Also, you should find out if your camera does lens corrections (vignetting, chromatic aberration, geometric distortion) automatically to JPG files. I don't know if this is the case for my camera, because I find slight exposure adjustments need to be done for about 50% of my shots, so I use RAW always and the JPG is there only for easily previewing the shots.

About 5% of my shots actually require so significant exposure adjustments that the JPG files are basically useless.

  • 2
    Have you considered using exposure compensation? Or changing the metering mode? – xiota Apr 22 at 10:47
  • @xiota When shooting fast-moving subjects such as flying birds, exposure compensation doesn't help (don't have time to use it), but I'll have to see whether metering mode could be adjusted. It may be the case the camera is so cheap that it can't be changed... – juhist Apr 22 at 10:48
  • I change exposure all the time with JPEG. Usually, I keep it to a fraction of a stop, but a couple stops is manageable. The problem with JPEG is when edits are stacked and banding occurs. – xiota Apr 22 at 10:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.