24
\$\begingroup\$

My early (Canon G2) photos are all JPG, but when I got my Nikon D90 I initially shot in JPG, then switched to RAW+JPG, and now I would like to switch to RAW only.

I have literally thousands of photos on my HDD. The photos are in sub-directories (by date) under a single directory called Import.

I am about to import all these photos into Lightroom 3.0, however, I would like to delete all the JPG files, but only where there is already a corresponding RAW file (ie. I no longer want to keep JPG and RAW versions of the same file).

If I can do this easily within Lightroom (after importing everything, including the duplicate JPG files) that would be great. It would also be OK if there were an easy way to do this before importing the files (but hopefully this wouldn't involve having to visit every directory looking for filenames with both JPG and NEF extensions).

Does anybody know of a way to do this (in Lightroom, or with some tool/script in Windows)?

\$\endgroup\$
10
  • \$\begingroup\$ Do all of your jpg files and RAW files have at a minimum the same string(other characters could be added to either)? Eg IMG_1234_portrait_picture.jpg & IMG_1234.CR2. \$\endgroup\$
    – dpollitt
    Oct 15, 2011 at 3:28
  • \$\begingroup\$ I think that all of my JPG/NEF pairs have exactly the same filename (except for the extension). \$\endgroup\$
    – seanmc
    Oct 15, 2011 at 3:29
  • 3
    \$\begingroup\$ I will vote to move this to Stack Overflow, where you should get an answer within minutes =) \$\endgroup\$
    – anon
    Oct 15, 2011 at 15:17
  • 1
    \$\begingroup\$ @anon: How exactly does this belong on StackOverflow? This is most definitely an on-topic question here, as it deals with image management and photo editing tools. Outside of a tangential reference to script...this has nothing to do with programming. \$\endgroup\$
    – jrista
    Oct 15, 2011 at 17:04
  • 2
    \$\begingroup\$ I'm voting to close this question as off-topic because this isn't really about photography, it is about managing files that just happen to be photographs. The question and answer would be the same if the two types of files in question were any other types of files that could be translated from one form to another, such as .doc and .pdf files, etc. \$\endgroup\$
    – xiota
    Aug 31, 2018 at 4:51

11 Answers 11

29
\$\begingroup\$

On Windows, go to the folder, and run this in a command prompt:

for /f "delims==" %r in ('dir /b *.nef') do del "%~dpr%~nr.jpg" 2> nul

Basically, it goes through the current folder, runs through the NEF files, and deletes the JPG if present. It ignores any errors if the JPG is not there.

If you want subfolders, include /s in the dir command.

\$\endgroup\$
3
  • 3
    \$\begingroup\$ Perfect, thanks! Of course before I ran it the first time, I changed "del" to "echo". Then I ran "help for" to understand what it was doing. It's obviously been a long time since I've looked at command prompt scripting, because I had no idea that the "for" command had so many options. \$\endgroup\$
    – seanmc
    Oct 16, 2011 at 20:14
  • \$\begingroup\$ No problem! As I was testing, I also used "echo" =) To see more output, remove the "2> nul." I'd been meaning to do something like this for my own NEF/JPG files, and this was the perfect opportunity. \$\endgroup\$
    – anon
    Oct 17, 2011 at 2:14
  • \$\begingroup\$ Fantastic. Cleaning up and old raw+jpeg folder structure made easy with one neat cmd :) \$\endgroup\$
    – HenningK
    Apr 5, 2020 at 11:20
10
\$\begingroup\$

Here is a modified version of Tomy's Python script. Differences:

  • multiple raw extensions allowed
  • remove jpg only if the pairs are in the same folder (avoid accidental removal of a jpg named like a raw file in an other folder)
  • case insensitive

#!/usr/bin/env python
# Script:      remove_jpg_if_raw_exists.py
#
# Description: This script looks in all sub directories for
#              pairs of JPG and RAW files.
#              For each pair found the JPG is moved to a
#              waste basket directory.
#              Otherwise JPG is kept.
#
# Author:      Thomas Dahlmann
# Modified by: Renaud Boitouzet

import os
import shutil

# define your file extensions here, case is ignored.
# Please start with a dot.
# multiple raw extensions allowed, single jpg extension only
raw_extensions = (".Dng", ".cR2", ".nef", ".crw")
jpg_extension = ".jPg"

# define waste basket directory here. Include trainling slash or backslash.
# Windows : waste_dir = "C:\path\to\waste\"
waste_dir = "/Users/marvin/Pictures/waste/"

##### do not modify below ##########

# find files
def locate(folder, extensions):
    '''Locate files in directory with given extensions'''
    for filename in os.listdir(folder):
        if filename.endswith(extensions):
            yield os.path.join(folder, filename)

# make waste basket dir
if not os.path.exists(waste_dir):
    os.makedirs(waste_dir)

# Make search case insensitive
raw_ext = tuple(map(str.lower,raw_extensions)) + tuple(map(str.upper,raw_extensions))
jpg_ext = (jpg_extension.lower(), jpg_extension.upper())

root=os.curdir
#find subdirectories
for path, dirs, files in os.walk(os.path.abspath(root)):
    print path
    raw_hash = {}
    for raw in locate(path, raw_ext):
        base_name = os.path.basename(raw)
        base_name = os.path.splitext(base_name)[0]
        raw_hash[base_name] = True

    # find pairs and move jpgs of pairs to waste basket
    for jpg in locate(path, jpg_ext):
        base_name = os.path.basename(jpg)
        base_name = os.path.splitext(base_name)[0]
        if base_name in raw_hash:
            jpg_base_name_with_ext = base_name + jpg_extension
            new_jpg = waste_dir + jpg_base_name_with_ext
            print "%s: %s = %s => %s" % (path, base_name, jpg, waste_dir)
            if os.path.exists(new_jpg):
                os.remove(jpg)
            else:
                shutil.move(jpg, new_jpg)
\$\endgroup\$
4
  • \$\begingroup\$ great script. I will use it because it has many good failovers. However you should add this line #!/usr/bin/env python to the beginning. Otherwise I had strange ImageMagick errors (seems my mac open .py files with ImageMagick) \$\endgroup\$ Apr 2, 2015 at 5:48
  • \$\begingroup\$ Just FYI: it also seems it does not work when the files are really named .jPg. It also does not work when files are on external disk and waste directory e.g. in /home dir. \$\endgroup\$ Apr 2, 2015 at 5:55
  • \$\begingroup\$ some corrections: gist.github.com/therealmarv/ec603bd4a91d51092a18 \$\endgroup\$ Apr 2, 2015 at 6:02
  • \$\begingroup\$ @therealmarv: actually what's happening with ImageMagick is that the script is being opened in the shell, not ImageMagick, but "import" is the name of an ImageMagick tool. \$\endgroup\$
    – Max
    Sep 2, 2016 at 8:50
6
\$\begingroup\$

Here's a Python script that moves JPG files when no corresponding RAW file exists. Useful on Mac OS X!

import os
import shutil

raw_ext = '.CR2'
jpg_ext = '.JPG'
destination = '/Users/JohnSmith/Desktop/jpgs/'

for filename in os.listdir('.'):
    (shortname, extension) = os.path.splitext(filename)

    if extension == raw_ext:
        if os.path.isfile(shortname + jpg_ext):
            print 'Moving ' + shortname + jpg_ext + '...'
            shutil.move(shortname + jpg_ext, destination)
\$\endgroup\$
5
\$\begingroup\$
  • Create an empty Library
  • From the Lightroom main menu, choose Edit > Preferences (Windows) or Lightroom > Preferences (Mac OS).
  • In the General preferences unselect "Treat JPEG Files Next To Raw Files As Separate Photos"
    • This should be the default.
  • Import all of your files (you can select search subfolders), telling it to move to a new location
  • The JPG files that have RAW files will be left in the original location for you to remove

As I understand it, the thumbnail in lightroom might say RAW+JPG, but the JPG isn't actually stored or accessible in any way.

You can also write a pretty simple batch script with any programming language.

\$\endgroup\$
2
\$\begingroup\$

I like the bash script for OS X (by T.Toivonen), but I have noticed there are a few issues.

  • It did not like my directory names, which contain spaces. That required a bit different handling of the find command.

  • The original script only works for lowercase extensions. I have slightly improved that part of the script to account for extensions that are in uppercase as well. Note, that it only accepts DNG+JPG or dng+jpg pairs, and it will ignore any combinations such as DNG+jpg or DnG+JpG.

  • The original solution proposed only one wastedir location, whereas my fix allows a subdirectory to be created on each directory branch as it travels though. You define a name of the directory before the loop.

  • I like to see what's going on, especially when mv or rm commands are used ;)

For the sake of the space I am showing only the last part of the script, from setting a basedir, wastedir and the loop.

[...]

#Now set it as a basedir
BASEDIR=$arg
WASTEDIR=duplicates
find "$BASEDIR" -iname '*.dng' -print0 | while read -d $'\0' filename 
    do
    filepath="${filename%/*}"
    basename="${filename##*/}"
    prefix="${basename%%.*}"
    suffix=${filename##*.}
    if [[ "$suffix" =~ [A-Z] ]]; then rsuffix="JPG"; else rsuffix="jpg"; fi 
    if [ -e "$filepath/$prefix.$rsuffix" ]; then
        let counter="$counter+1"
        if (( $isSetE==1 )); then
            echo "FOUND: $filepath/$prefix.$rsuffix"
        fi
        if (( $isSetM==1 )); then
            echo "Moving $filepath/$prefix.$rsuffix to $filepath/$WASTEDIR"
            if [ ! -d "$filepath/$WASTEDIR" ]; then mkdir "$filepath/$WASTEDIR"; fi
            mv "$filepath/$prefix.$rsuffix" "$filepath/$WASTEDIR"
        fi
        if (( $isSetD==1 )); then
            echo "Removing duplicate $filepath/$prefix.$rsuffix"
            rm "$filepath/$prefix.$rsuffix"
        fi
    fi
done
\$\endgroup\$
1
  • \$\begingroup\$ The question was tagged "windows" so you could do with saying how this could be made to work in a typical Windows system. I run Cygwin, for example (and I plan to have a better look at this answer when I'm on desktop, to change the behavior a bit) \$\endgroup\$
    – Chris H
    Apr 10, 2017 at 7:10
2
\$\begingroup\$

Here is a solution for bash (Linux or Mac OS X). On Windows, you can install Cygwin to get a copy of bash.

keep=$(ls | grep -v ps | grep -A1 JPG | grep NEF)
for i in $keep ; do
   mv $i $i.keep
done

ls | egrep -v '(JPG|keep)' | xargs rm -f

change=$(ls | grep keep | sed 's/.keep//g')
for i in $change ; do
   mv $i.keep $i
done
\$\endgroup\$
0
2
\$\begingroup\$

Here is another bash version using find (Linux). As with Ben Pingilley's answer, you can install Cygwin to get bash on Windows.

#!/bin/bash
read -p "please enter file suffix for raw format (e.g ORF, NEF, CR2): " suffix

find . -type f -iname "*.${suffix}" | \
while read line
do
  lowercase=$(echo "$line" | sed "s/${suffix}/jpg/gi")
  uppercase=$(echo "$line" | sed "s/${suffix}/JPG/gi")

  if [ -f "${lowercase}" ]
  then
    rm -v "${lowercase}"
  elif [ -f "${uppercase}" ]
  then
    rm -v "${uppercase}"
  else
    echo "${line}: no jpg present"
  fi
done
\$\endgroup\$
0
1
\$\begingroup\$

Here's my take on this issue. A lot of good ideas came from earlier scripts mentioned here.

This is a bash script for OS X. It looks for files that exist with same base filename and dng+jpg extensions. If a jpg is found with an exactly same name as dng, then that filename is displayed (-e), file is moved (-m) or deleted (-d).

It will go through subfolders, so you could use it for your entire catalog or just parts of it.

For other raw file extensions just substitute *.dng in the script with your preferred extension.

Warning: You could have two different images with same name, but different extension. Those are inevitable casualties of this script.

Here's how to use the script:

Usage: dng-jpg.sh [-m <path>] [-d <path>] [-e <path>] [-h]

-m: for move   (moves files to <path>/duplicates)
-d: for delete (deletes duplicate files)
-e: for echo   (lists duplicate files)
-h: for help 

Basic usage would work like this:

$ ./dng-jpg.sh -e /Volumes/photo/DNG/2015

That would echo all filenames of jpg files that match the criteria of having both dng and jpg file with same name.

Result would look like something like this:

Echo selected with path: /Volumes/photo/DNG/2015
/Volumes/photo/DNG/2015/03/18/2015-03-18_02-11-17.jpg
/Volumes/photo/DNG/2015/06/01/2015-06-01_05-10-50.jpg
/Volumes/photo/DNG/2015/06/01/2015-06-01_05-10-56.jpg
/Volumes/photo/DNG/2015/06/01/2015-06-01_05-11-39.jpg
/Volumes/photo/DNG/2015/06/01/2015-06-01_05-11-54.jpg
/Volumes/photo/DNG/2015/06/01/2015-06-01_05-12-26.jpg
/Volumes/photo/DNG/2015/06/01/2015-06-01_05-12-43.jpg
/Volumes/photo/DNG/2015/06/01/2015-06-01_05-13-21.jpg
/Volumes/photo/DNG/2015/06/01/2015-06-01_05-13-56.jpg
9 files found.

Now if I want to delete the files I would just switch the -e to -d:

$ ./dng-jpg.sh -d /Volumes/photo/DNG/2015

Or if I'd like to move the files to /duplicates I'd execute it with -m.

$ ./dng-jpg.sh -m /Volumes/photo/DNG/2015

Now the duplicate jpg files would be in /Volumes/photo/DNG/2015/duplicates

Here's the script: dng-jpg.sh

#!/bin/bash

# Init variables
isSetM=0
isSetD=0
isSetE=0
isSetCount=0
counter=0

#Display usage info
usage() {

    cat <<EOF

Usage: dng-jpg.sh [-m <path>] [-d <path>] [-e <path>] [-h]

-m: for move   (moves files to <path>/duplicates)
-d: for delete (deletes duplicate files)
-e: for echo   (lists duplicate files)
-h: for help 

EOF
  exit 1
}

#Check for parameters
while getopts ":m:d:e:h" opt; do
  case ${opt} in
    m)
        isSetM=1
        let isSetCount="$isSetCount+1"
        arg=${OPTARG}
      echo "Move selected with path:" $arg
      ;;
    d)
        isSetD=1
        let isSetCount="$isSetCount+1"
        arg=${OPTARG}
      echo "Delete selected with path:" $arg
      ;;
    e)
        isSetE=1
        let isSetCount="$isSetCount+1"
        arg=${OPTARG}
      echo "Echo selected with path:" $arg
      ;;
    h)
        let isSetCount="$isSetCount+1"
        usage
      ;;
    \?)
      echo "Invalid option: -$OPTARG" >&2
      usage
      ;;
    :)
      echo "Option -$OPTARG requires a directory argument." >&2
      usage
      ;;
    *)
      usage
      ;;
  esac
done

# If no parameters, show usage help and exit
if test -z "$1"; then
    usage
fi

# If multiple parameters (not counting -a), show usage help and exit
if (($isSetCount > 1)); then
    usage
fi

#Verify directory
if [ ! -d "$arg" ]; then
  echo "$arg is not a path to a directory." >&2
  usage
fi

#Now set it as a basedir
BASEDIR=$arg
WASTEDIR="$BASEDIR/duplicates/"
if (( $isSetM==1 )); then
    mkdir $WASTEDIR
fi

for filename in $(find $BASEDIR -name '*.dng' -exec echo {} \; | sort); do
   prefix=${filename%.dng}
    if [ -e "$prefix.jpg" ]; then
        let counter="$counter+1"
        if (( $isSetE==1 )); then
            echo "$prefix.jpg"
        fi
        if (( $isSetM==1 )); then
            mv $prefix.jpg $WASTEDIR
        fi
        if (( $isSetD==1 )); then
            rm $prefix.jpg
        fi
    fi
done

echo "$counter files found."
\$\endgroup\$
1
\$\begingroup\$

Here’s a bash script for Mac OS X. It may work on Linux with some changes.

#!/bin/bash
read -p "Delete JPEGs when DNG exists? Ctrl-C to cancel. [Enter] to continue: "

for FILE in *.dng; do
  JPG_FILE=$(echo "$FILE" | sed "s/dng/jpg/g")
  rmtrash "${JPG_FILE}" 1>/dev/null
done

rmtrash is a utility that moves files to the Trash, instead of deleting them outright. You can get it from MacPorts thus:

sudo port install rmtrash

If you’d like to avoid that, just replace rmtrash in the script with rm, which will immediately delete the JPG files.

\$\endgroup\$
1
\$\begingroup\$

I wrote the following Python script. Compared with ttaveira's script, it does some extra work.

  • Looks in sub-directories.
  • Creates destination waste directory.
  • Removes files that already exist in waste directory to avoid move errors.

# Script:      remove_jpg_if_raw_exists.py
#
# Description: This script looks in all sub directories for
#              pairs of JPG and RAW files.
#              For each pair found the JPG is moved to a
#              waste basket directory.
#              Otherwise JPG is kept.
#
# Author:      Thomas Dahlmann

import os, fnmatch

# define your file extensions here, case is ignored
raw_extension = "nef"
jpg_extension = "jpg"

# define waste basket directory here
waste_dir = "c:\image_waste_basked"

##### do not modify below ##########

# recursive find files 
def locate(pattern, root=os.curdir):
    '''Locate all files matching supplied filename pattern 
    in and below root directory.'''
    for path, dirs, files in os.walk(os.path.abspath(root)):
        for filename in fnmatch.filter(files, pattern):
            yield os.path.join(path, filename) 

# get base names from raw's
raw_hash = {}
for raw in locate("*." + raw_extension):
    base_name = os.path.basename(raw)
    base_name = os.path.splitext(base_name)[0]
    raw_hash[base_name] = True

# make waste basket dir
if not os.path.exists(waste_dir):
    os.makedirs(waste_dir)

# find pairs and move jpgs of pairs to waste basket    
for jpg in locate("*." + jpg_extension):
    base_name = os.path.basename(jpg)
    base_name = os.path.splitext(base_name)[0]
    if base_name in raw_hash:
        jpg_base_name_with_ext = base_name + "." + jpg_extension
        new_jpg = waste_dir + "\\" + jpg_base_name_with_ext
        print "%s => %s" % (jpg, waste_dir)
        if os.path.exists(new_jpg):
            os.remove(jpg)
        else:
            os.rename(jpg, new_jpg)
\$\endgroup\$
2
  • 2
    \$\begingroup\$ Hi and welcome to Photo.SE. How is your answer different from ttaveira's answer? \$\endgroup\$ Apr 8, 2014 at 14:56
  • \$\begingroup\$ The script does some extra work: also looks in all sub-directories, creates destination waste directory for jpg's if it does no exist and it does remove a jpg instead of moving if it if it already exists in waste directory (avoids moving errors) \$\endgroup\$
    – Tomy
    Apr 30, 2014 at 15:58
0
\$\begingroup\$

Working on Mac OS X, I was missing a sanity check for "same content" in the previous answers. I had duplicate names for different pictures because I had forgotten to enable the image counter in my camera. Here's my version, which checks the EXIF information for same capture time:

You need to run

sudo port install rmtrash exiv2

before you can use the following command. It was written to compare JPG with NEF files from my Nikon D90. Adjust the file extensions according to your needs.

find . -name \*.NEF |sed s/\.NEF/.JPG/g | xargs find 2>/dev/null | \
xargs perl -e 'foreach(@ARGV) {my $jpg=$_;my $nef=s/\.JPG/.NEF/r; my $tjpg = `exiv2 -g Exif.Photo.DateTimeOriginal -pt $jpg`; my $nef=s/\.JPG/.NEF/r; my $tnef = `exiv2 -g Exif.Photo.DateTimeOriginal -pt $nef`; if($tjpg eq $tnef) {print "$jpg\n"}}' | \
xargs rmtrash

without the sanity check, the whole thing would become a very short one liner:

find . -name \*.NEF |sed s/\.NEF/.JPG/g | xargs find 2>/dev/null | xargs rmtrash
\$\endgroup\$

Not the answer you're looking for? Browse other questions tagged or ask your own question.