Over the years, I've accumulated a lot of pictures, across a bunch of different machines.
I think I've mostly kept them in a yyyy/mm/ directory structure, but especially with some of the older pictures, there seem to be a lot of duplicates.
What's the best way to identify the duplicates? Are there good tools to do this, or am I going to have to write something on my own?
A number of programs can find duplicates based on image contents. Some asset management apps will do it and there are also dedicated programs. If you Google 'detect duplicate images' you'll see quite a few options. See which ones are available for your platform.
Another solution is to simply import the files with the 'ignore duplicate option'. Picasa, which is free, can do it.
dupeGuru Picture Edition is a customisable duplicate image finder for Windows, Mac OS X and Linux.
There's a few versions of dupeGuru (standard, music & picture editions), and the picture edition allows you to find visually similar images via a bitmap blocking comparison algorithm, among other methods (like EXIF original image timestamp, or the files being simply identical).
It has a variety of other useful features like excluded folders, support for iPhoto/Aperture libraries, and considerable customisation of how it detects duplicates and what it does with them.
For Linux, try FSlint. It's a bit slow, but it will definitely find duplicates.
Picasa has a duplicate file finder but it's hard to tell what you are looking at: Run Picasa; click Tools > Experimental > Show Duplicate Files. It works but be careful about what you are deleting.
Another option is the free "Fast Duplicate File Finder":
What I liked about this is it allows to you preview the picture before deciding you don't need the duplicate.
That works fine as long as the files are identical, but if you are looking for similar photos (e.g. same photo but resized, rotated, or manipulated), you may want to look into their upgraded version:
Visual Similarity Duplicate Images Finder
I haven't used this program but I used a very similar program (no longer made) and I found it very useful. They have a free download so it might be worth checking out.
digikam (linux, windows and mac) has a sidebar titled versions
The Versions tab shows the history and the saved versions of a photograph. With the three buttons in the top right corner you can choose between a simple list of the saved versions, a tree view and a combined list that shows the versions together with the actions performed with the selected photograph.
and another titled search
Quick Search is the adaptive search box at the top of the search dialog (Left Sidebar) and provides you with a simple means to search the digiKam database with a single query. You can enter any arguments in the Search: text field and it will be used to determine the results. For instance, you can enter the string 'birthday' to search the complete database for occurrences of the word 'birthday' in all metadata text fields, or '05.png' to search for a specific image file name '05.png' (no escaping as in regexpressions needed).
(It also has an advanced setting allowing multiple boolean searches)
and another titled fuzzy
digiKam characterizes every image by a lengthy number using a special technique (Haar algorithm) that makes it possible to compare images by comparing this calculated signature. The less numerical difference there is between any two image signatures, the more they resemble each other. This technique has three implementations in digiKam:...
Though the generation of the "signature's" and cross checking of them on a large number of images can be a slow process. Also be aware that some people are not fans because it is a KDE project that doesn't integrate totally with other desktop environments and it tries to be an all in one application with image download, catalogue / keywording, RAW development, image maniplation and more included.
Similar.pictures is an online program to find near and exact duplicates.
I wrote it because Picasa used to rather find exact copies. And now Picasa is gone.
My two cents. I have used:
One thing to consider is the "fuzziness" of the search. Sometimes the program finds similar photos, for example, variations of the same photoshoot. Don't have a happy trigger when deleting your images.
Please, backup your images before deleting anything.
Acdsee is a simple/popular photo browsing app that has this feature (Tools -> Find duplicates).
On my Mac, I've had excellent success with PhotoSweeper (http://overmacs.com/). It can work directly through the file system or via popular photography software such as Apple Photos and Adobe Lightroom. It is able to find either exact duplicates or similar image files. And at $10, it's a steal.
I'd recommend the upcoming version of XYplorer (File Manager). It will let you sort photos by visual similarity.
DeDup-Image is a bash script to automatically find and delete identical images / duplicate photos, even if the metadata differs. It always retains the largest copy, which probably contains the most metadata and deletes all other found objects based on the SHA256-HASH of the real image information without any metadata. Because the script needs to process a variant from each image without metadata, it's a bit slower and processor-intensive than programs or scripts that can only search for completely identical files.
