16

Over the years, I've accumulated a lot of pictures, across a bunch of different machines.

I think I've mostly kept them in a yyyy/mm/ directory structure, but especially with some of the older pictures, there seem to be a lot of duplicates.

What's the best way to identify the duplicates? Are there good tools to do this, or am I going to have to write something on my own?

10 Answers 10

7

A number of programs can find duplicates based on image contents. Some asset management apps will do it and there are also dedicated programs. If you Google 'detect duplicate images' you'll see quite a few options. See which ones are available for your platform.

Another solution is to simply import the files with the 'ignore duplicate option'. Picasa, which is free, can do it.

  • Google detect duplicate images actually works as promised. There is a good number of such software to be found. Can you recommend anything else than Picasa? – Esa Paulasto Mar 20 '14 at 6:48
4

For Linux, try FSlint. It's a bit slow, but it will definitely find duplicates.

  • FSlint will search files with same hash value. As such it will find duplicates of any file. It won't help with 'same-image-different-metadata' scenario or with resized versions of the original. – seb Jul 28 '18 at 6:06
4

dupeGuru Picture Edition is a customisable duplicate image finder for Windows, Mac OS X and Linux.

There's a few versions of dupeGuru (standard, music & picture editions), and the picture edition allows you to find visually similar images via a bitmap blocking comparison algorithm, among other methods (like EXIF original image timestamp, or the files being simply identical).

It has a variety of other useful features like excluded folders, support for iPhoto/Aperture libraries, and considerable customisation of how it detects duplicates and what it does with them.

3

Picasa has a duplicate file finder but it's hard to tell what you are looking at: Run Picasa; click Tools > Experimental > Show Duplicate Files. It works but be careful about what you are deleting.

Another option is the free "Fast Duplicate File Finder":

Fast Duplicate File Finder

What I liked about this is it allows to you preview the picture before deciding you don't need the duplicate.

alt text

That works fine as long as the files are identical, but if you are looking for similar photos (e.g. same photo but resized, rotated, or manipulated), you may want to look into their upgraded version:

Visual Similarity Duplicate Images Finder

I haven't used this program but I used a very similar program (no longer made) and I found it very useful. They have a free download so it might be worth checking out.

  • I tried the picasa dupe finder, but it seems to be matching similar images, not identical files. – chris Jan 4 '11 at 22:55
2

digikam (linux, windows and mac) has a sidebar titled versions

The Versions tab shows the history and the saved versions of a photograph. With the three buttons in the top right corner you can choose between a simple list of the saved versions, a tree view and a combined list that shows the versions together with the actions performed with the selected photograph.

and another titled search

Quick Search is the adaptive search box at the top of the search dialog (Left Sidebar) and provides you with a simple means to search the digiKam database with a single query. You can enter any arguments in the Search: text field and it will be used to determine the results. For instance, you can enter the string 'birthday' to search the complete database for occurrences of the word 'birthday' in all metadata text fields, or '05.png' to search for a specific image file name '05.png' (no escaping as in regexpressions needed).

(It also has an advanced setting allowing multiple boolean searches)

and another titled fuzzy

digiKam characterizes every image by a lengthy number using a special technique (Haar algorithm) that makes it possible to compare images by comparing this calculated signature. The less numerical difference there is between any two image signatures, the more they resemble each other. This technique has three implementations in digiKam:...

  1. Duplicates
  2. Image (Similar items)
  3. Sketch

Though the generation of the "signature's" and cross checking of them on a large number of images can be a slow process. Also be aware that some people are not fans because it is a KDE project that doesn't integrate totally with other desktop environments and it tries to be an all in one application with image download, catalogue / keywording, RAW development, image maniplation and more included.

1

Similar.pictures is an online program to find near and exact duplicates.

I wrote it because Picasa used to rather find exact copies. And now Picasa is gone.

0

Acdsee is a simple/popular photo browsing app that has this feature (Tools -> Find duplicates).

0

On my Mac, I've had excellent success with PhotoSweeper (http://overmacs.com/). It can work directly through the file system or via popular photography software such as Apple Photos and Adobe Lightroom. It is able to find either exact duplicates or similar image files. And at $10, it's a steal.

0

My two cents. I have used:

Pixiple github.com/saolaolsson/pixiple

VisiPics visipics.info

And Awesome Duplicate Photo Finder duplicate-finder.com

One thing to consider is the "fuzziness" of the search. Sometimes the program finds similar photos, for example, variations of the same photoshoot. Don't have a happy trigger when deleting your images.

-1

DeDup-Image is a bash script to automatically find and delete identical images / duplicate photos, even if the metadata differs. It always retains the largest copy, which probably contains the most metadata and deletes all other found objects based on the SHA256-HASH of the real image information without any metadata. Because the script needs to process a variant from each image without metadata, it's a bit slower and processor-intensive than programs or scripts that can only search for completely identical files.

  • 4
    This program looks very dangerous because it automatically deletes. I really, really would not want any program to be deleting files here and there. – Rafael Jan 14 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.