18
\$\begingroup\$

Can anyone suggests a good photo duplication detection utility that works well when I am dealing with about a 100gb of data (collected over the years)?

I would prefer something that works on Ubuntu.

Thanks in advance!

Edit: Is there a tool that will help me reorganize my collection and remove duplicates, once they have been detected?

Edit2: The hard part is figuring out what to do once I have the output consisting of thousands of duplicate files (such as the output of fdupes).

Its not obvious if I can still safely delete a directory (i.e. if a directory might contain unique files), which directories are subsets of other directories and so on. An ideal tool for this problem should be able to determine file duplication an then provide a powerful means of restructuring your files and folders. Doing a merge by hardlinking (as fslint does) does indeed free up diskspace but it does not solve the underlying problem which gave rise to the duplication to start with -- i.e. bad file/dir organization.

Improve this question
\$\endgroup\$
1

7 Answers 7

Reset to default
8
\$\begingroup\$

ImageMagick to the rescue. I think the first step to any solution is to reduce the size of your collection. If you want to compare the photos by its content, especially when some are slightly modified versions of one another, a very good start is to reduce them to thumbnails and then compare the thumbnails. This is particular helpful when you want to find almost-alike photos and want to "ignore" unimportant differences during comparison.

My suggestion is, at a high level, that you:
1- Use ImageMagick's mogrify tool to reduce the photos to thumbnails. This will take some time but it will make the actual comparison steps much much faster and more accurate.
2- Use ImageMagick's compare tool that allows you to set a threshold for comparison, i.e. it allow you find photos that are 85% alike. You would want to do a controlled experiment to find out the threshold value that you like most.

Improve this answer
\$\endgroup\$
2
  • \$\begingroup\$ I really like this idea of making thumbnails first. What does it do once you have found the duplicates? Does it just display a list? I have 10s of thousands of duplicates and a nice GUI to help resolve these would be very useful. \$\endgroup\$
    – Fasterz
    Commented Aug 10, 2012 at 16:27
  • 2
    \$\begingroup\$ Since you use Ubuntu, you automatically have access to a host of specialized tools, each solving a very specific task such as the 2 tasks I mentioned. It's a Lego game, you can do whatever you want, you just need to put the pieces together. Technically, you feeds 2 photos to the 'compare' tool and it will tell you how much one resembles the other. One way to solve your problem is to group all similar photos into folders so you can go thru them to filter out false positives. Then you run 'compare' again on the false positives and repeat the process until all are in their correct places. \$\endgroup\$
    – cody
    Commented Aug 11, 2012 at 1:06
4
\$\begingroup\$

The open source photo viewer / organizer Geeqie has a powerful Find Duplicates Feature. It can use several different strategies for finding duplicates:

  • File name (case sensitive or insensitive)
  • File size
  • File date
  • Image dimensions
  • MD5 checksum.
  • Similar image content (to several thresholds)

This gives a results list which can include thumbnails so you can confirm manually.

This will probably be slow for thousands of files, but I think just using it and letting it run for a few days or whatever is probably less effort overall than finding or making something tailored for the case — unless checksum match is all you need.

Improve this answer
\$\endgroup\$
2
  • \$\begingroup\$ That sounds nice. What does it do once you have found the duplicates? Does it just display a list? I have 10s of thousands of duplicates and a nice GUI to help resolve these would be very useful. \$\endgroup\$
    – Fasterz
    Commented Aug 10, 2012 at 16:26
  • \$\begingroup\$ It displays them in a GUI window. \$\endgroup\$
    – mattdm
    Commented Aug 10, 2012 at 16:59
3
\$\begingroup\$

There is a little utility called "fdupes" that may do what you wish?

There is also another utility called "fslint" that you might want to try out too. (This one has a GUI).

Improve this answer
\$\endgroup\$
6
  • \$\begingroup\$ I just tried fslint on a smaller set of pictures (a few gig or so) and its frustrating that it just sits there and spins. No progress indicator, estimate of time left, nothing. \$\endgroup\$
    – Fasterz
    Commented Aug 8, 2012 at 21:39
  • 1
    \$\begingroup\$ These tools appear to look for identical files. Even an identical (pixel for pixel) image can be different file contents. I'm guessing you want to match up not only the same look-alike image, but also do so in different formats, and sizes, including crops and other processing you have done, such as to collect all variations of the same photo in one directory. This would be a soft comparison of images that would have a confidence match factor, and could match up different photos of the same scene. \$\endgroup\$
    – Skaperen
    Commented Aug 9, 2012 at 3:12
  • \$\begingroup\$ @Skaperen What you suggest is great, but do such tools exist for Ubuntu? I have seen one mentioned somewhere for Windows -- but that seemed to have a hideous interface.. etc. \$\endgroup\$
    – Fasterz
    Commented Aug 9, 2012 at 16:34
  • \$\begingroup\$ ImageDupeless is a windows app that will catch photos that look alike, but have some differences. It will catch some rotations, crops, resizes, color tint changes, watermarks, etc... you have to scan your library and tell it how much difference you accept, and it will merrily show you the files. BUT it would be extraordinarily cumbersome for hundreds of files, and thousands of files would be terrible. I too am looking for a linux equivalent to ImageDupeless. An app that does wavelets or some other imaging magic to tell when images are similar. \$\endgroup\$
    – Therealstubot
    Commented Aug 9, 2012 at 23:37
  • 1
    \$\begingroup\$ Start with fdupes as this answer suggests to look for and delete the obvious bit for bit same dupes, then optionally do a second pass with a more sophisticated tool that looks for similar / same images. \$\endgroup\$
    – rrauenza
    Commented Jan 17, 2018 at 16:53
1
\$\begingroup\$

dupeGuru Picture Edition is a customisable duplicate image finder for Windows, Mac OS X and Linux.

There's a few versions of dupeGuru (standard, music & picture editions), and the picture edition allows you to find visually similar images via a bitmap blocking comparison algorithm, among other methods (like EXIF original image timestamp, or the files being simply identical).

It has a variety of other useful features like excluded folders, support for iPhoto/Aperture libraries, and considerable customisation of how it detects duplicates and what it does with them.

Improve this answer
\$\endgroup\$
0
\$\begingroup\$

What do you mean by duplicate photos? Do you mean files that are identical, say just copied an extra time or two? or do you mean photos that "look" to be the same.

If you mean identical files, you can use 'shasum' on all of the files, then order the results and find the unique lines with 'uniq' and run a 'diff' to see what has been eliminated. All easy in a Ubuntu shell.

Improve this answer
\$\endgroup\$
7
  • \$\begingroup\$ None of this is easy or convenient. fdupes mentioned below will already do a better job than merely calculating SHA. Now are there unix tools that will look for image similarity? If so, that would be awesome. \$\endgroup\$
    – Fasterz
    Commented Aug 9, 2012 at 16:31
  • \$\begingroup\$ Easy and convenient for someone used to using the unix tools, which is what uniq, sort, diff, shasum, etc. are. But I agree that if you don't use them regularly, they can be hard to use. I don't know of anything that can do "looks like" Everything I've seen, including in Aperture and Lightroom, do file-is-identical, which is really just a md5 or shasum \$\endgroup\$
    – Pat Farrell
    Commented Aug 9, 2012 at 22:33
  • \$\begingroup\$ I regularly use unix tools and I find this answer somewhat silly. First, doing SHA blindly is slow, when a file size comparizon resolves things. Second, SHA or MD5 can collide -- so SHA comparisons alone arent enough. If you factor in both these, then you get to what fdupes does. \$\endgroup\$
    – Fasterz
    Commented Aug 10, 2012 at 16:18
  • \$\begingroup\$ Also, once you have correctly conjured the incantation that does this, the output is still not very useful. At best you get the output of fdupes which is a just a dump of similar files. In my case I have 10s of thousands and it is very hard to pick through that data to see how I can eliminate the duplicates. \$\endgroup\$
    – Fasterz
    Commented Aug 10, 2012 at 16:22
  • 1
    \$\begingroup\$ SHAs do collide in theory, but not in practice. Yes, it takes forever. Nothing that is going to work is going to be fast. But you should be able to kick it off and come back in a day or two. Its just a suggestion, I'm not going to get into a war over it. \$\endgroup\$
    – Pat Farrell
    Commented Aug 11, 2012 at 0:00
0
\$\begingroup\$

There is an aplication called "bleachbit", which finds duplicate files by size, name and other filters. You can install it from the synapctic package manager in ubuntu.

Improve this answer
\$\endgroup\$
1
  • \$\begingroup\$ What does it do once you have found the duplicates? Does it just display a list? I have 10s of thousands of duplicates and a nice GUI to help resolve these would be very useful. \$\endgroup\$
    – Fasterz
    Commented Aug 10, 2012 at 16:25
0
\$\begingroup\$

There is a brand new version of Excire Foto (2.0) that adds a sophisticated and tuneable AI-based duplicate finding/weeding function.

I've been very happy with Excire Foto 1.3.4 for its AI-based keywording, and for its ability to sort through photos by similarity, or by faces. But it looks like version 2.0's dup finding is just what I needed!

You can set a slider to choose the degree of similarity required to call something a "duplicate." At the most strict setting, the images have to be identical, although not the same pixel dimensions. At the most loose setting, it can help you cull excess shots by including ones that differ subtly, so you can pick the best one.

Find duplicates Excire Foto 2.0 "Find Duplicates" dialogue.

As you can see, you can allow for sequences, which doesn't suit my needs, as I do a lot of time-lapses that have more than eight seconds between shots.

Sequence parameter

Once you've set the parameters, it displays progress while it works. My imagebase of several hundred thousand images (previously indexed by Excire Foto) took about ten minutes to return nearly 70,000 duplicates. (It's a long, sad story about trying to get my images out of Apple Aperture multiple times, after Apple abruptly stopped supporting it.)

When this process ends, you'll have your images grouped by similarity.

Duplicate result set. This is the listing of some 70,000 duplicate hits.

You can change how they are grouped and sorted, and you can see in the first row, a time-series, that several of these images are subtly different. Enter the "Flagging Assistant."

Flagging assistant. Flagging assistant. The "Flagging Assistant" allows you to automagically choose from among duplicates.

This allows you to select and flag images for review (or not!) and deletion, based upon criteria you can select.

Grouping by ascending photo count allowed me to put the sequences to the end, and focus on the true duplicates. Sorting by pixel dimensions made it easy to cull the second ones, which were of equal or lesser quality.

Duplicates Grouping by ascending photo count shows the least number of duplicates first.

I've been looking long and hard for such a tool, and this is the best I've come across for the task! It's been out for not even a day yet, so I still have a lot of exploring and learning to do.

Getting the initial index built takes a long time for large collections. My ~300,000 images took about three days to index. If you are impatient, and if, like me, you have images in a file hierarchy, you can index individual sub-hierarchies if you'd rather. I just threw my entire terabyte Pictures folder at it, to see how it would do!

Indexing takes time because it is doing much more than finding duplicates — it is using an AI-trained model to assign keywords, which is wonderful in its own right. That's why I originally bought it, and I'm delighted that this new version read my mind and made it simple to find duplicates!

Improve this answer
\$\endgroup\$

Not the answer you're looking for? Browse other questions tagged or ask your own question.