Westminster fountain at sunset

by Jorge Córdoba

submit your photo


Hall of Fame
View past winners from this year

Please participate in Meta
and help us grow.

74 reputation
14
bio website
location
age
visits member for 2 years, 2 months
seen Aug 20 '12 at 20:55

Jul
19
awarded  Notable Question
Mar
20
awarded  Nice Question
Jul
8
awarded  Popular Question
Aug
10
comment How can I find duplicate photos in a very large pool of data (tens to hundreds of gigs)?
I really like this idea of making thumbnails first. What does it do once you have found the duplicates? Does it just display a list? I have 10s of thousands of duplicates and a nice GUI to help resolve these would be very useful.
Aug
10
comment How can I find duplicate photos in a very large pool of data (tens to hundreds of gigs)?
That sounds nice. What does it do once you have found the duplicates? Does it just display a list? I have 10s of thousands of duplicates and a nice GUI to help resolve these would be very useful.
Aug
10
comment How can I find duplicate photos in a very large pool of data (tens to hundreds of gigs)?
What does it do once you have found the duplicates? Does it just display a list? I have 10s of thousands of duplicates and a nice GUI to help resolve these would be very useful.
Aug
10
comment How can I find duplicate photos in a very large pool of data (tens to hundreds of gigs)?
Also, once you have correctly conjured the incantation that does this, the output is still not very useful. At best you get the output of fdupes which is a just a dump of similar files. In my case I have 10s of thousands and it is very hard to pick through that data to see how I can eliminate the duplicates.
Aug
10
comment How can I find duplicate photos in a very large pool of data (tens to hundreds of gigs)?
I regularly use unix tools and I find this answer somewhat silly. First, doing SHA blindly is slow, when a file size comparizon resolves things. Second, SHA or MD5 can collide -- so SHA comparisons alone arent enough. If you factor in both these, then you get to what fdupes does.
Aug
9
comment How can I find duplicate photos in a very large pool of data (tens to hundreds of gigs)?
@Skaperen What you suggest is great, but do such tools exist for Ubuntu? I have seen one mentioned somewhere for Windows -- but that seemed to have a hideous interface.. etc.
Aug
9
revised How can I find duplicate photos in a very large pool of data (tens to hundreds of gigs)?
added 679 characters in body
Aug
9
comment How can I find duplicate photos in a very large pool of data (tens to hundreds of gigs)?
None of this is easy or convenient. fdupes mentioned below will already do a better job than merely calculating SHA. Now are there unix tools that will look for image similarity? If so, that would be awesome.
Aug
8
awarded  Editor
Aug
8
revised How can I find duplicate photos in a very large pool of data (tens to hundreds of gigs)?
added 121 characters in body
Aug
8
comment How can I find duplicate photos in a very large pool of data (tens to hundreds of gigs)?
I just tried fslint on a smaller set of pictures (a few gig or so) and its frustrating that it just sits there and spins. No progress indicator, estimate of time left, nothing.
Aug
8
awarded  Student
Aug
8
asked How can I find duplicate photos in a very large pool of data (tens to hundreds of gigs)?