3
\$\begingroup\$

I'm currently in the process of recovering all the digital photos that were ever taken in my family. I've extracted data in multiple ways from damaged hard disks, broken CDs, almost unreadable floppy drives and so on.

My main problem now, having collected so much data (and in so many copies, especially with broken data supports I tried different data carving tools in order to recover as many photos as possible) is how to automatically remove damaged copies and remove duplicates.

I've used bad Peggy in order to move to a different folder the "damaged" files, yet some of them are showing up perfectly even if the program categorized them as damaged, while I keep find a few that appear to be severely damaged and yet haven't been moved.

I managed to clean up all the mess with the still damaged pictures in the "good" folder by hand, and did multiple scans with visipics, alldup (with every picture hashing method), Gemini and PhotoSweeperX on Mac. Now every "good picture" doesn't have any duplicate, but I still need to sort out the "damaged folder" removing damaged copies of the photos I already have in the good one so that I can sort the "damaged ones" in order to save the few ok ones and the few damaged but still usable ones.

The thing is most picture comparison softwares actually make a low-res image comparison or use other "content aware" hashing method. This works perfectly when using duplicates of not damaged photos, yet when you're working with damaged jpegs, that usually have just half or (sometimes way) less of the image, this doesn't work at all as the duplicate finding software detects the damaged jpg like some sort of solid gray image.

Does anyone know of any photo comparison software that compares images in a pixel by pixel way? By pixel by pixel I mean that it compares the colored pixel starting from top left and going down just like us LTR language readers do on a printed page?

Thanks in advance.

Improve this question
\$\endgroup\$
3

2 Answers 2

Reset to default
1
\$\begingroup\$

The free software AllDup allows to check duplicates only using a small sector in the top left of the image thus partly solving the problem

Improve this answer
\$\endgroup\$
0
\$\begingroup\$

One possible way is to create hashes of all the files in question. And then compare those hashes. This can be done with SHA algorithm (you need to install coreutils). This is not bitwize compare, but its enough with very high probability.

Also you can try ACDsee Pro to find duplicated images (you can set the degree of difference between images)

Improve this answer
\$\endgroup\$
6
  • \$\begingroup\$ This is a good first step, as it will help eliminate truly duplicate files. It will not find images that might have been (however slightly) modified from the original - even just re-encoding an image will usually create a different file with a different hash. Beyond that, there are some comparison utilities out there, like in ImageMagick (or its GraphicsMagick fork) \$\endgroup\$
    – twalberg
    Commented Jun 14, 2020 at 20:20
  • \$\begingroup\$ @twalberg, pixel by pixel compare (as requested by OP) will not work for one bit change. \$\endgroup\$
    – Romeo Ninov
    Commented Jun 14, 2020 at 20:25
  • \$\begingroup\$ I’ve already used different hashing methods with multiple programs to delete the “obvious” duplicates, yet I still don’t know how to pair the first pixels of a damaged jpg to its not damaged duplicate. I’ve actually just discovered AllDup has a bit-by-bit comparison mode with also an option to ignore jpg/cr2 metadata, but over 28000 pics it only found 8 duplicates, which is not credible at all. \$\endgroup\$
    – gio91ber
    Commented Jun 14, 2020 at 21:45
  • \$\begingroup\$ @gio91ber, please check my edited answer \$\endgroup\$
    – Romeo Ninov
    Commented Jun 15, 2020 at 7:46
  • 2
    \$\begingroup\$ "bit-by-bit comparison mode" But a bit by bit method could never work to link an intact to a corrupt copy. If files would be bit identical the corrupt one wouldn't be corrupt. Rather than ignore EXIF, EXIF might be just the thing you actually want to compare. In my experience with visually corrupt JPEGs, the 'header' and EXIF marker is often intact. \$\endgroup\$
    – Joep van Steen
    Commented Jul 12, 2021 at 0:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.