0

This question already has an answer here:

Let's assume a JPG file (or any other image/video file format), that I duplicate the file, and I modify the tags on the copy. Is there a way to verify that the image inside both JPG files is actually the same? I mean, if I do a digest of both files, I will obviously get a different hash. But is there a way to extract from the JPG only the bytes representing the image, so that I can do a digest of these bytes and see that both files are actually the one and same image?

The above is in the context of trying to automate how I manage my photos, i.e. extracting photos from my phone to my pc, detecting duplicates before I back things up, etc. In case it would influence the answers, my home made tools are written in Java, sometimes using external tools like exiftool.

Addendum

There are two other threads (How can I identify duplicate image files? and Consolidate photos and eliminate duplicates? [duplicate]) that appear to be duplicates, but in my opinion, they're not. Firstly, the second thread is all about Mac and about file naming conventions based on EXIF data. That is most certainly not what my question is about. As for the first thread, it asks for tools that can find duplicates. While a plethora of answers were provided, none of the tools were doing exactly what I wanted (and most were not free) except for the very last answer "Dedup-image" (which incidentally had the lowest score in that thread...). When I dug into the script, I found exactly what I wanted, and this happened to be exactly the answer provided by @Kahovius

In short, the answer from @Kahovius:

  • is free
  • is able to recognize that two files having different hashes (because they have different EXIF metadata) actually contain the exact same image (i.e. exact same bytes)
  • can be automated (in BASH scripts for instance)
  • (as a bonus) can even detect images that are duplicate when the only transformation was a lossless rotation (the actual bytes of the image are the same as the rotation is just a tag that was added to the file).

All other tools mentioned in the first thread were either not free, or would compare files as blobs (i.e. two files with same image bytes but different EXIF metadata would be seen as different), or would do some heuristics when comparing images (e.g. an image which has red eye removed could be considered duplicate of the original image). None of this is what I wanted.

marked as duplicate by mattdm, Hueco, Michael C, Romeo Ninov, xiota May 14 at 2:55

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • Consider converting them to .BMP and hashing them. – rackandboneman May 13 at 15:32
  • I agree that it's probably not a duplicate. The OP of this question specifically says exact matches (same image, same metadata) are good enough (though many of the answers then actually answer the related but different question about detecting similar images). Whereas this question seems to be about detecting duplicates based on filename. – Kahovius May 14 at 16:49
2

Here's a quick and dirty solution with exiftool and cmp:

1) Make copies of the two image files:

cp img1.jpg tmp1.jpg
cp img2.jpg tmp2.jpg

2) Use exiftool to remove metadata:

exiftool -all= tmp1.jpg
exiftool -all= tmp2.jpg

3) Use cmp to compare the temporary files:

cmp tmp1.jpg tmp2.jpg

cmp exits with a message if the two files are different; if they're the same, there's no output.

2

There are different levels of "The same image"

Some programs mentioned in the "Possible duplicate of How can I identify duplicate image files? analyze a fingerprint of an image. I understand that they analyze sort of a low-resolution version of the images and compare them.

This is to compare a small matrix of data to reduce time.

Some of them mark "similar images" for example photos from the same photoshoot.

Some of them actually identify pretty well the same image, even if they are a different resolution for example. They are the same but not really.

Another level of "The same image" can be the same resolution but different compression.

But this would need to examine all the data and compare them.


Some programs only read the byte count and date for example. If they both measure exactly the same bits and has exactly the same date, they are probably a duplicate.


But if you want to see if they are exactly the same image, as a photographer you can stack one on top of the other and use the difference blending mode. If the image is totally black, this means the image has no difference in pixels.


I guess one way to compare them is to open an image as a binary data or as hexa data. For example, in Irfanview, you can open an image, press F3 and see both formats.

enter image description here

There are some libraries for PHP that let you convert an image to hex code as text, so probably you can use some of that.

0

If you think the only difference is metadata (which is enough to break comparison of finger prints), you can extract the actual data without decoding it. Be careful that JPEG is really two formats: the old JFIF and the more recent EXIF (which isn't only about metadata), but I doubt that applications will convert from one to another without re-encoding (and thus breaking a comparison).

Not the answer you're looking for? Browse other questions tagged or ask your own question.