4
\$\begingroup\$

Problem description

Hi photography community,

as a hobby photographer I organize quite a lot of photos for my family and friends. I have a lightroom library of around 90k photos that accumulated over the last 25+ years. All of them are JPG files and I organized them by:

  • Renaming them to a common file name scheme IMG_YYYYMMDD_MMHHSS_Author
  • Corrected date taken metadata (older pictures often had no date/time set at all or wrong clocks)
  • added rating, keywords, removed duplicates, etc.

A lot of these pictures are taken by friends and family members and not my own cameras.

Now besides having this nicely organized library, I recently got a lot of RAW original files that correspond to those in my library. They are completely unorganized.

Actual question

Is there a tool/software that would help me match/associate those RAW files to the organized ones? There is no way to match them using any metadata, because both filename and date taken are typically completely different. The tool would have to analyze the image content and match them based on that.

Own research already done

All I found on the internet are tools for finding duplicates based on image content, but they do not explicitly try to find 1-to-1 matches of one (RAW) folder to another (organized JPG) folder.

Thank you very much in advance for any helpful hints!

\$\endgroup\$

5 Answers 5

2
\$\begingroup\$

Not a full answer because I cannot suggest a tool.

If you didn't destroy the EXIF, there is plenty of information in there which, in combination, is possibly fairly unique per picture:

  • Camera model and serial number will discriminate pictures from different cameras.
  • With the same camera a combination of ISO, Exposure, Aperture, Measured EV, Focus distance, Focal length, and possibly some AF information should be fairly unique and be encodes identically in the JPEG and the raw file.

Experiments:

Generate data for the collection on my hard disk:

exiftool -progress -r -ISO -Aperture -ExposureTime -MeasuredEV -MeasuredEV2 -FocusDistanceLower -FocusDistanceUpper -FocalLength -csv . > allData.csv

Churn all the data with a quick Python script:

#! /usr/bin/python3

import sys,os

# Key is EXIF data, value is the file name
jpegs={}
raws={}

collisions=0

with open(sys.argv[1],'r') as file:
    for l in file:
        name,data=l.split(',',1)
        if 'IMG_' not in name:
            continue
        if name.endswith('.JPG'):
            if data in jpegs:
                collisions+=1
                print(f'{name} collides with {jpegs[data]}')
            else:
                jpegs[data]=name
        elif name.endswith('.CR2'):
            if data not in raws:
                raws[data]=name
        else:
            pass; # ignore other types 

print(f'Jpegs: {len(jpegs):d}, Raws: {len(raws):d}')
print(f'Collisions: {collisions:d}')

orphanRaws=0
for data,name in raws.items():
    if data not in jpegs:
        orphanRaws+=1
    
print(f'Orphan raws: {orphanRaws:d}')

` On my collection this simplistic approach yield 12% collisions, but....

  • I found that I had a significant number of duplicates
  • Most collisions come from either bursts or pictures taken in all manual mode during the same session, so these are at least the same subject.

The last test n the script also shows that given the EXIF, most CR2s can be linked back to their JPG (and those that are not turn out to be culling leftovers that should be erased)

\$\endgroup\$
5
  • \$\begingroup\$ I have to admit that is a pretty creative idea, basically fingerprinting all bits and pieces in the metadata without looking at the image content. I like the idea and I will write a script and try this out. Thanks already! I will report how things went \$\endgroup\$ Jan 17, 2023 at 21:40
  • 1
    \$\begingroup\$ Just camera model/serial and date/time stamp are likely enough to pair most JPG/RAW combos. For those that were taken in burst mode (I did a lot of sports photography of my kids), it'll get you 2, 3 or 4 that are matched to the second and you can sort those by hand if there aren't too many. \$\endgroup\$
    – FreeMan
    Jan 19, 2023 at 18:31
  • 2
    \$\begingroup\$ @Freeman If you read the OP, the problem is that the timestamp cannot be trusted. \$\endgroup\$
    – xenoid
    Jan 19, 2023 at 20:53
  • \$\begingroup\$ I'm no EXIF expert, @xenoid, but isn't the timestamp in the EXIF data (like date/time taken) going to be consistent, even if the OS timestamp is changed because the pic was edited? \$\endgroup\$
    – FreeMan
    Jan 20, 2023 at 12:51
  • 3
    \$\begingroup\$ @Freelan See the OP's posts & comments: "Corrected date taken metadata", " yes, I have corrected the DateTimeOriginal via exiftool ". This whole question arises because these cannot be trusted in this particular photo set. \$\endgroup\$
    – xenoid
    Jan 20, 2023 at 12:57
1
\$\begingroup\$

IMatch (from photools.com) can easily do this. It was originally created based on the author's graduate work on matching images based on what they look like instead of just matching bytes. But it costs $150 (one time cost), and you would have to load your entire collection of images into it's database. Now, it doesn't move files when you load them in, it just adds them to the database.

Personally, I use it for all of my Digital Asset Management, because I didn't like Lightroom's features way back in the day. Now I use it because I don't like Adobe's subscription model. But, I also find IMatch to just be so much more powerful and flexible than anything else out there.

\$\endgroup\$
1
  • 1
    \$\begingroup\$ Thanks a lot! I think this is actually what I was looking for \$\endgroup\$ Jun 21, 2023 at 10:01
0
\$\begingroup\$

If the JPG files have all the meta saved you can use for example one XMP tag (if available):

PreservedFileName   

This tag (on my workflow) keep the point to the name of original RAW file from which JPG was rendered. Also you can check for

RawFileName

If none of them exist you can search for

Date/Time Original  
Create Date     

And map with them in RAW files.

All this mean using commandline tool like exiftool and map the data in tables. Unfortunately I am not aware of tool which can do this automatically

P.S. You can try ACDsee, when I use it in the past this tool had option to compare images by content.

\$\endgroup\$
6
  • \$\begingroup\$ Unfortunately no, the jpg files do not have the preservedfilename set, also not the original date/time. I know that because I organized them with various tools like bulkrename and exiftool. So, as I said there is no way to link them via metadata, but I need a tool that really looks at the image contents. \$\endgroup\$ Jan 16, 2023 at 8:03
  • 1
    \$\begingroup\$ Did not know of RawFileName before, but that does not help unfortunately - It just returns the same jpg name :/ C:>exiftool IMG_20220924_100407_Phil.jpg -rawfilename -> Raw File Name : IMG_20220924_100407_Phil.jpg. For this file I know that the original raw is named IMG_0464.CR2 \$\endgroup\$ Jan 16, 2023 at 8:11
  • \$\begingroup\$ @PhilipDaubmeier, this (same name) mean for me these images are edited many times, RAW->JPG->JPG. :( You can try ACDsee, when I use it in the past this tool had option to compare images by content. \$\endgroup\$ Jan 16, 2023 at 8:15
  • 1
    \$\begingroup\$ Thanks for the hint, I will have a look at ACDsee \$\endgroup\$ Jan 16, 2023 at 8:16
  • 1
    \$\begingroup\$ You mean the "CreateDate" EXIF data in the JPEG and the matching RAW are no longer the same or is it just a mismatch of the file time stamp? \$\endgroup\$
    – xenoid
    Jan 16, 2023 at 11:07
0
\$\begingroup\$

Hugin or other panorama softwares are using SIFT type algorithms („scale invariant feature transform“) to identify matching points when automatically stitching panoramas. These algorithms are used for object detection and matching those objects from different view points.

Theoretically they can be used to match images as you want. You might get some false positives, if a series of pictures was taken or the same object was photographed repeatedly. Also beware of clouds.

Unfortunately, I do not know of an implementation, that is able to do what you want. I hope you‘ll be able to find something with this hint.

\$\endgroup\$
0
\$\begingroup\$

If you want to easily compare the image content, reduce the content. Make low rez thumbnails of your images, say 16px*24px, and grayscale them. That should help raw and jpg match by glossing over white balance issues. Make sure the thumbnail file names match the orig, so that you can find your way back from the breadcrumbs, and make sure to use jpg to encode, quality something like 25% (yeah it should be crappy).

You can now sort a combined (raw+jpg) thumbnail export folder by file size and the thumbnails should sort by image content, regardless of if they came from jpg or raw. Thus, the same photo versions will be right next to each other in the sorted view. I've used this method to "re-connect" ~85% of a batch of 200-some wedding reception pictures whose filename got stepped on by an inadvertent workflow naming issue.

I never bothered since it was only a couple hundred images, but I wanted to mess around with smart file renamers or a batch file that would automatically name every-other image in a size-sorted list after the other image. I would seriously play around with that if i had 10s of 1000s of orphaned siblings to re-unite.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.