Tool for finding original RAW files to already organized jpeg files

Question

Problem description

Hi photography community,

as a hobby photographer I organize quite a lot of photos for my family and friends. I have a lightroom library of around 90k photos that accumulated over the last 25+ years. All of them are JPG files and I organized them by:

• Renaming them to a common file name scheme IMG_YYYYMMDD_MMHHSS_Author
• Corrected date taken metadata (older pictures often had no date/time set at all or wrong clocks)
• added rating, keywords, removed duplicates, etc.

A lot of these pictures are taken by friends and family members and not my own cameras.

Now besides having this nicely organized library, I recently got a lot of RAW original files that correspond to those in my library. They are completely unorganized.

Actual question

Is there a tool/software that would help me match/associate those RAW files to the organized ones? There is no way to match them using any metadata, because both filename and date taken are typically completely different. The tool would have to analyze the image content and match them based on that.

Own research already done

All I found on the internet are tools for finding duplicates based on image content, but they do not explicitly try to find 1-to-1 matches of one (RAW) folder to another (organized JPG) folder.

Thank you very much in advance for any helpful hints!

Answer 1

Not a full answer because I cannot suggest a tool.

If you didn't destroy the EXIF, there is plenty of information in there which, in combination, is possibly fairly unique per picture:

• Camera model and serial number will discriminate pictures from different cameras.
• With the same camera a combination of ISO, Exposure, Aperture, Measured EV, Focus distance, Focal length, and possibly some AF information should be fairly unique and be encodes identically in the JPEG and the raw file.

Experiments:

Generate data for the collection on my hard disk:

exiftool -progress -r -ISO -Aperture -ExposureTime -MeasuredEV -MeasuredEV2 -FocusDistanceLower -FocusDistanceUpper -FocalLength -csv . > allData.csv

Churn all the data with a quick Python script:

#! /usr/bin/python3

import sys,os

# Key is EXIF data, value is the file name
jpegs={}
raws={}

collisions=0

with open(sys.argv[1],'r') as file:
    for l in file:
        name,data=l.split(',',1)
        if 'IMG_' not in name:
            continue
        if name.endswith('.JPG'):
            if data in jpegs:
                collisions+=1
                print(f'{name} collides with {jpegs[data]}')
            else:
                jpegs[data]=name
        elif name.endswith('.CR2'):
            if data not in raws:
                raws[data]=name
        else:
            pass; # ignore other types 

print(f'Jpegs: {len(jpegs):d}, Raws: {len(raws):d}')
print(f'Collisions: {collisions:d}')

orphanRaws=0
for data,name in raws.items():
    if data not in jpegs:
        orphanRaws+=1
    
print(f'Orphan raws: {orphanRaws:d}')

` On my collection this simplistic approach yield 12% collisions, but....

• I found that I had a significant number of duplicates
• Most collisions come from either bursts or pictures taken in all manual mode during the same session, so these are at least the same subject.

The last test n the script also shows that given the EXIF, most CR2s can be linked back to their JPG (and those that are not turn out to be culling leftovers that should be erased)

Answer 2

IMatch (from photools.com) can easily do this. It was originally created based on the author's graduate work on matching images based on what they look like instead of just matching bytes. But it costs $150 (one time cost), and you would have to load your entire collection of images into it's database. Now, it doesn't move files when you load them in, it just adds them to the database.

Personally, I use it for all of my Digital Asset Management, because I didn't like Lightroom's features way back in the day. Now I use it because I don't like Adobe's subscription model. But, I also find IMatch to just be so much more powerful and flexible than anything else out there.

Answer 3

If the JPG files have all the meta saved you can use for example one XMP tag (if available):

PreservedFileName   

This tag (on my workflow) keep the point to the name of original RAW file from which JPG was rendered. Also you can check for

RawFileName

If none of them exist you can search for

Date/Time Original  
Create Date     

And map with them in RAW files.

All this mean using commandline tool like exiftool and map the data in tables. Unfortunately I am not aware of tool which can do this automatically

P.S. You can try ACDsee, when I use it in the past this tool had option to compare images by content.

Answer 4

Hugin or other panorama softwares are using SIFT type algorithms („scale invariant feature transform“) to identify matching points when automatically stitching panoramas. These algorithms are used for object detection and matching those objects from different view points.

Theoretically they can be used to match images as you want. You might get some false positives, if a series of pictures was taken or the same object was photographed repeatedly. Also beware of clouds.

Unfortunately, I do not know of an implementation, that is able to do what you want. I hope you‘ll be able to find something with this hint.

Answer 5

If you want to easily compare the image content, reduce the content. Make low rez thumbnails of your images, say 16px*24px, and grayscale them. That should help raw and jpg match by glossing over white balance issues. Make sure the thumbnail file names match the orig, so that you can find your way back from the breadcrumbs, and make sure to use jpg to encode, quality something like 25% (yeah it should be crappy).

You can now sort a combined (raw+jpg) thumbnail export folder by file size and the thumbnails should sort by image content, regardless of if they came from jpg or raw. Thus, the same photo versions will be right next to each other in the sorted view. I've used this method to "re-connect" ~85% of a batch of 200-some wedding reception pictures whose filename got stepped on by an inadvertent workflow naming issue.

I never bothered since it was only a couple hundred images, but I wanted to mess around with smart file renamers or a batch file that would automatically name every-other image in a size-sorted list after the other image. I would seriously play around with that if i had 10s of 1000s of orphaned siblings to re-unite.