3
\$\begingroup\$

I have 15 years or digital photos and I am in the process of creating a 'master archive'. The photos are spread across multiple HDDs with different folder structures, with many duplicate photos. I am looking for a script or off-the-shelf app which will scan a HDD for any photos (common file formats, mainly JPEGs) and give me a list of the photos which are NOT in the master archive. I can then manually add the photos into the archive (It pretty much needs to be done manually so I can categorize them as I go, but I am ok with that).

I have searched the web extensively and cannot find a straightforward way to do this (which does surprise me - I can't be the only person in the world with a messy collection of photos across multiple HDDs looking to sort it out!). Most duplicate finding software focuses on finding the duplicates - I want to find the EXCEPTIONS, those missing from the master archive. Synchronization software is good at finding the exceptions (missing files), but most expect folder structures to be the same - As I have tried a few categorization methods over the years, the folder structures aren't identical.

99%+ of the photos will be bit-identical copies with identical timestamps. I don't need to worry about getting file hashes/checksums or finding resized versions of the same file or visually similar photos. Besides. doing that on 1TB of photos would take too long anyway.

Can anyone think of a way to do this? I have been searching for some time and haven't found a good way to do this. I know some photo library management software (including Picasa) does duplicate checking, but I don't really want to hand folder management over to Picasa - I tried it a few years ago and it messed with the metadata, changing file modified dates and making whole management process harder overall. If anyone knows of an Open Source photo management package which will NOT enforce its own library structure, please let me know.

Btw, Windows is my target platform.

Thanks in advance.

Improve this question
\$\endgroup\$
2
  • \$\begingroup\$ For copies of unchanged files (binary identical) there are utilities for duplicate detection and cleaning. I use ThumbsPlus for managing photo files, catalogging my directories as they are including off-line media. \$\endgroup\$
    – JDługosz
    Commented Jan 8, 2015 at 20:12
  • \$\begingroup\$ Do the duplicate photos also share the same filenames? Also, could you clarify your question a bit? I am not sure if you want to check whether there are "single" images in a given HDD or compared to a "master" HDD, meaning that you don't care about how many copies are there on the same HDD, but on different ones, or if you want to have at least a single copy on your master and don't need any other. Also, does any of the files have spaces in it or in the folder structure? \$\endgroup\$
    – NuTTyX
    Commented Jan 11, 2015 at 1:50

2 Answers 2

Reset to default
3
\$\begingroup\$

Hashes are actually the key to doing this and getting it right.

If you're up to getting your hands a little dirty, this would be an easy effort as a shell script:

  • Generate a list of hashes of every file in your master archive. Call this list "MA ("master archive").
  • Generate a two-field list of hashes and paths to every file in your other archives. Cal this list OA ("other archives").
  • Extract the list hashes in OA. Call this list OH ("other hashes").
  • Pull out a list of every hash in OH that isn't in OA. (Off the top of my head, fgrep -xv -f MA OH). Call this ML ("missing list").
  • Pull out a list of every file in OA that matches a line in ML. (fgrep -f ML OA | awk -f '{ print $2 }',).

The end result of the last command will be a list of the files in your other archives that don't match one in the master.

For Windows, there's a package called Cygwin that includes all of the utilities you'd need to do this (bash, find, sha1sum, grep and awk).

Improve this answer
\$\endgroup\$
0
\$\begingroup\$

You might try looking at rsync. If you set it up right and then turn on a flag that will output like it's transferring files, but isn't actually, it'll probably tell you which files are missing.

Improve this answer
\$\endgroup\$
3
  • \$\begingroup\$ I have used rsync previously, but I thought it required identical folder structures. Do you know if there is a flag to ignore folders and just look at the files themselves? \$\endgroup\$
    – John Smith
    Commented Jan 8, 2015 at 4:24
  • \$\begingroup\$ @JohnSmith you would have to map one folder to another specifically, which could be quite intensive depending on how different your structures are. If the differences are limited to the top of the tree it'd be easier than if they're different all the way to the lowest level folder. \$\endgroup\$
    – tenmiles
    Commented Jan 8, 2015 at 4:26
  • \$\begingroup\$ I have found a StackExchange article which may do this very thing. Now I just need to think through how this would work for me.... stackoverflow.com/questions/14661033/… \$\endgroup\$
    – John Smith
    Commented Jan 8, 2015 at 4:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.