4
\$\begingroup\$

Today I found out "by accident" that some of my CR2 Canon raws are corrupt. The EXIF data and the file structure is fine, it's the raw image data itself that appears to be broken. The Adobe DNG converter doesn't report any errors and I can also import the files into Lightroom without warning. The previews look flawless at first but then they start to show distortions. I assume the initial preview I see is the embedded JPG and once the files are imported, Lightroom generates new thumbnails based on the raw data. My problem is that we're talking about 18.000 files that can be affected and I have no idea how to identify the broken images other than looking at each and every file. I tried a program called ImageVerifier but it also doesn't report any error. Don't CR2 files have checksums that can be used to spot broken files? If they do, is there any program that can verify the integrity?

Edit: Here are some pictures showing the problem:

enter image description here

And:

enter image description here

\$\endgroup\$
9
  • \$\begingroup\$ my knowledge on this subject is fairly limited. but it is my understanding cr2 don't have a checksum and that if you used adobe's dng it creates a checksum, so one advantage to using that file format. \$\endgroup\$ Dec 12, 2016 at 2:25
  • 2
    \$\begingroup\$ Creating a DNG file from a corrupt .cr2 file will just give a checksum that verifies the DNG is the same data as contained in the corrupt file from which it was created. \$\endgroup\$
    – Michael C
    Dec 12, 2016 at 8:49
  • \$\begingroup\$ @smares Can you post an example of what the "distortions" to which you refer look like? \$\endgroup\$
    – Michael C
    Dec 12, 2016 at 8:50
  • \$\begingroup\$ @MichaelClark Updated the post \$\endgroup\$
    – user59122
    Dec 12, 2016 at 23:58
  • 1
    \$\begingroup\$ I would probably regenerate the thumbnails for the CR2 files separately, then use some image comparison software to check for differences between the original JPG thumbnail and the newly generated one. If the difference is larger than say 5% it's probably corrupted. I think you can probably use imagemagick for the image comparison, and potentially also for the extraction as well \$\endgroup\$
    – SztupY
    Dec 13, 2016 at 0:07

1 Answer 1

2
\$\begingroup\$

This has gone unanswered for along time so I'll take a shot at a summary.

Most raw files, including as best I know Canon, do not include a checksum in the original file as delivered by the camera, so there is no sure way to know a given copy is correct. Some may be so damaged they fail to import, but often they are wrong when viewed but still "work" in most programs.

I believe the closest most of us come to a workable solution to the issue is to (1) review the files as soon as you can, and fix any that have issues, plus (2) use a workflow that is designed to detect and correct "bit rot" afterwards.

Often if detected early, while a copy of the image(s) are still on the card, recovery software or just downloading again may correct the issue. Obviously the problem is noticing quickly, and I have no good solution there other than review soon after shooting.

The second half is, in my opinion, even more important - digital images can "rot" over time. Yes, they can become corrupt just sitting there with you doing nothing to them. Copying to replacement computers, disks, edits that go bad, corruption on disks (e.g. index problems that cause file A to also be saved over top of part of file B), and just disk read errors can silently corrupt images. Prolific photographers can quickly start pushing the expected error rate of hard drives as they collect terabytes of data; and some such errors are not detected by hardware, and even if detected by hardware the desktop operating systems we use tend not to handle them reliably (or even report them at times).

A number of techniques can address this:

1) Use tools that do checksums of the file once loaded, and check them against recalculated checksums later. Image Verifier was a commercial tool for that, and others have contributed public domain tools (I wrote one called LR Validate for windows/Lightroom).

2) Use file systems that build in such detection, e.g. zfs and btrfs, and microsoft's new refs (I personally think it is too immature). A "zfs scrub" for example will check every file to see if it still checksums the same.

3) Convert to DNG and use the checksum validation (this is only a partial validation of the file contents; it checks the image portion but not all the metadata, so it may detect image corruption over time, but fail to report that your develop settings were lost). This is the easiest technique for most people (though it comes at a cost of using DNG -- I personally do not do so).

All of these are predicated on the idea you do backups, and when a "rotten" situation is detected can recover from one of (presumably) several copies backed up.

However, I realize this does not address the core issue of "if I may have allowed corruption in, how can I find them now". The best suggestion I have is to start with a mass DNG conversion (you need not do this to keep, you can just run the program standalone against folders to an output folder you plan to trash). Image Verifier, if I recall, does this internally. This will not catch all errors, but it may find some.

The rest, I believe you will need to find by reviewing by eye. You can just let lightroom (if you use it) rebuild the previews and scan them by eye; gross problems may jump out, minor ones likely will not.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.