21

Sometimes when you're downloading an image and the connection breaks mid stream, you are left with a half downloaded image. If you try to view it you get the upper part of the image and the bottom part is usually coloured grey or green or some other color. In other words, it is corrupted.

Is there a way to check whether the image is damaged in that way or otherwise corrupted?

15

If you are talking about JPEG files, then the utility jpeginfo is exactly what you're looking for. It can check files for different types of JPEG errors and corruption and either return an error code (the most useful thing for scripting), or just delete files with errors.

I use this as part of my initial file transfer, to make sure everything copied okay without relying on manual checking. (After that, I make sure their checksums don't change as part of my normal backup/bitrot protection.)

The program is command-line, and comes as source code, but it should be easy to build and use on any Linux distribution or on a Mac with a development environment set up properly. I'm sure you could even do it on Windows with Cygwin or MinGW. (For example, although I can't vouch for its integrity, this blog post seems legit and includes a precompiled download.) To build it yourself:

$ git clone https://github.com/tjko/jpeginfo.git
Cloning into 'jpeginfo'...
[...]
Checking connectivity... done
$ cd jpeginfo/
$ ./configure && make

This should create a jpeginfo command which you can either run in place or copy wherever you want (possibly using make install).

Then, you run it like this:

$ ./jpeginfo -c *.jpg
test1.jpg 1996 x 2554 24bit Exif  P 6582168  [OK]
test2.jpg 1996 x 2554 24bit Exif  P 6582116  Premature end of JPEG file  [WARNING]
test3.jpg  Corrupt JPEG data: 1 extraneous bytes before marker 0xe2 1996 x 2554 24bit Exif  P 6582169  [WARNING]

Here, test1.jpg is perfectly fine, and test2.jpg I deleted a few bytes from the end, and test3.jpg I changed some random bytes in the header.

If you have RAW files, check out this page from the American Society of Media Photographers on DNG Validation, or one on data validation details, which covers using Adobe's DNG converter to batch-validate proprietary RAW formats. (Unfortunately, this is a GUI operation and not necessarily easily scriptable.)

If you have a camera which natively outputs the 1.2 version of DNG, that's even better, as this includes a built-in MD5 checksum of the image data. Unfortunately, this doesn't seem to be stored with the normal image metadata — or at least exiftool and exiv2 don't recognize it, and they do read 1.2 DNG files in general — which means that as far as I know currently the Adobe validation tool is the only way to take advantage of that too.

  • Do you know whether Windows binaries for jpeginfo exist somewhere? – Rook Aug 25 '15 at 13:47
  • 1
    Using jpeginfo tool by git clone doesn't seem to be possible on Windows, because 'aux' seems to be a Windows reserved name, and git cannot clone the aforementioned directory into existence. – Rook Aug 30 '15 at 10:25
  • --- resuming the conversation from the other post here; Unzipping the archive gives an error because of 'aux'. Renaming 'aux' within the archive helped with unzipping and then renaming it back to 'aux' within cygwin solved that problem. But running make from cygwin resulted in numerous errors still; something about wrjpgcom.c:87:54: warning: incompatible implicit declaration of built-in function 'exit' [enabled by default] #define ERREXIT(msg) (fprintf(stderr, "%s\n", msg), exit(EXIT_FAILURE)) (just one of many) – Rook Aug 30 '15 at 15:02
  • @ldigas I built a MinGW binary which you can find at mattdm.org/misc/jpeginfo-w32/jpeginfo.exe. I built this on Linux as a cross-compiled executable, so haven't tested it, but it seemed to build okay. I can't promise that it works, but I do promise that it just the upstream code and has no viruses or anything. :) – mattdm Aug 30 '15 at 15:04
  • Upvoted this a few minutes ago for the effort you're making, but it doesn't seem to work all that well on Windows. jpeginfo -c any_jpeg_file.jpg I provide it, it seems to report Premature end of JPEG file JPEG datastream contains no image [ERROR]. – Rook Aug 30 '15 at 15:20
2

ImageVerifier did what you wanted. Unfortunately it is not available for download anymore and support has been discontinued on 31-December-2017 (see Ingestamatic and ImageVerifier no longer for sale).

Old answer for historical reasons

ImageVerifier (IV for short) traverses a hierarchy of folders looking for image files to verify. It can verify TIFFs, JPEGs. PSDs, DNGs, and non-DNG raws (e.g., NEF, CR2).

IV is designed to process large numbers of images. Folder hierarchies with 100,000 images or more should be no problem. In one test run, IV ran for 14 hours.

There are two kinds of verification that IV performs: Structure checking and hash checking.

http://basepath.com/site/detail-ImageVerifier.php

  • It sounds like you are associated with the ImageVerifier, if so, can you please disclose this in your answer. – damned truths Aug 31 '15 at 1:01
  • 1
    I am not associated with the product at all. I had need to verify some image files after a NAS crash and used this tool. I just cut pasted the text from the site to give a description. – Kez Aug 31 '15 at 13:39
  • FWIW - Its good for camera files (jpgs and various RAW formats - its primary intended use) but not so good for other file types without codecs, etc. The -identify function of ImageMagick is another option – Kez Aug 31 '15 at 13:42
1

If this is not about downloading images from your camera, but a computer-to-computer transfer, a common approach to file integrity are checksums.

Unfortunately, as far as I know, common "end user" image formats (jpeg, png, gif, …) are not integrity-checked on their own. But as I understand the question to imply automated processing, integrating checksum tools (CRC32, MD5, …) into the workflow could be a viable solution. A common approach to store the checksum is to have a file with the same filename, just with an added extension, like: img123.jpg → img123.jpg.md5.

This approach has the added benefit that you can also check the integrity of (for example) sidecar files or anything else you want to transfer in a similar mechanism. And if you keep the checksum files around, even in the future. (And it has the downside of not being integrated in PS, LR, or the other common tools to the extent of my limited knowledge.)

  • 1
    It is worth noting that DNG does contain a checksum, and can be verified directly in Lightroom. – Hampus Nilsson Jan 15 '14 at 18:53
  • I was not aware of that! Excellent. Makes sense, too. I edited the answer to make clearer I aimed for "end user" formats more than archival formats, though it's sweet that DNG helps with checksums. – Cornelius Jan 16 '14 at 11:41
  • I use "Advanced Checksum Verifier" (ACSV) by Irnis Haliullin, to calculate MD5 checksum files that are copied to backup medium along with the original files. ACSV runs in batch or interactive. The integrity of the copy can be verified any time by re-calculating the checksum, and comparing to original. – Pierre Jul 31 '16 at 13:04
1

I developed check_media_integrity a simple python script check_mi.py, you can download it from GitHub:

https://github.com/ftarlao/check-media-integrity

I quote the guide intro:

check-mi is a Python 2.7 script that automatically checks the integrity of media files (pictures, video, audio). You can check the integrity of a single file, or set of files in a folder and subfolders recursively, finally you can optionally output the list of bad files with their path and details in CSV format.

The tool tests file integrity using common libraries (Pillow, ImageMagik, FFmpeg) and checking when they are effectively able to decode the media files. Warning, image, audio and video formats are very resilient to defects and damages for this reason the tool cannot detect all the damaged files.

check-mi is able, with 100% confidence, to spot files that have broken header/metadata, truncated image files (with strict_level >0), and device i/o errors.

check-mi is, usually, not able to detect all the minor damages--e.g. small portion of media file overwritten with different values. In detail, I have tested strict_level 1 with a small randomized experiment, executed on a single 5MB jpeg picture:

Overwriting a portion (interval) of image file with zeros, you need interval size = 1024KBytes in order to get 50% chance of detecting the damage. Overwriting a portion (interval) of image file with different random values, you obtain about 85% detection ratio, for interval sizes ranging from 4096bytes to 1024Kbytes.

In the case you know ways to instruct Pillow, Wand and FFmpeg to be stricter when decoding, please tell me.

0

The accepted answer refers to the use of jpeginfo, which is a really old and non-maintained tool written in C (and also not very modular / extensible). Also, that tool seems to just look for some specific EXIF data points (skim through the source code for ~5 minutes).

IMO, a better tool called, file-type, is very easy to use – basically copy-paste their example code and modify the file name if you don't know how to code. It checks the magic numbers associated with certain known filetypes and lets you know what kind of file you're dealing with.

I'm still looking for more layers of protection than just this. For instance, if arbitrary data is stored past (or in) the EXIF metadata, or after the magic numbers, that can pose security problems. I'll continue to look into more security measures and hope to later update this answer.

Here's the example code copied from their webpage, for the lazy:

// Node.js
const readChunk = require('read-chunk');
const fileType = require('file-type');

const buffer = readChunk.sync('unicorn.png', 0, fileType.minimumBytes);

fileType(buffer);
//=> {ext: 'png', mime: 'image/png'}

FYI, this tool is constantly being updated (3 days ago was the last update, as of my original answer here), and they currently have 3,691,850 weekly downloads – so that's probably a good indication.

  • Typical magic number based file type identifiers usually just focus on the first n bytes, so this might not help with a partially committed image file, which is the basis of the question posed here. That is, it is very common to have a JPEG or PNG that POSIX file (which operates in this same manner) will report on correctly, but will fail to render because much of the data is actually missing. – user31502 Jan 25 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.