1

I need to test all my raw NEF files to see if any are corrupt. Had a hard drive issue, and while a few folders seem fine, I found one with a number of corrupt raw files. Need to scan my entire catalog to see what is affected.

  • What do you mean by 'corrupt'? You cannot load them at all? Or the files are shown but with wrong colors/patterns etc.? Can you upload a sample somewhere? – John Thomas Apr 2 '14 at 5:28
  • Corrupt -- can't open in Photoshop or Lightroom, I have pink bands across the bottom of the image. – pedram Apr 2 '14 at 13:08
  • Aha! Common effect of one (or more) bad sectors. Oh well... Or, to be more precise: You CAN open in Photoshop BUT it has those pink bands on the bottom of the image. (this means that the file structure/header is intact, just data is damaged). I'll update my answer. – John Thomas Apr 2 '14 at 13:40
  • Thanks I will try this at home tonight and accept if all goes well. Cheers – pedram Apr 2 '14 at 15:04
  • 1
    As expected (unfortunately). That's why a (command-line) program which 'automatically' detects corrupt files cannot exist because, as I said, the file (the header and 'tile' boundaries/structure) can be ok, but the image can be damaged because NTFS/ext4/etc. blocks are much smaller that a 'a data tile' in TIFF on which NEF (and almost any Raw format) is based, hence in most cases the file is not 'corrupted' but the data can have bad (FFFF00FFFF00...) areas/blocks. That's why I gave more stages in detecting the corruption, included the quick visual inspection. To be continued... – John Thomas Apr 3 '14 at 4:51
1

Ok - because if you didn't answer to my comment I try to give you a general, cross-platform solution.

Download XnView MP for the platform you need (it supports Win, MacOSX and Linux in 32/64bit).

  1. With the Explorer find how many files are in your archive (right-click, Properties etc.)

  2. Go with the browser of XnView MP in the root of your photo archive, being it D:\ , X:\Photos etc.

  3. Select the folders by using

    • Shift-click in order to select a range ...and/or...
    • Ctrl-Click in order to select separate folders one by one. ...or...
    • Press Ctrl+A (or Edit | Select All...) to select all folders

    Now you have selected the folders which you want to check.

  4. Go to Tools | Batch Convert (or press Ctrl+U) (...if you have many files you need to wait a little)

  5. In the Batch Convert dialog which appears go to first tab called 'Input'. You should see something like this:

Batch Convert

As you see in the name of the tab is written the number of files ready for the Batch Convert.

First Check: Are all the files present in there?

If not, select fewer folders (half of them) and repeat the steps from 0.

Second Check: Are all thumbnails correctly displayed?

If yes, then the NEF files have the embedded thumbnail/preview ok (hence you can recover something) but this does NOT mean that the RAW data is intact. Hence a simple import in Lightroom/Darktable isn't enough because these programs extract the embedded preview from RAW file.

Now go to Settings tab. There press the Load format settings button and for Camera RAW choose in the "Use" drop-down the 'Full Size' value. We want now to test the integrity of the actual RAW file.

Here is a checkbox saying 'Ignore read errors'. Experiment the next steps with this checkbox disabled and after that, enabled.

Go to Output tab. You should see something like this:

Output tab

In the Folder field enter a folder path of your choice (be sure to have enough space, if needed!).

EDITED: DO THIS VARIANT: In the Filename leave it as is if you want to see all the raw files converted. This is useful to see if perhaps the NEF file structure is intact but is was altered only one or two blocks inside of it (changed colors, random colored blocks etc.). However doing like this, it will take you disk space to store the output files.

EDITED: THIS VARIANT IS NOT NEEDED: But if you're interested just to see if the NEF files reads correctly then in Filename field write a static text like 'temp' and in the Options pane choose 'Replace' for the 'When output files already exist' drop-down.

EDITED: THIS IS NEEDED: Also in the Options pane, if you check the Keep folder structure check-box, the entire tree structure of your photo archive will be duplicated at destination. Pretty useful if you intend to keep the destination and/or replace the corrupted NEFs.

In the Format pane choose an easy-to-view-and-small file format (usually JPEG) with medium resolution (click 'Settings') but you're free to choose anything else if you want to save disk space and/or time (WebP, JPGXR etc).

  1. Run the Batch by pressing the Convert button. This can be done overnight. Anyway, the Batch engine will tell you in how much time will be ready and, generally, will present you the status of happenings.

  2. At the end you'll see the log:

Processing Log

Have a loot at it to see where are the errors by scrolling / maximizing the window. Alternatively you can copy the log to a word processing program (Notepad/MSWord etc.) to do a Find/Search for 'error' etc.:

  • Right click and choose Select All (or press Ctrl+A)
  • Right click and choose Copy (or press Ctrl+C)

And now paste it in your text processing program of your choice and analyze it.

All the above is much more straightforward and simple than it sounds.

EDITED: ADDED: With XnView MP browser go to your destination folder (in my example c:\output\temp) and in the Folders Tree Pane right-click on the folder and choose (if your number of files vs computer memory permits) Show all files (recursive):

Show all files

This will present you a flattered view of all your subfolders from your temp directory and you can inspect the saved files with ease. Also, if you have two monitors you can put the Preview/FullScreen pane on the 2nd monitor (see Tools | Settings for this).

EDIT 2: If the file structure is ok but the Raw data is damaged, then you have the following options:

  1. Extract the embedded previews which on most modern cameras are 100% Full Quality JPEGs of the actual Raw data (ok, sometimes the previews are a bit 'doctored' by the camera). How to do it: In Batch Convert dialog, go to Settings tab and press Load Format settings button. In Camera Raw tab choose for the 'Use' drop-down the 'Embedded Preview' value. Save the setting and re-run the batch on the files in another destination folder. Perhaps you will use a loseless format like PNG. Now you have two folders: one with the actual raw data and one with the embedded previews from which you can choose the best (least corrupted) image (NOT file - all the files are ok).

  2. Move/Delete/Rename the 'hopeless' NEFs. Go with XnView MP to the corrupted files. Tag them. There is a tag box in the top-left corner, or much better, press 'T' key to tag/untag the selected files. Be sure that in Tools | Settings | Browser | Tag the value for Tag filelist must be cleared has the 'Never' value. Alternatively, you can change the shortcut from Tools | Settings | Interface | Shortcuts. When all your files are tagged, you can go to Filter Pane in the top-left corner of the browser (in the default layout) and choose from there 'Special Items\Tag'. Now you see only the tagged files. You can Select All (Ctrl+A) and Move them (Edit | Move to...) and/or Rename Them (Tools | Batch Rename...) - eg. to append at the end a "_corrupted", and/or delete them. It is up to you.

...boy, I wrote an entire novel here... :) But it is your data and it is pity to lose it.

  • 1
    Some info added based on your response at my comment at your question. If you find that this is the solution at your problem don't forget to accept it. :) – John Thomas Apr 2 '14 at 13:47
  • 1
    Some more info added - how to try to extract all the valid data (because usually the files are ok, just the image data is corrupted) and how to manage the files which contain the corrupted images. – John Thomas Apr 3 '14 at 5:11
  • Wow..great ideas, thanks again. marked as best by a long shot. – pedram Apr 3 '14 at 14:16
0

I had face the same problem few times.

My approach was, importing the root folder containing all the sub-folders and RAW files into my RAW processing software(Lightroom/Darktable). Corrupted files appears with a special 'broken' thumbnail.

  • This approach, even if it can find problems, isn't bulletproof because usually these program use the 100% embedded JPG preview which resides in the RAW file. Hence, the thumbnail can be ok while the actual Raw data is damaged. – John Thomas Apr 2 '14 at 12:56
0

Before you try messing with the programs below, make a backup of your pictures!!!

You could use

ufraw-batch --out-type=jpeg *.NEF

to create JPEG versions, and quickly review those. (Man page can be found here.)

Alternatively, you could try using exiftool for data manipulation or dumping values and verifying the data you suspect to be damaged. (Man page can be found here.)

E.g.

exiftool -b -JpgFromRaw image.NEF > image.jpg

Both run on Linux, but you can use them in Cygwin or MinGW on Windows.

If some more elaborate finding is needed:

find -iname "filename_with_wildcard*.NEF" -print0 | xargs -0 -I {} exiftool ...

... meaning your command parameters, where you put {} instead of the actual file name. E.g.

find -iname "Cats*.NEF" -print0 | xargs -0 -I {} exiftool -b -JpgFromRaw {} > {}.jpg

(A bit ugly, because it will emit Cats1.NEF.jpg files, but let's keep to the simple example instead of beautifying shell commands...)

  • ufraw-batch is poor in options for this case IMHO. For example, the folders must be done one by one and the output text from console must saved in a file by using redirecting / pipes etc. Also, command line is a little bit scary for a common photographer. (Btw, I should down-vote you for that 'Cats*.NEF' but Linux saved the day :) - FYI, exiftool exists also on Win) – John Thomas Apr 2 '14 at 13:03
  • I'm not afraid of command line, I have experience with Linux. If I wanted to review individual files, I could just review each RAW file. I want an automated option that spits out a list of corrupt files. If there's a command line program that can verify whether a RAW file is valid or not that would be ideal. – pedram Apr 2 '14 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.