13

I'm not a professional photographer but I do value my photos very much. Sometimes when I look through them months or years after I've taken them, some of them just show up corrupt for no evident reason. I often get things such as gray areas, altered colours starting from a certain point on or even completely undisplayable photos. I use both Windows and Linux OS on my PCs.

Why does this happen and, most importantly, how to avoid this?

  • Are these old photos? i.e., photos that used to look good, and that you didn't move around on the disk afterward? Or are they photos that you dumped onto your disk recently, only to find that they are now corrupted? – Mehrdad Jul 10 '16 at 1:14
  • 4
    I'm voting to close this question as off-topic because it's about general IT management and hard/software failures. Principles that apply to photographic files also apply to any other document, video or application. – James Snell Jul 10 '16 at 8:46
  • 3
    Answered here already. – JDługosz Jul 10 '16 at 20:27
  • How to avoid: use ZFS on a file server where I keep my photos. – JDługosz Jul 10 '16 at 20:30
17

There are two main causes. The first is data degradation. Bits stored on magnetic media (such as your hard disk) can lose their magnetic orientation over time, corrupting the bit. In harsher conditions (high heat and humidity) the physical media itself can start to degrade. For solid state media such as an SSD, the mechanism is different but the outcome is similar.

The more common occurrence is silent data corruption, whereby an error occurs during the writing or reading of data. Though the image at the previous link is an extreme example, oftentimes a single flipped bit can corrupt an image.

There are two solutions to these issues, and assuming you rotate your backups, backups alone won't help. The more complicated and technical approach is to use a filesystem with built-in data integrity mechanisms (such as ZFS or btrfs), while a simpler approach would be to compute a hash for each file between backups to detect any changes.

  • ZFS or Btrfs on *nix systems, or ReFS on Windows. – a CVn Jul 9 '16 at 14:44
  • 7
    Not a photographer, but a data integrity geek. Had to join this community just to upvote this answer. While I'm already writing a comment, I could touch a bit more on "a single flipped bit can corrupt an image". This is because JPEG and other formats by design removes as much redundancy as they can, that's their whole point. If you can flip a bit and still recognize the image, that bit wasn't worth storing (roughly). Thus, uncompressed formats can be more resilient, and degrades more gracefully. Probably not worth the extra 1000% penalty in size. – pipe Jul 9 '16 at 17:24
  • 1
    @pipe Probably not worth the extra 1000% penalty in size... I'd say exactly the opposite - shoot RAW, store RAW, backup RAW. You keep the negatives, not the prints. – J... Jul 10 '16 at 1:28
  • 1
    @J... Just be careful that your RAW format isn't compressed with a lossless algorithm, then you still have a similar problem. The best option is to follow the recommendation in the answer we are commenting on. Keeping the RAW files have, of course, other benefits. :) – pipe Jul 10 '16 at 1:34
  • 1
    Go ahead and compress and use par2 to add useful redundancy. – JDługosz Jul 10 '16 at 20:32
17

I'm not going to worry too much about how this happens, because hard disks will fail; just like anything else in this world, they aren't perfect. You can't get to a state where you're never going to lose a file.

However, you can get to a state where you never lose a photo - you do this by having multiple copies on separate hard drives, including at least one in a separate physical location. Then it didn't matter of one disk fails, because you can just get the photo from another location.

  • I wonder if there is a program that can automatically check the integrity of image files. Maybe not - a corrupt JPEG might still be a valid JPEG I suppose. – osullic Jul 9 '16 at 10:59
  • 5
    As answer-commented elsewhere, don't worry about trying to work out if they're corrupt or not, just worry about whatever they've changed. – Philip Kendall Jul 9 '16 at 12:00
  • If photos get corrupted at a filp of a bit and become unusable, why does this not happen with regular files? I suppose that if this happened as it does with photos, OSs would probably randomly fail, all sorts of documents would randomly become unreadable... but I have not seen that as of yet. – Petru Dimitriu Jul 14 '16 at 12:00
  • @PetruDimitriu Pictures and video take up a lot of space, if a single bit flips, and 90% of your hard drive contains pictures, that bit has 90% chances on being in a picture and not in any other file. If you are unlucky, and that bit happens to be in an important part of an important system file (not a readme) your computer will likely crash at some point and you will not know why. – Jens Feb 1 at 17:01
2

The answer to the why this happens depends on the type of drive, type of file, filesystem and operating system used, so it's impossible to give a complete answer to this question.

But regarding how to avoid this, the other answer touches on having backups - however, with file corruption like this, you need a way to identify when a particular copy of your file has been corrupted and avoid copying this to your backups. It's very easy to end up with 2 corrupted copies of a file. The technical answer is to use checksums and compare these to identify corrupted files.

The best way to avoid this these days, is to use an online storage service which will handle all redundancy, and file integrity for you.

  • "use an online storage service which will handle all redundancy, and file integrity for you."... and their client automatically detects the change to the local copy of the photo and uploads the new version, which is corrupted. Cloud storage is no substitute for good backup policies and some way of actually detecting corruption. Compare What's needed for a complete backup system?. – a CVn Jul 9 '16 at 14:53
  • @MichaelKjörling using the online storage as a secondary slave copy of the data will have this effect, but storing the master copy in uncorruptible storage. In general always sync from more reliable storage to less reliable storage. – Harry Harrison Jul 9 '16 at 15:31
  • 1
    @MichaelKjörling their client automatically detects the change to the local copy of the photo and uploads the new version, which is corrupted. Actually, this is an interesting point, because unless the corruption is in a software layer the client won't actually be able to detect the corruption unless you re-write the file. – Michael Jul 9 '16 at 17:00
  • 2
    @HarryHarrison I disagree. I don't want to automatically sync from (e.g.) my NAS to my local hard disk just because my NAS is more reliable - there's still a chance it's the NAS that's screwed up, not my local hard disk (or actually far more likely, I fat fingered something and messed up the file on the NAS myself). Manual intervention is what's needed here - it shouldn't be that common, unless you're Associated Press. – Philip Kendall Jul 9 '16 at 18:42
  • 1
    Nobody has uncorruptible storage. Even S3 is "only" 12 nines. – Philip Kendall Jul 10 '16 at 10:05
1

Photo files aren't special in this regard, any file on a computer can get corrupted. JPEGs are probably more likely to show up corruption in an obvious way than a lot of other files types though, so you may notice it more (see @pipe's comment on @alldayremix's answer).

Files can be corrupted in many ways, (non SSD) hard drives can be put too close to a magnet, CDs and DVDs can gradually rot, and electrical interference and bugs in firmware or software can corrupt files in transit. There are other reasons too, but I'd suggest asking on a more computer related stack if you want to go into more detail.

In terms of what can be done about it, the best answer I can come up with is calculating hashes and keeping multiple copies of the photos. To this end I wrote a free, open source, program to address this issue for myself: Archiverify.

It works by computing hashes, storing the hashes alongside the images, and comparing the files against the hashes when you run it. As long as you have more than one copy of the file it can automatically refresh the corrupted copy from the good one (assuming both don't get corrupted at once).

Alternatively, I'm sure that there are paid storage/backup services available that will do the hashing and checking for you periodically behind the scenes.

  • 1
    In theory, this kind of answer is fine, but it's probably worth you reading How not to be a spammer (ignore the slightly inflammatory title, I don't think individual SE communities have any control over that). However... this post doesn't actually answer the question as asked ("Why do photos get corrupted?") so it could potentially be deleted for that reason instead. – Philip Kendall Jul 9 '16 at 19:22
  • Thanks, that's true, it doesn't answer the headline question, I'll try and address that. It does answer the "most important" subsidiary question though. – Dan Jul 9 '16 at 19:47
  • @scottbb The answer has been significantly edited since I posted that comment. Please see the edit history. – Philip Kendall Jul 10 '16 at 6:45
  • @PhilipKendall my apologies, I didn't notice the history. – scottbb Jul 10 '16 at 12:36
0

There are multiple reasons as stated above. It can just be natural wear over time that corrupts bits of files or physical degrading. If you are continually finding photos and other files severely corrupted it can be a sign of a computer virus. If you don't want your files to be corrupted the first two ways then invest in an SSD as it won't degrade as quickly and it 10 times faster. (You might also want to do a quick virus scan too.)

0

Why does this happen and, most importantly, how to avoid this?

You should ask this question on Security.SE.

To avoid corruption you should make backups.

In information technology, a backup, or the process of backing up, refers to the copying and archiving of computer data so it may be used to restore the original after a data loss event. The verb form is to back up in two words, whereas the noun is backup.

Writable digital optical media such as CD-R and DVD-R could be corrupted even by sunlight. To avoid this you might use something like double backup strategy. However, nowadays, we have such beautiful online methods to backup photos, like Instagram or other online services. For example, behind Google Drive there are large amount of data centers. Every one of your photos will be stored on the Google File System (GoogleFS or GFS) on servers hosted in these data centers. This filesystem was designed by Google to protect users from losing data. Your photos will be saved in three exemplars in three different geographic locations, to protect against data loss because of local disasters from lightning, tsunami, earthquake, etc.

Try to use multiple online services simultaneously (such as megaupload, dropbox, wuala, 4shared, etc.).

Thereafter, for the greatest peace-of-mind that none of your photos were modified or corrupted, you would use checksums to verify your data integrity.

A checksum is a small-size datum from a block of digital data for the purpose of detecting errors which may have been introduced during its transmission or storage.

The best way is to use cryptography for such task. Use the latest stable hash functions, such as SHA-256 or SHA-512. Older algortihms, such as MD5, are now considered insecure. A modern attacker could make forgery of your data with the same md5sum.

When you are done, you will have a distributed backup system which will not be easily broken, even by the most sophisticated and dedicated attackers (such as nation state -backed attacks).

  • Two DVD copy with identical datas inside:

       1. Photo1.jpg
       2. Photo2.jpg
       3. Photo3.jpg
     ...
     999. Photo999.jpg
    1000. SHA256SUM
    

    Where this SHA256SUM shall contain something like:

    e5347dce99eb8cf694cf708d4a17d83abb3ec378241b5878c0abdab045859b24  Photo1.jpg
    b497a12b608def869a0429d7e6bbbd112bd413256201647a5aff6773de3b7bd9  Photo2.jpg
    b15b0d99bf8135286f444fc62bcf70278a89e60650252ab2bd3b6fffd40c4255  Photo3.jpg
    ...
    209732fbdb499f0cad6fd3311b45185667bbb40e501106997d3ac2c49cb30a7e  Photo999.jpg
    

    The lines 209732fbdb499f0cad6fd3... are unique hashes of your photos. When one bit of your photo becomes corrupt, this hash will be changed to another, so you could test hashes against this list to be calm that your photos are keeping integrity.

  • Secondly, all of this photos and this SHA256SUM-hashes file are uploaded to e.g. dropbox and somewhere else.

Now, you have 5 copies of your photos. 2 off-line, 2 on-line, 1 on your current hard drive.

The scheme might be evolved to something more robust of course. Your imagination is your trump card.

-1

I cannot rule out that the jpeg standards between your old camera and your latest image viewer have diverged somewhat, so the photos only look corrupted. Sometimes software complies only with a preliminary version of a standard. Try an older viewer (on an old computer).

  • 1
    I can rule that out. The latest update to JPEG standards was in 2012. And really the fundamental updates are from the 1990s. There's really no point where there's a state of "the photos only look corrupted". (I mean, this all sounds reasonable, but it doesn't match the actual history.) – mattdm Jan 31 at 0:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.