Incense

by Bart Arondson

submit your photo


Hall of Fame
View past winners from this year

Please participate in Meta
and help us grow.

Take the 2-minute tour ×
Photography Stack Exchange is a question and answer site for professional, enthusiast and amateur photographers. It's 100% free, no registration required.

like NTFS, HFS+ or ext4, for SD cards? After all, journaling decreases the chance of data loss, which would be important for photographers. I lost an SD card containing maybe a thousand photos when in Bali -- a place I haven't got a chance to visit before or since.

Are there any precautions I can take before a trip the next time? Format the cards in camera?

Am I correct in understanding that SDXC (exFAT) and Sony Memory Stick offer no more reliability than SD cards?

share|improve this question
1  
running any of those filesystems on a SD can kill the flash memory pretty darn quickly. –  Tim Seguine Jan 5 at 23:54
1  
@PhilipKendall Not my source, but this SE answer mentions it: serverfault.com/questions/41674/… ... anyway SSD hard drives actually need special logic to avoid trashing the flash when using normal filesystems. Cheap flash memory like that in SD cards is even less well suited to this kind of load. FAT is a very simple filesystem well suited for the sequential storage loads that cameras make and causes low flash memory wear.The primary issue is already in the provided answers though: added complexity for no gain. –  Tim Seguine Jan 6 at 0:15
2  
The error here was keeping 1000 photos on a card without backing up. If traveling to a remote location where you will be without access to a computer on which to backup you should be carrying a backup device. –  Jim Garrison Jan 6 at 1:26
1  
@KartickVaddadi What's your source for your numbers that a journaled file system will reduce life by 10% (5 years to 4.5 years)? Do you have some research you can point to? –  Philip Kendall Jan 6 at 8:32
1  
@KartickVaddadi: The mapping layer between the logical sector numbers and physical disk blocks creates failure modes which are unlike any normally associated with magnetic media; file systems which don't understand the mapping layer which is hidden from them cannot avoid any failure modes posed by that layer. –  supercat Jan 6 at 17:33

10 Answers 10

up vote 27 down vote accepted

Let's do a little cost benefit analysis:

  1. A journaled file system is more complicated - this means longer development time, more bugs, more battery power drain, higher production cost etc.

  2. the problem solved by a journaled filesystem - corrupted FS data but file data intact - is handled pretty well by 3rd party data recovery tools.

  3. journaled file system does not solve all problems, you really need good backups - and not only that systems with built-in backups exists (dual card slots) it's a feature that is used to make pros get more expensive cameras.

  4. there isn't a big memory card reliability crisis, those cards are pretty reliable and failure is relatively rare.

  5. and finally, there is no journaled file system that is supported out-of-the-box on both Windows and Mac.

So - if you were the product manager in charge would you approve a project that 1. solves an already solved (with 3rd party tools) problem in an incomplete way, 2. is not important enough to be a selling point and 3. will make a significant part of the market unable to use the camera (at least without installing additional software they won't need with the competing brands)?

share|improve this answer
2  
Actually, OSX can read NTFS volumes, and write to them with some terminal foo. –  Justin Dearing Jan 5 at 21:12
    
@JustinDearing: neat! You should cross-post that as a QA into askdifferent. –  o.v. Jan 6 at 2:02
    
the default filesystem configuration in OS X (meaning, the configuration that all Macs come preinstalled with) is HFS+ with journaling enabled. in fact, Time Machine requires journaling to be enabled. –  strugee Jan 9 at 3:02
    
@strugee - I didn't say OS X doesn't have a journaling file system - I said there is no single system that both OS X and Windows can use out of the box (Windows doesn't understand HFS+ at all and Macs (by default) can't write NTFS) –  Nir Jan 9 at 7:33
    
@Nir ah, nevermind. I misunderstood. –  strugee Jan 9 at 17:49

One obvious reason: because a journaling filesystem on a camera very likely would not have helped you (or anyone).

As a very high level overview, here's what a journaling filesystem does: Before each write to the metadata (or data, if data-journaled as well), first write what you're going to change to the journal. Only once you're sure that's on disk, go ahead and write the change. Basically, this means that if power is interrupted during the write, you can recover the filesystem by using the journal—you go ahead and perform any actions in the journal.

This is valuable on a desktop PC, where the power might go out, or the user may hit the reset button, or pull the plug, etc. Also valuable, but less so, on servers (power failure) and laptops (reset button).

A camera is battery-powered. It has an off switch, but this normally tells the firmware to shut it down—it isn't a physical power disconnect. There isn't usually a reset button, or if there is, its basically never used. So, you don't need journaling, the firmware can just finish the write. The only exception would be if you physically removed the battery. Maybe that'd happen with an external power pack, but other than that, a camera should never experience an unclean shutdown.

Also, almost no flash devices actually handle unexpected power failure well. Get them in the middle of a sector relocation (wear leveling), and all bets are off. So even if you had a journaling filesystem, you'd still not be safe from power failure.

A journaling filesystem does not protect you from:

  • Bugs in the flash controller on the SD, etc. card.
  • Bugs in the camera's SD host hardware
  • Bugs in the filesystem code on the camera
  • Bugs in the firmware's SD drivers
  • Loss of sectors on the media
  • Hardware malfunction (e.g., due to cosmic rays, static discharge, EM noise, water, ...)

In fact, a journaling filesystem is more complicated, so you are actually more likely to have filesystem bugs. It amplifies writes, so you're more likely to hit flash controller or SD host bugs. And you're going to wear out the flash slightly sooner.

share|improve this answer

Journaled File systems are bad for SD cards (or any NAND Flash device).

Write operations are expensive for NAND Flash devices and journaled file systems tend to write more than non-journaled file systems for the same activities.

So the SD card will work slower and will last less with a Journaled file system.

FLASH-based storage, at its core, uses a technology called NAND FLASH. NAND FLASH is readable and writable, but with several wrinkles.

  1. The fundamental read/write unit is a "page", not a sector. FLASH devices of the 2007-2008 generation have a 2K page size, migrating to a 4K page size in the 2009 generation and 16K page sizes have been observed in 2011 generations.

  2. You can't write a page anytime you want - before you write to it, you must first erase it. But you can't erase a single page at a time - you must erase an entire "erase block" of (typically) 64 consecutive pages (128Kbytes or 256Kbytes depending on the generation). And after you have erased the block, you can't write to the pages in an arbitrary order, you must write them sequentially starting at the first one.

  3. Blocks tend to wear out over time. After a certain number of erase cycles, a block will "go bad" permanently, so that it will no longer reliably hold data. Pages can also develop data errors as a result of write activity to other pages, and even as a result of reads!

http://wiki.laptop.org/go/How_to_Damage_a_FLASH_Storage_Device

Edit: It is worth to mention that Journaled File systems will not bring significant advantage over non-Journaled File systems.

share|improve this answer
1  
Flash devices use a block remapping layer (FTL), so you're not writing to the same physical block again and again. Android uses filesystems like ext4, so I don't see the validity of your argument that it's unsuitable for flash. –  Kartick Vaddadi Jan 6 at 6:05
    
Android devices usually have some RAM as well as flash, don't they? –  Michael Clark Jan 6 at 7:39
1  
Block remapping doesn't increase the total number of writes per block before a block goes bad, it just spreads write operations out over the entire card so that almost every block is worn at the same rate. Journaled systems use more write operations to do the same thing than non-journaled systems do, so the total number of writes before a card goes bad will occur sooner in its life cycle with a journaled system. –  Michael Clark Jan 6 at 7:42
1  
Android have some problems related to the storage (I/O lags) and they are implementing the TRIM command to improve the situation. SD cards were made to be cheap and small, not to be robust. There are alternatives more robust but they are more expensive. –  S182 Jan 6 at 17:05
1  
Android uses JSF because those devices are constantly writing information from several processes and they are prone to shout down unexpectedly (OS blocks, low battery, etc). It is not the best but they need it. In the other hand persistence operations in Cameras are much more simpler and a JFS would bring more problems than solutions. A Journaling File System is more resilient and is less prone to corruption, but not immune., in most cases you can repair a non journaled FS with a "scandisk". –  S182 Jan 6 at 17:25

1, God cannot save you, if you physically lost the card. What do you mean you lost a card in Bali?

2, Journaled FSs are built for occasions like sudden OS-failure or sudden power-failure. They keep the FS meta-data consistent, when those bad things happen. They are not helping if you want your deleted files come back.

3, Bad-block is the most vital problem of the NAND FLASH based storages. Bad-blocks come up when writings occur. Hence, when choosing FS for a NAND FLASH storage, writing frequency is the first thing you should consider. Obviously, like all the others said, Journaled FSs bring more things to write.

4, Journaled FSs take more power, of course. More complicated, sure. But these are not the dominant reasons that we don't adopt them for NAND FLASH, I think.

TADA~~ That's it.

share|improve this answer
    
1. See my comment on AJ's answer. 2. I did not manually delete the files. 3. Like I wrote on the other comments, how do you explain Android's use of a journaled FS on flash? It's not as bad as you're making it out to be. Not losing photos is more important than a marginal reduction in card lifetime. –  Kartick Vaddadi Jan 6 at 10:27
3  
Most Journaled FSs need one or more daemon processes/threads to manage their "journals". (For an instance, kjournald in Linux for EXT3) It will be hard to adopt them if the env is not a full fledged OS, where we have no concept of process/thread. –  Garf Jan 6 at 13:41
    
@KartickVaddadi Again, please give a pointer to some research showing that a journaled file system produces only a "marginal" reduction in card lifetime. This is the second time you've asserted it. –  Philip Kendall Jan 6 at 21:52
    
Fair question, but keep in mind Android uses it, as I said many times already. They wouldn't have used it if causes a drastic reduction in the life of the media, would they? Besides, I can just as well ask you to cite research showing that it causes a drastic reduction in life :) –  Kartick Vaddadi Jan 7 at 6:30
    
Maybe we should just ask @supercat, since he's an expert, and neither of us have cited data. –  Kartick Vaddadi Jan 7 at 6:39

Assuming the card was simply corrupted, and you haven't tossed it or overwritten it, I strongly suggest you try PhotoRec. (It got me out of a slightly less bad situation a few months ago. It even found a few images that had survived being deleted for a year or two.)

http://www.cgsecurity.org/wiki/PhotoRec

Regarding a journaling FS, I've had the same question many times. As others have said, current flash media actually is fragile compared to magnetic media, and journaling is hard on it. Since the usage pattern for cameras is generally take a bunch of photos, read them, then delete them all, there isn't much need for advanced FS features. Simple, tested implementations are probably more important than the marginal benefit of journaling. As an added benefit, the dumb allocation strategy of FAT makes it easier for tools like PhotoRec.

share|improve this answer
    
I think I did use PhotoRec in that case. Thanks for the link anyway. –  Kartick Vaddadi Jan 7 at 6:29

It comes down to resolving "is there a market?" and "what are the barriers to adoption?". Each of those presents a huge barrier to adoption even if it were worthwhile.

NTFS would incur costs for licencing even if a suitable library even exists for the camera's processors (which is not guaranteed) and support outside of Windows would be patchy. While HFS+ and ext4 have no native support in Windows, eliminating much of the potential customer base. So there's no market for those.

As you mention, exFAT is required by the SCXD standard so you'll see that as support for larger and faster cards appears but it's not as simple as that since more code is also more to go wrong, and with embedded systems like cameras, you really don't want to push out firmware updates so expect that while writes to an exFAT card might be readable and in the right format, it may not actually use any of the exFAT features that might offer any protection. So there are significant barriers to adoption too.

The failure mode of most cards is as likely to be the controller as a memory cell it is a lot of work (cost to manufacture) for little benefit.

Sony MS (MemoryStick) is still SLC or MLC flash memory, it's just the controller and physical connection which differs between the systems. Your best protection in the situation you've experienced is to take a small portable backup device with you, they are pocket sized and relatively inexpensive (and probably incompatible with Journaled filesystems too.)

share|improve this answer
    
NFS is not an on-disk file system, it is a network protocol (roughly on par with its considerably more familiar cousin FTP in terms of the problem it solves). Did you mean HFS+ (the file system used natively by Mac OS)? –  Michael Kjörling Jan 5 at 19:48
    
I did indeed mean HFS+, will edit :) –  James Snell Jan 5 at 22:36

Different file systems require different amount of RAM in a system that is using them. A system that needs to write a file to a FAT file system could in theory get by with a single 512-byte buffer, though performance would be pretty dreadful. Expanding to two or three 512-byte buffers would improve things enormously. Going beyond that would improve things somewhat more, and getting optimal performance from a larger card would require more memory than getting optimal performance from a smaller one, but a camera which only included enough buffers to achieve optimal efficiency with smaller cards would still be able to work with larger ones, even if less efficiently.

A trickier issue centers around the fact that memory-card standards specify that each card behave as a numbered collection of 512-byte sector that can be read and written independently in arbitrary sequence, but that's not how the data is stored on the chips within the cards. The memory chips used in a typical memory card are divided into 528-byte pages; those in turn are grouped into blocks of 256 or more. Once a page is written, it cannot be rewritten without erasing it and all the other pages in its block. In theory, it would be possible for an SD card to honor a request to write a 512-byte sector by copying to RAM all the data in its block, erasing the block, and writing the whole block back but with new data in one sector. In practice, performance would be dreadful. Instead, writing a sector will cause the SD card to pick a blank page, write the data there along with its sector number and various ancillary information (the reason pages are 528 bytes rather than 512), and somehow keep track of that being the proper location for the data. When blank pages get to be in short supply, the controller will identify a block whose pages have been mostly superseded by pages written more recently, copy all still-current pages from that block to blank blocks, and then erase the entire now-redundant block. All this logic is handled entirely by the card itself, without any intervention by the camera.

All this logic means that in addition to the FAT32 or other file system seen by the camera, the SD card will need to have its own block allocation and management system. Any problems that occur in that system are likely to cause data loss, regardless of what sort of system sits on top of it. In theory, many memory cards are designed to ensure that even if power is unexpectedly removed during some operation the card will be able to either roll back the state of the card to what it was before the operation began, or else run it to completion (if all the necessary data had been written, and the card was simply cleaning out redundant data). Unfortunately, cards differ in how well they implement such logic. If unexpected power loss clobbers the storage management tables of a card, software which understands the inner workings of such tables might be able to recover data which is invisible to any software that simply uses the sector-based read-write interface.

Personally, I think it would have been better for the SD Consortium to specify a file system independent of FAT32, or at minimum specify that even if a card had to be readable as a FAT32 volume, it should be written using a file-based communications protocol. A card which knows which groups of sectors are members of each file could optimize its defragmentation routines around that, and could also do a better job of protecting against data loss than could one which had to present the disk as a bunch of independent 512-byte sectors, but for better or for worse that's not how things are specified.

share|improve this answer
    
I think there's already a standard solution: a block-remapping layer, with a standard filesystem (NTFS, HFS+, ext4) on top. And it's used in mobile as well, on Android. Camera OSs may be more primitive, but that needs to be fixed. –  Kartick Vaddadi Jan 6 at 6:09
    
@KartickVaddadi: The block-remapping layer is standard; my point was that if the memory card which implemented the block-remapping layer was at least somewhat cognizant of file-system layout, it could optimize the remapping layout more effectively than is possible without such knowledge. –  supercat Jan 6 at 7:40
    
Sure, but I'd prefer to take something tried and tested rather than come up with new interfaces between the block device layer and the filesystem. We're not talking CS research here :) I want to take something that works on my computer, and my phone, and put it on my camera. –  Kartick Vaddadi Jan 6 at 10:25
    
@KartickVaddadi: I've designed some wear-leveling flash file systems for embedded devices with various constraints. If the wear-leveling system is told "I want to write a file; here –  supercat Jan 6 at 17:41
    
...here's the data; that's it. I want to write another file; here's the data; that's it", it can behave somewhat more intelligently than if it is simply given a bunch of individual sectors to work with and has no idea what they represent. Among other things, all the data blocks associated with each file could be marked with tags saying e.g. "block 347 of file id 193,291,374, update 273,837,199." –  supercat Jan 6 at 17:49

The file system itself does not need to be complex because images are simply written to the card, there isn't hardly any editing done to a file after initial creation and there aren't simultaneous file I/O concerns that need to be worried about on the camera.

The data integrity issues are actually solved at a hardware level because ALL flash memory is inherently unstable. The controller within the SD card does a lot of its own checks and storage tricks to ensure that the data is valid. A journaling file system would do nothing to help with this as it deals with the integrity of the data storage rather than the integrity of the file operations.

A camera uses such simple (and high speed) file operations, that a complex file system would incur additional cost and complexity, causing slower I/O and potentially introducing further bugs that could result in data loss due to more complex file handling, while not gaining anything of use to a camera.

share|improve this answer
    
In my case, the filesystem got corrupt, perhaps due to a bug in the camera's filesystem code that triggers in rare cases. Journaling will ensure that if that happens, the filesystem is more likely to remain undamaged, meaning you're not going to lose thousands of photos. –  Kartick Vaddadi Jan 6 at 6:06
3  
@KartickVaddadi - are you sure it was the file system that became corrupt and not the SD card itself? A file corruption issue with a FAT table should never result in failure of the entire card, it should be easily recoverable for most of the photos unless the card itself failed. How are you sure of what exactly failed. –  AJ Henderson Jan 6 at 6:14
    
I was able to recover most of the photos from my laptop using one of those recovery tools that ignores the filesystem, reads all the blocks on the device and tries to figure out what the files are. I could not read it in camera, which meant I couldn't take photos for the rest of the day, and I never again visited the places I had visited on that day, so it was a missed opportunity. –  Kartick Vaddadi Jan 6 at 6:19
2  
@KartickVaddadi - yes, but that could have been a file system or SD card failure. If the TOC gets wiped out, you are still going to have to do recovery on the file system, regardless of if you use FAT or NTFS. I'm still not sure that a journaled file system would help. A journaled file system's main strength is just that it can recover from a file being partially written because it knows the file or directory record is bad. What you are dealing with there most likely was corruption to the allocation table which could be either disk or file system failure. –  AJ Henderson Jan 6 at 14:03
2  
@KartickVaddadi: I would think that as a matter of principle one should always try to have a spare card on hand, and if the card one is using shows any sign of trouble switch immediately to the spare, so as to avoid destroying any data that would have been recoverable from the problematic card. –  supercat Jan 6 at 17:59

As far as I know, all digital cameras produced to be sold on the retail market incorporate the Design rule for Camera File System (DCF). Part of the DCF standard is that the FAT file system must be used by compliant devices. This standard was adopted as the de facto standard for storing digital image and sound files in memory devices by the digital camera industry to insure interoperability from one brand to the next.

See http://photo.stackexchange.com/a/46387/15871 for more information about DCF.

share|improve this answer
    
Would the standard preclude a camera vendor from using NTFS. HFS+, or other file system if a card was inserted that was formatted with one of those systems, or would the camera be required to simply say "card unusable"? –  supercat Jan 5 at 22:19
    
At one point the spec didn't include FAT32 IIRC. At present (DCF v2, published 2010) the spec is limited to all the FAT variants + exFAT. So there are precedents for DCF to be extended in the future to include other filesystems if the members wanted that. –  James Snell Jan 5 at 22:50
    
@supercat It would be outside the standard as it is now written. Standards are always subject to revision. But the question seems to be asking why don't any current cameras support journaled file systems. –  Michael Clark Jan 6 at 7:32
    
@JamesSnell Regular FAT16 also tops out at 2 GiB per partition, so the move to allow something slightly more modern solved a very real problem. Widespread support for FAT32 in non-Microsoft systems seems to have been implemented in the years around 2000, and FAT32 tops out at a significantly more useful even today 2 TiB per partition when using a 512 bytes logical sector size. –  Michael Kjörling Jan 8 at 15:50
    
@MichaelKjörling - I'm well aware of the limitation of FAT16 thanks and I'm not saying FAT32 was added in 2010 (that's when exFAT was added). The point is that CIPA found it useful to extend the specification and could do so with future filesystems if they wished. Clearly they saw a need / desire for something beyond FAT32. –  James Snell Jan 8 at 18:00

Journaled file-systems only ensure the integrity of the file-system. If a card truly fails, it fails with the whole file-system. Now if you have some bad memory cells, you would only use whichever photo occupied that space and a journaled file-system would not help either. In other words, this is the wrong solution to the incident you describe.

The real solution is redundancy which is why you will find high-end offerings from Nikon, Pentax and Canon which offer dual memory-card slots and the ability to write images to both cards at once. This gives you an instant backup. If those cameras are not convenient to you, the you have to find some other way to make frequent backups. Some people do it daily onto a laptop, portable drive, optical-disk.

While I have not tried this yet and am not sure how practical it is, you can also use a WiFi device or card (SD/SDHC only AFAIK) which automatically sends your images as they are captured to another networked device, maybe a tablet or something with good storage.

While SDXC comes format as exFAT by default, you can format it yourself at FAT32 too. Most cameras will accept it both ways. The difference in reliability is probably zero though.

share|improve this answer
    
Yes, but that's not the only failure mode, is it? In my case, a stress test of writing multiple times to the entire card did not detect an error, so I don't think it's a question of bad memory cells; just some corruption. Regarding bad memory cells, a journaled filesystem will ensure that only photos stored there will be lost and not the whole filesystem, with thousands of photos, right? If a journaled filesystem is the wrong solution to the problem, I'm afraid I don't see what the right solution is. When I travel, I don't always have my laptop, tablet or portable disk to backup photos to. –  Kartick Vaddadi Jan 5 at 14:40
    
Journaled file-systems make sure the whole file-system is consistent but they really do not do anything for corruption. You would need some redundancy for that. –  Itai Jan 5 at 15:34
    
And making sure the whole filesystem is consistent means that only files that have corrupt blocks will be lost, not the whole thing. –  Kartick Vaddadi Jan 5 at 15:44
1  
@KartickVaddadi I think it's best to assume when you buy any type of flash memory it will fail at some point in time. The best you can do if you are unwilling to invest to reduce the risk when you are out in the field is to make sure you purchase reliable cards from reputable manufacturers. –  Peng Tuck Kwok Jan 6 at 5:56
4  
@KartickVaddadi You are trying to swallow a camel to avoid straining to eat a gnat. If you've spent 'thousands of dollars' on your gear, what is another $20 for an extra memory card that you would have to purchase anyway to take advantage of a second card slot? –  Michael Clark Jan 6 at 7:46

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.