File recovery is a subset of data recovery. Two methods for file recovery are at our disposal; file system based recovery and raw or signature based recovery.

File recovery is only one part of data recovery. Data recovery consists of:

  • Diagnostics
  • Repair of media (if required)
  • Cloning/imaging media
  • File Recovery: Recovery of individual files

Repair of Media

Repair of media falls outside the scope of this post. In general you may assume media damage if:

  • Media is not detected or not detected with proper size
  • If reading media results in frequent lock ups of operating system or file recovery software.

To address this type of situation you need the proper hardware and experience! It is easy to turn a minor issue into a disaster.

Cloning/imaging media

We'll discuss file recovery in more detail but a quick word on cloning first.

I strongly urge to create a sector by sector duplicate of the media where data was lost from. Seen the nature of this 'stack' we'll be mostly addressing file recovery from memory cards. These memory cards are easily cloned to a file, any decent file recovery tool offers an option to create a disk image. This disk image can then be treated as a drive and files can be recovered from it.

Since these are raw disk images, file size will correspond with size of the device you are imaging.

File Recovery

Background: File systems (such as FAT or NTFS) organize our files. But what is a file?

From a file system perspective a file is:

  • Meta data (filename, creation date, parent (folder), pointer to file data)
  • And the actual file data

Each file system has it's own 'format' to store meta data. FAT file systems use directory entries to store meta data such as filename etc.. The directory entry points to first cluster of the file, subsequent clusters must be decoded from the file allocation table. NTFS stores everything in a structure called the master file table (MFT).

Although the smallest addressable unit on a drive is one sector, file systems work with clusters which consist of one but often multiple sectors. This reduces overhead. But eventually cluster addressed need to be converted to sector addresses either by the file system driver or in case something goes wrong, by file recovery software. It is therefor vital file recovery software is able to determine cluster size and offset.

File system based recovery

File system based recovery is always preferred as it gives us filenames and original folder structure.

For file system based recovery we need meta data + actual data.

In many scenarios file system meta data can be detected. For example, file deletion in many file systems merely 'flags' the file as deleted. When a file system is (quick) formatted a large portion of meta data actually survives. In case we deal with a RAW file system we may be dealing with relatively minor damage to meta data.

In many cases file system based recovery is possible. Good generic file recovery software will always try to reconstruct a virtual file system before falling back to RAW or signature based recovery.

There are some limitations too, specially if we consider memory cards and the file system that is frequently used on those: some flavor of FAT (FAT32, exFAT).

When we delete a file from a FAT based file system all FAT entries keeping tack of clusters allocated to the file are 'reset'. In case of formatting the entire FAT is reset. This is a problem when the file we need to recover is fragmented.

RAW or signature based recovery

In case we deal with severe file system damage we ignore this meta data and scan for actual file data. You may or may not know that for example JPEG files start with a 0xFF,D8,FF byte sequence and end with 0xFF,D9.

So what a raw scanner does it detect for example 0xFF,D8,FF byte sequences and assume it has found the start of a JPEG file. Now all it has to do is save data until it reaches 0xFF,D9 byte sequence, assign a generic filename and if all is well it has recovered a JPEG. This is a RAW scanner in it's simplest form and in reality it would not work. RAW scanners have to be smarter than this, they need to have a minimal set of knowledge about the file types they're designed to recover.

Example: JPEGs frequently embed one (or some times even more) thumbnails or previews which are in themselves complete JPEGs. So the first end of file byte sequence 0xFF,D9 would not be the end of the file, it would be the end of the first embedded JPEG. This is just one of the problems a raw scanner needs to deal with. RAW scanners are often regarded as kind of dumb tools, a well deigned RAW scanner is not!

Although a RAW scanner does ignore the file system, it is actually nice to know some properties about the file system such as the cluster size as it greatly reduces the places it has to check for signatures. So preferably the tool makes some efforts to determine the cluster size.

While RAW scanners are great solutions if file system based recovery tools fail, they do have many drawbacks too:

  • They do not recover file name and folder structure
  • Signatures they scan for are not unique so false positives are to be expected
  • Can only detect files they have signatures (byte sequences) for
  • Can not deal with file fragmentation
  • Can not detect duplicate file easily

Picking a good file recovery tool

Although I do have personal preferences I will not name names. I will try to come up with a set of properties that define a good file recovery tool and explain some of them.

  1. Tool has a disk image feature
  2. Has ability to scan a raw, dd-type, sector-by-sector disk image
  3. Excels at building a virtual directory tree
  4. Tool has an image preview option

1 & 2 I explained before. A data recovery professional never works on the 'patient drive' and you not doing either is solid advice.

Point 3 is important. I frequently see people complain about a file recovery product taking hours to scan. 9 out of 10 times this is due to the software being unable to come up with a virtual file system solution and has fallen back to a RAW scan. For a RAW scan you need to scan the entire drive and the end result is a tree with files sorted by type with all drawbacks of a RAW scan (no filenames etc.).

A good file recovery tool is often able to solve a file system solution and build a folder tree within minutes.

And this is where point 4 comes into play:

Building a folder tree with seemingly correct file names is one thing. These are structures that are easily recognized. But without some vital parameters solved, a seemingly intact tree means nothing. We need to solve the file system offset and cluster size.

To determine if a tool correctly solved those we can use the image preview: If you are able to preview say 10 large image files, it is safe to assume the file recovery tool correctly solved the file system offset and cluster size.