What processing is done on RAW files in the camera?

Question

Recently I came across a debate about whether it's fair to call RAW 'unprocessed' or 'unaltered' sensor data of a DSLR. As far as I know, the analog sensor data is processed into a digital RAW file using the ISO setting, a process during which the Bayer-sensor data is demosaiced as well. However, those are just the necessary steps for saving a digital file while maintaining maximum quality.

Are those assumptions correct? If so, are there any additional steps that need to be performed between the sensor receiving light and the saving of the RAW file? And can those be reasonably be described as an 'alteration' of the sensor data, or is it fair to say that a RAW file holds the unaltered sensor data?

Answer 1

It varies highly from camera to camera. Some designs do a minimum of processing on the image sensor itself, others do a little more. Those that do more do so mainly in the area of noise reduction either before or after sending the analog data to be converted to digital data. One method is the relative amplification of the signal from pixels masked for red, green, or blue (which is done for reasons related to the different noise characteristics of pixels filtered for the different colors of the Bayer Mask). Another method used after analog-to-digital conversion is to average pixels with a much higher luminance value than their neighbors to a value much closer to the surrounding pixels.

Even different camera models that share the same sensor design may apply different processing to the output from the sensor, either before or after it is converted to digital information, prior to it being saved as a raw data file. Information about the conditions under which the data was obtained (camera model, sensor characteristics, ISO, WB, etc.) will be appended to the file so the application eventually converting the data to a viewable image will (hopefully) know how to convert it. Demosaicing is not normally done to sensor data before it is saved as a raw file. That is done when the raw data is converted to something else, such as an image displayed on a monitor by a raw conversion application, or converted as output as a jpeg or tiff.

Answer 2

Back in the 1980s, Craig Anderton wrote a book called Home Recording for Musicians in which he observed that a recording studio isn't a capturer of reality, it's a processor of reality. You should think of your camera the same way. The light making up an image enters the lens, which changes it, reaches the sensor, which changes it, is passed through various bits of analog circuitry, which change it, and finally becomes digital data by way of an analog-to-digital converter (ADC) which, as you might have guessed, changes it.

Any change is an alteration, and it might be reasonable to call alteration done with a purpose processing. There are undoubtedly some things done on purpose in the analog stages of your camera. ISO adjustment is the elephant in that room because analog amplification is necessary to get the most out of the ADC.

You're not getting information straight from the sensor because it's an analog signal, and analog signals are notoriously difficult to store and reproduce well. Whether you call what happens between there and when you have digital information alteration or processing is debating the trivial. If you're going to split those hairs, the answer is that it's probably some of both.

The ideal place for capturing data for raw files is the output of the ADC, because anything that happens to it beyond that is algorithmic and can be done off-camera. If you find a better way to do something (demosaicing, application of curves, etc.), it's much more economical to update a raw converter that runs on a computer than it is to create, test and distribute new camera firmware. Providing raw data is also something of an acknowledgment that the decisions made by the manufacturer aren't going to please everyone. Doing as little as possible to the data post-ADC gives the photographer the same flexibility he'd have had in a darkroom.

Whether or not the contents of a raw file is straight from the ADC or somehow adulterated is known only to the manufacturers. One implies that it doesn't, and my own inclination is to believe that to be true for reasons that go beyond the scope of answering this question.

As Michael Clark points out in his answer, everything in a raw file that isn't image data is bonus information that can be used in making decisions about how to process it.