RAW files store 3 colors per pixel, or only one?

Question

Ken Rockwell says that camera makers consider the individual R/G/B sensors when they talk about megapixels. So the image below would be a 6x6 pixel camera, not 3x3 as you would imagine.

If that's true, a RAW file would contain only one color information per pixel (be it R, G or B) as a 10, 12 or 14 bit number.

My confusion comes as I read in some places stuff like:

• RAW files store an avarage of the two green sensors per pixel.
• RAW files use 12 bits per pixel, but there are 3 colors so that's actually 36 bits per pixel.

Which would obviously be false, if Ken's claim is correct.

So what's the truth?

Answer 1

Raw files don't really store any colors per pixel. They only store a single brightness value per pixel.

It is true that with a Bayer mask over each pixel the light is filtered with either a Red, Green, or Blue filter¹ over each pixel well. But there's no hard cutoff where only green light gets through to a green filtered pixel or only red light gets through to a red filtered pixel. There's a lot of overlap. A lot of red light and some blue light gets through the green filter. A lot of green light and even a bit of blue light makes it through the red filter, and some red and green light is recorded by the pixels that are filtered with blue.

Since a raw file is a set of single luminance values for each pixel on the sensor there is no actual per-pixel color information to a raw file. Color is derived by comparing adjoining pixels that are filtered for one of three colors with a Bayer mask. But just like putting a red filter in front of the lens when shooting black and white film didn't result in a monochromatic red photo (or a B&W photo where only red objects have any brightness at all), the Bayer mask in front of monochromatic pixels doesn't create color either. What it does is change the tonal value (how bright or how dark the luminance value of a particular color is recorded) of various colors by differing amounts. When the tonal values (gray intensities) of adjoining pixels filtered with the three different colors used in the Bayer mask are compared then colors may be interpolated from that information. This is the process we refer to as demosaicing.

A lot of math is done to assign an R, G, and B value for each pixel. There are a lot of different models for doing this interpolation. How much bias is given to red, green, and blue in the demosaicing process is what sets white/color balance. The gamma correction and any additional shaping of the light response curves is what sets contrast. But in the end an R, G, and B value is assigned to every pixel. In your 6x6 pixel example in the question the result of demosaicing would be a 36 pixel image with 36 pixels that each have a Red, a Green, and a Blue value.

A little bit of resolution is lost in translation. It turns out that in terms of the number of alternating black and white lines per inch or mm that can be resolved by a sensor with an RGGB Bayer mask and well-done demosaicing the absolute resolution limit of a Bayer sensor is about 1/√2 compared to a monochromatic sensor that has no Bayer mask and thus needs no demosaicing (but can only see in Black & White).

Even when your camera is set to save raw files, the image you see on the back of the LCD screen of your camera just after you take the picture is not the unprocessed raw data. It is a preview image generated by the camera by applying the in camera settings to the raw data that results in the jpeg preview image you view on the LCD. This preview image is appended to the raw file along with the data from the sensor and the EXIF information that contains the in-camera settings at the time the photo was shot.

The in camera development settings for things like white balance, contrast, shadow, highlights, etc. do not affect the actual data from the sensor that is recorded in a raw file. Rather, all of those settings are listed in another part of the raw file.

When you open a "raw" file on your computer you see one of two different things:

• The preview jpeg image created by the camera at the time you took the photo. The camera used the settings in effect when you took the picture and appended it to the raw data in the .cr2 file. If you're looking at the image on the back of the camera, it is the jpeg preview you are seeing.

• A conversion of the raw data by the application you used to open the "raw" file. When you open a 12-bit or 14-bit 'raw' file in your photo application on the computer, what you see on the screen is an 8-bit rendering of the demosaiced raw file that is a lot like a jpeg, not the actual monochromatic Bayer-filtered 14-bit file. As you change the settings and sliders the 'raw' data is remapped and rendered again in 8 bits per color channel.

Which you see will depend on the settings you have selected for the application with which you open the raw file.

If you are saving your pictures in raw format when you take them, when you do post processing you'll have the exact same information to work with no matter what development settings were selected in camera at the time you shoot. Some applications may initially open the file using either the jpeg preview or by applying the in-camera settings active at the time the image was shot to the raw data but you are free to change those settings, without any destructive data loss, to whatever else you want in post.

Canon's Digital Photo Professional will open a .cr2 raw file in the same Picture Style as was selected in camera when shot. All you have to do to change it is use the drop-down menu and select another Picture Style. You can even create a "recipe" for one image and then batch apply it to all of the images before beginning to work with them. Other manufacturer's raw processing software is similar and there's usually an option to have the application open an image with the in camera development settings applied.

With third party raw processing applications such as Adobe's Lightroom or Camera Raw, Apple's Aperture or Photos, PhaseOne's Capture One Pro, DxO Lab's OpticsPro, etc. getting images to display according to the in camera settings can be a bit trickier. Adobe products, for instance, ignore most all of the maker notes section of a raw file's EXIF data where many manufacturers include at least some of the information about in camera settings.

_{¹ The actual colors of the Bayer mask in front of the sensors of most color digital cameras are: Blue - a slightly violet version of blue centered at 450 nanometers, Green - a slightly yellowish version of green centered on about 540 nanometers, and Red - a slightly orange version of yellow. What we call "red" is the color we perceive for light at about 640 nanometers in wavelength. The "red" filters on most Bayer arrays allow the most light through at somewhere around 590-600 nanometers. The overlap between the "green" and "red" cones in the human retina are even closer than that, with "red" centered at about 565 nanometers, which is what we perceive as yellow-green.}

Answer 2

Ken is right in the claim you quote — sort of. It is correct that digital cameras today (with the exception of those with Sigma's Foveon sensors) work by using a Bayer matrix, and sensor resolution is quoted as the size of the matrix. Your example image represents a "36 pixel" sensor. However, it's important to recognize that cameras turn this into a full color image of the full specified size in actual pixels, and that this is not as bad as Ken makes it out to be.

Several things he says in that article are downright wrong, starting with:

As of 2006 these clever algorithms allow starting with one-third the data and making it look about as good as having one-half the number of pixels claimed.

This was nonsense in 2006 and is nonsense today. The process works on some simple assumptions. More of which are laid out here, but the basic one is that you can predict what the "missing" information should be looking at the different-colored neighbor pixels. This turns out to be a good assumption most of the time, and very wrong other times. In cases where there is not a lot of very detailed transition between colors, the result is just as good as if each sensel recorded full color. In cases where the assumption is wrong, it's much worse. In the real world, the former is actually very common and works much better than "one-half" — but the important thing is that it is context-dependent.

RAW offers no advantages here, except for one potential gamble. Bayer interpolation takes place in the software opening the raw data. Future advances in Bayer interpolation algorithms could be incorporated in future raw software, if and only if your camera maker continues to support yesterday's cameras in tomorrow's software. Just as likely, your camera maker may no longer support your old camera in tomorrow's raw software!

He's right in that shooting RAW does not change the fundamentals, but the idea that old files will stop working is basically nonsense. Since old cameras use the same basic principle, and fundamentally similar file formats, it doesn't cost much to bring support for old models along indefinitely, and vendors have plenty of incentive to do so — and even if that would happen, there are great open source decoders.

And of course, keeping RAW files offers other advantages not related to demosaicing.

But it's also silly to say that the possibility of future improvements is the only advantage. As I said, there are different assumptions which can be made about the content of your image, and different algorithms (or tweaks to those algorithms) will suit different real-world situations better, so if you find yourself in a situation where you are getting moire or other artifacts, you may be able to deal with that. (Although, I should add that this is at the very fussy level — there's very rarely a situation where peeking this closely is worthwhile.)

There's also a factor which Ken may be excused for because the article is a decade old. In 2006, most cameras were in the 5-8 megapixel range, with high-end DSLR models stretching to 12. Now, typical low/mid-range DSLRs and mirrorless cameras offer 16 and 24 megapixels, and it goes up from there. At this point, quibbling about color detail at the pixel-peeping level is really academic, because in the real world it's very rare that lighting, lenses, stability, and everything else line up so well that this is the limiting factor.

In general, a lot of Ken Rockwell's site is like this. (See this answer for more.) This is unfortunate, as he actually has a lot of interesting things to say and some good advice, but there's a lot of nonsense as well, and rather than admitting that or improving it, he tends to double down, and then claims that the whole site is satire.

Oh, and a bonus fun-fact: camera rear LCD screens and EVFs also use three colored sub-pixels to represent one digital pixel, and these screens are usually marketed with the count of the sub-pixels — effectively 3× what you might expect from the way computer screen resolution is given.

Answer 3

It is all true, but the interpretation can be stretched.

That specific raw color pattern is called a Bayer pattern.

Yes, raw is one color per pixel, and that one pixel is (typically) 12 bits. So there are three colors of raw pixels, some are blue, some are red, and 2x those counts are green.

Then later, the raw processing software (to make RGB JPG, it could be immediately in the camera, or could be external much later) converts the raw data into a RGB image so we can use it. This is interpolation, neighboring pixels of the other two colors are combined into each of these RGB pixels, but all do become RGB pixels. At that point, it is 36 bit RGB pixels, however the spatial resolution is slightly compromised, with the various pixel data being shared with neighbors. We may end up with (for example) 6000 RGB pixels of sensor width, but it came from 2000 blue and 2000 red sensors, etc. (and the data is also shared vertically, it comes from more than three pixels). This is called demosaicing ... which can be found online.

Answer 4

Cameras with CFA that are not Sigma Foveon literally have an array of single sensors each having a single capacitor. Naturally, raw file would be something directly read out of this array. It makes no sense to rearrange this information in the raw file because it's purpose is to be fed to an advanced program on a large computer which can make all the transformation on itself.

RAW files store an avarage of the two green sensors per pixel.

That's a double lie because:

• raw files are not organized with regards to colour
• debayerization does not combine pixels, instead it interpolates the absent colours for each pixel.

RAW files use 12 bits per pixel, but there are 3 colors so that's actually 36 bits per pixel.

That's a triple lie:

• debayerization does not combine pixels
• even if it would, it would mean 48 bits per RGGB block
• a lot of expensive cameras have 14 bit sensor readout.