How much light and resolution is lost to color filter arrays?

Question

Color digital cameras are typically implemented by putting a color filter array (CFA) like a Bayer filter, plus an infrared cut filter, in front of a sensor that is sensitive to light frequencies spanning the full visible spectrum plus some range to either side of it.

The filters have two degradative effects:

1. They exclude light from reaching the sensor. (E.g., a "green" sensor pixel may only receive photons that are within the range 500-570nm. Most others are rejected.)

2. Resolution is lost to "mosaic" effects. (E.g., a green image component is only seen by half of the pixels in a Bayer filter.)

How are these losses quantified, and what is their typical magnitude in practice?

Answer 1

The idea that any particular wavelength is only allowed to pass through one particular color of the three colors used in a Bayer masked filter has been perpetuated to death. Fortunately, it is false.

Here's a typical enough spectral response curve of a specific camera sensor.

The visible (to humans) spectrum ranges from 390 to 700 nanometers. Notice that the "green" pixels respond, to one degree or another, to the entire range of visible light. That response is greatest between about 500 and 570 nanometers, but it is by no means zero at other wavelengths. The same is true of the "red" and "blue" filters. Each allows some light from the entire visible spectrum to pass. What differentiates them is in just how much of the light of a particular wavelength is allowed to pass through and how much is reflected or absorbed.

There are Bayer masked CMOS sensors in current DSLRs that have quantum efficiencies approaching 60%. That should be enough to eliminate the fallacy that only 1/3 of visible light that falls on a Bayer masked sensor is allowed to pass the filter and be measured by the pixel wells. If that were indeed fact then the highest quantum efficiency of a Bayer masked filter would be limited to 33%.

Note that the human response to visible light is similar. The cones in our retinas also overlap significantly in their spectral response.

What we perceive as colors are the differences in the way our brains process the varying response of of our blue, green, and red cones to different wavelengths and combinations of wavelengths.

In theory the infrared cut filter doesn't reduce any light visible to human vision because none of the light it prevents from reaching the sensor is visible to human eyes. Infrared, by definition, begins just outside the range of visible light at 700 nanometers and extends wavelengths of 1,000,000 nanometers (1 mm). Digital sensors are typically sensitive to IR light from between 700 and 1,000 nanometers. In practice sometimes the near-infrared wavelengths just under 700 nanometers are attenuated slightly by the IR-cut filters.

So just how bad are the "degradative effects" identified in the question?

They exclude light from reaching the sensor. (E.g., a "green" sensor pixel may only receive photons that are within the range 500-570nm. Most others are rejected.)

As covered above, the best current CMOS sensors in DSLRs and other cameras have quantum efficiencies in the visible spectrum ranging from between 50-60%. In one sense you could say they lose roughly half the light that falls on them, or one photographic stop. But that's not a whole lot different than the human retina so the argument could be made that they don't lose much of anything compared to what we see with our eyes.

Resolution is lost to "mosaic" effects. (E.g., a green image component is only seen by half of the pixels in a Bayer filter.)

Again, all three colors in a typical Bayer array are sensitive to at least some of the "green" wavelengths between 500-570 nanometers. This overlap is leveraged when the monochromatic luminance values from each pixel well are demosaiced to create R, G, and B values for each pixel on the sensor. It turns out that in terms of the ability to resolve alternating black and white lines a Bayer masked sensor has absolute resolution that is about 1/√2 of a non-masked monochromatic sensor of the same pixel pitch.

Also, don't assume that that "white light" has equal amounts of energy across the entire visible spectrum. It does not. Anytime we are talking about camera sensor efficiency, this is often the elephant in the room that all too often ignored. If the light we are interested in capturing with our cameras is stronger in the green range, then a sensor that is more responsive to "green" wavelengths will be more efficient for our purposes than one that has a flatter response to energy distributed evenly across the entire visible spectrum!

Sunlight filtered by Earth's atmosphere, and the artificial light sources we create to mimic that light, has much more intensity in the mid-range (green) wavelengths than in the shorter and longer wavelengths on either side of the visible spectrum. Our retinas evolved to be most sensitive to the most energetic portion of "white light" and our cameras have sensors that mimic that.

Please also see this answer to Why is the sky never green? It can be blue or orange, and green is in between! which includes this graphic:

Answer 2

There's no "lost" resolution. A manufacturer may engage in "specmanship" by advertising X-megapixels, but the resolution is defined by the pixel size, fill factor (what percent of the pixel surface is light-sensitive), and the number of pixels per color group. Further, there are well-developed algorithms for 'retrieving' resolution within an RGBG group based on a posteriori analysis of the RGBG group's, and its neighboring pixels', signals.

As to filter thruput: the spectral transmission curves for common camera RGB (and RGBY for some esoteric designs) are readily available on the web. Use them with care, since the signal loss for a given photopic (retina + brain) color, typically caused by several different incoming photon wavelengths, can vary considerably from one color to another. However, camera manufacturers are well aware of this, so both the RAW lookup tables and the internal JPG converters perform a color-rebalancing algorithm to compensate.

Answer 3

Summarizing from comments here, an upper bound on light loss due to an RGB filter is indeed a factor of 3, or 1.6 stops. In reality the response of each color filter element has some spectral overlap, so it's not quite that severe. Matt Grum estimates a factor of 2.5, or 1.3 stops.

Answer 4

Loss of resolution:

Physically, a factor of 4 is lost on the resolution because 4 pixels are needed to fully represent the color of a certain point. In practice the loss of resolution is negligible. This is because color filter arrays have twice as many green fields as blue or red and the eyes more sensitive to green, especially when it comes to resolution and because camera manufacturers use very sophisticated de-mosaicing algorithms that can recreate much of the lost resolution. unfortunately this does sometimes cause undesired artifacts, as is described in this article:
https://www.dpreview.com/articles/3560214217/resolution-aliasing-and-light-loss-why-we-love-bryce-bayers-baby-anyway

light loss:

At least about 50% of the light is lost due to a color filter array.

To show this, it is interesting to look at the Quantum Efficiency of the Sony IMX249 image sensor.

This sensor is particularly interesting because it is sold both as a color and as a monochrome sensor for which the manufacturer has plotted the quantum efficiency in the same graph. This makes it possible to compare the efficiencies of both and calculate the difference.

To calculate the loss of efficiency, measurement points need to be extracted from the graph with the online program WebPlotDigitizer. This gives the following result:

The measurement points from this graph have been entered in Excel for further processing. The efficiency of the red, green and blue channel of the filter can now easily be calculated by dividing the total efficiency of the channel by the monochrome efficiency, for every wavelength. This gives the following result:

We know know the efficiency of every channel at every wavelength. We also know that the sensor has 2 green, 1 red and 1 blue filter for every 4 pixels. So we can now calculate the total efficiency of the sensor by taking the average of 2 times the green, 1 times the red and 1 times the blue efficiency. This gives the following result:

This shows that the quantum efficiency of this filter peaks at about 57% at a wavelength of about 585 nm. The average is much lower though. If the average is taken between 410 and 700 nm, an efficiency of 40.4% is found. So a little more than half the light is lost due to the filter. The graph also shows that the efficiency is particularly bad for the blue channel, to compensate for this additional gain is often applied to the blue channel. That is why noise is much more pronounced for this channel: Why is the blue channel the noisiest?

For fun we can also calculate the total efficiency of the sensor with the filter. For this the same procedure was used as for the filter alone, this gives the following result:

This shows that the total efficiency of the sensors peaks at 37% for a wavelength of 505 nm. (exact values found in excel table). The average efficiency between 410 and 700 nm is found to be 23.8%.

It should be noted that there is some apparent gain. The quantum efficiency only tells us what percentage of the photons is registered by the sensor, not how it is perceived by the viewer. Our eyes are not equally sensitive to all colors:

Source: http://hyperphysics.phy-astr.gsu.edu/hbase/vision/colcon.html

Take for instance the blue channel at 500 nm. At that wavelength, the blue channel only has a total efficiency of 30 %. If a photon with a wavelength of 500 nm is registered by the blue channel, the sensor will not register that the wavelength was 500 nm. So this photon will be emitted by a monitor at 445 nm. At 445 nm the blue sensitive cones in the eyes are much more sensitive than at 500 nm. So even though 70% of the light with a wavelength of 500 nm gets lost, the 30% that does go through has a much higher chance of getting registered by the eyes than the original light. This gives some apparent gain. It is however not nearly enough to compensate the light lost. It is generally accepted that about 50% of the light is lost due to a color filter array. There are many sources for this:

"the Bayer layer is responsible for 50-70% light loss: https://books.google.dk/books?id=xmknCgAAQBAJ&pg=PA53&lpg=PA53&dq=light+loss+due+to+color+filter+array&source=bl&ots=JM3C0_s_UH&sig=ACfU3U1qie_zYkgaI2LcB9LsOkvnG_08dA&hl=en&sa=X&ved=2ahUKEwjvy9fngYfqAhVFiIsKHZL8AHwQ6AEwAnoECAYQAQ#v=onepage&q=light%20loss%20due%20to%20color%20filter%20array&f=false

"the Bayer design throws away around 1EV of light": https://www.dpreview.com/articles/3560214217/resolution-aliasing-and-light-loss-why-we-love-bryce-bayers-baby-anyway

"The sensitivity of a color sensor is therefore at least 2-3x lower than that of a monochrome sensor": (pop up at bottom left) http://sciencestatic.aws.aaas.org.s3.amazonaws.com/publishing/posters/zeiss/zeiss.html