48
\$\begingroup\$

Currently, most (all?) commercially available cameras capture light in three colour channels: red, green and blue. It seems to me that it would be very useful to have a camera with a greater spectral range and resolution, and so I'm wondering why cameras aren't available that capture more than three colour channels.

What do I mean exactly?

There were some queries in the comments (since deleted) about what I meant, so I'd like to give a better explanation. Visible light ranges from around 390-700nm wavelengths. There are an infinite number of wavelengths in between these two end points, but the eye has a very limited capacity to distinguish them, since it has only three colour photoreceptors. The response curves for these are shown in part (a) of the figure below. (Bigger version.) This allows us to see different colours depending on the frequency of light, since low frequency light will have more of an effect on the blue receptors and high frequency light will have more of an effect on the red receptors.

enter image description here

A digital sensor in a camera works by having filters in front of its pixels, and usually there are three types of filter. These are chosen with response curves as close as possible to figure (a) above, to mimic what the human eye sees.

However, technologically speaking there is no reason why we couldn't add a fourth filter type, for example with a peak in between blue and green, as shown in figure (b). In the next section I explain why that would be useful for post-processing of photographs, even though it doesn't correspond to anything the eye can see.

Another possibility would be to add additional channels in the infra-red or ultraviolet, as shown in figure (c), extending the spectral range of the camera. (This is likely to be more technically challenging.)

Finally, a third possibility would be to divide up the frequency range even more finely, producing a camera with a high spectral resolution. In this version, the usual RGB channels would have to be constructed in software from the more fine-grained data the sensor produces.

My question is about why DSLRs don't commonly offer any of these options besides (a), and whether there are cameras available that do offer any of the others. (I'm asking about the kind of camera you'd use to take a picture - I know there are scientific instruments that offer these kinds of feature.)

Why would this be useful?

I've been playing around with editing black and white photos, from colour shots taken with my DSLR. I find this process interesting because when editing a B&W photo the three RGB channels just become sources of data about the scene. The actual colours they represent are in a way almost irrelevant - the blue channel is useful mostly because objects in the scene differ in the amount of light they reflect in that range of wavelengths, and the fact that it corresponds to what the human eye sees as "blue" is much less relevant.

Having the three channels gives a lot of flexibility in controlling the exposure of different aspects of the final B&W image. It occurred to me while doing this that a fourth colour channel would give even more flexibility, and so I wonder why such a thing doesn't exist.

Extra colour channels would be useful for colour photography as well as black and white, and for the same reason. You'd just be constructing each of the RGB channels in the same way that you construct a B&W image now, by combining data from different channels representing light of different frequency ranges. For most purposes this would be done automatically in software, but it would offer a lot more flexibility in terms of post-processing options.

As a simple example of how this could be useful, we know that plants are very reflective in near-infrared. This fact is often used to generate striking special effects shots, in which plants appear to be bright white in colour. However, if you had the infra-red image as a fourth channel in your editing software it would be available for processing colour images, for example by changing the exposure of all the plants in the image, while leaving less IR-reflective objects alone.

In the case of infra-red I understand that there are physical reasons why it's hard to make a sensor that isn't IR-sensitive, so that digital sensors usually have an IR-blocking filter in front of them. But it should be possible to make a sensor with a higher spectral resolution in the visible range, which would enable the same kinds of advantage.

One might think that this feature would be less useful in the age of digital processing, but I actually think it would come into its own around now. The limits of what you can do digitally are set by the data available, so I would imagine that a greater amount of spectral data would enable processing techniques that can't exist at all without it.

The question

I would like to know why this feature doesn't seem to exist. Is there a huge technical challenge in making a sensor with four or more colour channels, or is the reason more to do with a lack of demand for such a feature? Do multi-channel sensors exist as a research effort? Or am I simply wrong about how useful it would be?

Alternatively, if the does exist (or has in the past), which cameras have offered it, and what are its main uses? (I'd love to see example images!)

\$\endgroup\$
3
  • \$\begingroup\$ Hey all, there has been a massive amount of discussion in comments here, more than enough to trigger the automatic systems. If you want to get in a discussion on the questions and/or answers, please take it to our chat room. Thanks. \$\endgroup\$
    – Joanne C
    Commented Oct 25, 2016 at 10:25
  • \$\begingroup\$ Related (but different) question: Why we simply use RGB instead of filters for all wavelengths \$\endgroup\$
    – user63664
    Commented Jun 6, 2017 at 7:38
  • \$\begingroup\$ What it WOULD be useful for is offering better options to deal with the problems caused by some modern types of lighting gear... \$\endgroup\$ Commented Nov 4, 2018 at 8:06

6 Answers 6

40
\$\begingroup\$

Why don't cameras offer more than 3 colour channels?

It costs more to produce (producing more than one kind of anything costs more) and gives next to no (marketable) advantages over Bayer CFA.

(Or do they?)

They did. Several cameras including retailed ones had RGBW (RGB+White) RGBE (RGB+Emerald), CYGM (Cyan Yellow Green Magenta) or CYYM (Cyan Yellow Yellow Magenta) filters.

It seems to me that it would be very useful to have a camera with a greater spectral range and resolution, and so I'm wondering why cameras aren't available that capture more than three colour channels.

The number of channels is not directly related to spectral range.

Is there a huge technical challenge in making a sensor with four or more colour channels, or is the reason more to do with a lack of demand for such a feature?

The lack of demand is decisive factor.

Additionally CYYM/CYGM filters cause increased colour noise because they require arithmetic operations with big coefficients during conversion. The luminance resolution can be better though, at the cost of the colour noise.

Do multi-channel sensors exist as a research effort? Or am I simply wrong about how useful it would be?

You are wrong in that spectral range would be bigger with more channels, you are right in that fourth channel provides a number of interesting processing techniques for both colour and monotone.

Alternatively, if the does exist (or has in the past), which cameras have offered it, and what are its main uses?

Sony F828 and Nikon 5700 for example, they and few others are even available second-handed. They are common-use cameras.


It is also interesting to know that spectral range is limited not only by the hot mirror present in most cameras but with the sensitivity of the photodiodes which make up the sensor. I do not know what type of photodiodes exactly is used in consumer cameras but here is an exemplary graph which shows the limitation of semiconductors:

Comparison of photosensitive semiconductors


Regarding software which may be used to extract fourth channel: it is probably dcraw but it should be modified and recompiled to extract just one channel.

There is a 4x3 matrix for F828 in dcraw.c which makes use of the fourth channel. Here is an idea: { 7924,-1910,-777,-8226,15459,2998,-1517,2199,6818,-7242,11401‌​‌​,3481 } - this is the matrix in linear form, most probably every fourth value represents the Emerald. You turn it into this: { 0,0,0,8191,0,0,0,0,0,0,0,0 } (I do not know what number should be there instead of 8191, gust a guesswork), recompile and the output image gets the Emerald channel after demosaicing in the red channel (if I understand the sources correctly).

\$\endgroup\$
2
  • \$\begingroup\$ Comments are not for extended discussion; this conversation has been moved to chat. \$\endgroup\$
    – Joanne C
    Commented Oct 25, 2016 at 10:39
  • 3
    \$\begingroup\$ Euri, there was some super useful information in your comments about dcraw etc., now moved to the chat room. Is there any chance you could edit that into your answer for posterity? \$\endgroup\$
    – N. Virgo
    Commented Oct 25, 2016 at 11:36
19
\$\begingroup\$

A few notes from this long-time optical systems engineer. First, there are things called "hyperspectral" cameras which use gratings or equivalent to break the incoming light into dozens or even a couple hundred color(wavelength) channels. These as you might imagine are not used, or useful, for producing color photos per se, but are great for distinguishing narrow-band spectral lines emitted, or reflected, from specific materials. geologists, for example, use these to identify mineral deposits by using a hyperspectral camera mounted in an airplane.

Next, there's a huge difference between the colors produced by each single wavelength (photon energy) and the colors our eyes perceive. We have three, or for some lucky folks, four different cones, each with different spectral response curves. You can find these curves all over the net, including the first picture on that Wikipedia page. Next, the range of colors / hues we perceive covers a whole map, while the colors produced by any single photon wavelength form a line in that map's area.

Any number of experiments, including some spectacular ones run by Edwin Land, have shown that mixing RGB is sufficient to allow the eye to reconstruct all possible vision colors. (Actually, it turns out that only two colors plus a greyscale representation of another will suffice. The optical processing in the brain is really weird)

\$\endgroup\$
0
14
\$\begingroup\$

RGB camera sensors are so popular because they reproduce human vision

That's what most people need - making photos that look like what we see.

Replacing RGB subpixels with more different kinds of filters to distinguish more bands with better spectral resolution would work but:

  • only for a single purpose. Everyone needs roughly the same RGB filters to make decent photos, but there are unlimited number of possible spectral bands that might be useful for someone. You cannot make an universal camera this way.

  • it would decrease the overall sensitivity of the sensor. Every given subpixel is useless for all light except for the narrow band it accepts. More filters = more wasted light.

So, instead of making narrowly specialized sensors, it's better to have a sensor without any built-in filters at all, and simply exchange filters during image acquisition. This way the whole sensor area is used with each filter, not just a small fraction having the matching subpixel.

\$\endgroup\$
14
  • 2
    \$\begingroup\$ This is not strictly true, 3 primary colors can't provide all the hues a human can see. If you look at the CIE chromaticity diagram it looks kind of triangular, but when you see a practical RGB triangle inscribed inside you see just how many colors you're missing. A 4th primary in the blue-green area would create a quadrilateral with maybe 20% more colors. \$\endgroup\$ Commented Oct 25, 2016 at 20:49
  • \$\begingroup\$ P.S. the only reason people are satisfied with RGB is that the colors outside of the typical gamut are rarer than the ones RGB is capable of reproducing. \$\endgroup\$ Commented Oct 25, 2016 at 20:51
  • 1
    \$\begingroup\$ How would the Bayer filter not restrict the color space to a triangle just as a display does? It's a subject that I don't think gets enough attention; it's never mentioned when talking about sensor chip differences for example. \$\endgroup\$ Commented Oct 25, 2016 at 21:18
  • 1
    \$\begingroup\$ the same question applies to the human retina having only 3 kinds of "filters" - and the answer is, i guess, that displaying and sensing are different animals! perhaps, unlike displays, where gamuts are fighting one another, it is even so boring and easy that nobody talks about it. \$\endgroup\$
    – szulat
    Commented Oct 25, 2016 at 21:27
  • 1
    \$\begingroup\$ @MarkRansom do you talk about displaying or recording? There is huge difference. The XYZ space is a tristimulus space and it's chromaticities have coordinates on that graph which I named. Here it is pictured (the outer, shaded one). \$\endgroup\$ Commented Mar 15, 2019 at 19:41
4
\$\begingroup\$

Your question really had multiple questions within! You've received several nice answers showing why having additional channels is not commonly done.

I feel your implied question about sensing other parts of the spectrum was inadequately addressed, though.

You can modify (or more likely, pay someone to modify) your digital camera by removing the "hot mirror" that reflects away infrared light. This results in a so-called "full spectrum" camera.

Now, at least most of the photons of various energies are present at the sensor. There is the matter of the "Bayer" filter, which separates pixels into red, blue, and green (and even more green) spectra.

The good news is that the Bayer filter passes a significant amount of ultra-violet and infrared energy to the sensor. This is why there is such a thing as a "hot filter" in the first place, and a reason why "UV filters" are common.

So, with the help of a yellow filter (Tiffen or Wratten #12) to knock out the blue in the imaged scene, you can record almost pure IR in the blue channel.

enter image description here

Then, you can subtract that IR from the red channel, and assign it to green and subtract that IR from the green channel, and assign it to blue, finally assigning the IR/blue channel to the red channel.

The result is pretty close to Kodak's Infrared Ektachrome film, also known as Aerochrome film.

So, without adding any hardware to the camera body, you can effectively "spread out" and visualize the spectrum that is recorded by shifting the recorded light by one primary colour. This is also called the "RGB —> IRG transform."

\$\endgroup\$
2
\$\begingroup\$

There are three color sensors in the human eye. Their spectral profiles are broad and they overlap. They each send nerve signals to the brain where the input is interpreted as color. The comment in the previous answer about the processing in the brain being weird is correct. This being the case only 3 stimuli for a given color are needed. Look at the Wikipedia article on color vision for more details.

\$\endgroup\$
0
1
\$\begingroup\$

There have been multi spectral cameras too with additional channels for IR and UV light but not as a consumer product.

\$\endgroup\$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.