What would happen if a camera used entirely different primary colors?

Question

So, as many people know, humans have three cones cells, enabling us to see three distinct "primary" colors, which may combine to form the entire spectrum that we are capable of seeing. Meanwhile, many other animals have four or more cone cells, enabling them to see an even broader, or more well-defined, spectrum.

Now, digital cameras typically record the light using an array of photosensitive "pixels". The pixels are generally arranged in groups of four, with two specialized (using filtering materials) for green, one for red, and one for blue. The intensities detected by each pixel and then converted to an RGB file using some algorithm. The intensities recorded by each specialized pixel can be mapped to the hue spectrum below.

This is what we generally want, as the resulting image makes perfect sense to our eyes and is sufficient to record a scene for most intents and purposes. But why must we restrict a camera to capturing and recording light the way humans see it?

Let's say we changed the filters over the photosensitive "pixels" to optimally admit different wavelengths, particularly ones that we don't normally see, or ones closer together in a specialized color range that would provide more detail. From there, we could stretch the hue spectrum, with 0/360 being the first color, 120 being the second color, and 240 being the final color.

I'm very curious to see what the result of this would be, if for example we picked the wavelengths of 800 nm, 400 nm, and 200 nm in order to see a little bit more into the infrared and ultraviolet. Or, if we had a collage of something that appeared blue, we could choose the wavelengths of 450 nm, 475 nm, and 500 nm in order to distinguish similar shades more easily. Another possibility would be to detect four different wavelengths and map these onto the hue spectrum. This would allow for something like "tetrachromatic" photography.

Here is a mockup of what one might expect (changed to better reflect the question):

Here are some things to answer:

Is this already being done? If not, why not? (I've seen ultraviolet and infrared photography before, but it is usually black / white or black / magenta. Why use one dimension and why not stretch the spectrum?)

What exists in terms of consumer technology to take images this way?

Are there limitations in technology to what wavelengths can be captured?

Answer 1

Color photography is indeed based on the tri-color theory. The world saw the first color picture in 1861 made using red, green, and blue filters by James Clark Maxwell. Today’s color photography is based on his method. In 1891, Gabriel Lippmann demonstrated full color images using a single sheet of black & white film, no filters, no colored dye or pigment. This process fell by the wayside because the beautiful images could not be copied or duplicated. In the 1950’s Dr. Edwin Land of the Polaroid Corporation demonstrated that he could make beautiful color pictures using just two colors (579 & 599 nanometers). This too fell by the wayside.

Imaging engineers, long ago wanted to image using the non-visual portion of the spectrum. It was quickly discovered that ordinary photo plates and film imaged only recording violet and blue light as well as ultraviolet (4 nanometers to 380 nanometers). They discovered that films record X-Ray and infrared.

What other portions of the spectrum can be imaged? Astronomers image via radio frequencies Weathermen and the aviation industry, image via radar. The optical microscope is limited to about 1000X, however the electron microscope images molecules and atoms.

We image the human body using sound waves (ultrasound). We image the human body using radio waves (magnetic resonance imaging, MRI).

There are countless other ways to image. At first images made using the non-visual portion of the spectrum were presented only in black & white. After all, we can’t see via this radiation, so any graphic image we present will be an incorrect presentation.

Now doctors looking at X-rays are looking for subtle changes in shades of gray. With computer logic we can change black & white tones into false colors to better differentiate. Thus the modern X-ray and sonogram are displayed with false colors. The other imaging disciplines of science follow suit. False color images made from the non-visual portions of the spectrum are routine.

Answer 2

Is this already being done?

Sure. The Hubble Space Telescope senses the near IR, visible, and near UV spectrum. Any images you see from Hubble that contain information outside of the visible spectrum are false color images.

Similarly, images from Chandra, which observes the X-ray spectrum, can only be visualized by mapping its "tones" to the visible light spectrum.

In the non-astronomical domain, the millimeter-wave scanners at airports map the, well, millimeter-range signals, into the visual domain.

What exists in terms of consumer technology to take images this way?

FLIR cameras, for one.

Are there limitations in technology to what wavelengths can be captured?

That question is overly broad (there's always limits in technology).

Answer 3

Some general use photographic cameras actually record outside the visible spectrum, so there is some experience with that. Leica M8 was notoriously known for recording IR. The extended range had bad impact on color accuracy and Leica had to give customers IR/cut filters for their lenses to resolve that.

Extending to UV is difficult as glass in the lenses blocks UV.

The effect of capturing wider spectrum at once - at least as seen with the Leica or modified cameras - is not particularly pleasant, interesting or useful. Even if you manage to process the data in some interesting way, you will get a single trick ponny.

There are companies that will remove the filters from the sensor, if you are interested. You could use color filters with different spectra on top of your lens, create three exposures with different filters and blend them in software.

Answer 4

The intensities recorded by each specialized pixel can be mapped to the hue spectrum below.

The Bayer matrix does not map to any color. The image is interpolated to yield a full-color-per-pixel image, where each pixel has an R, G, and B component. These RGB components can be mapped to a color space, such as sRGB or adobeRGB, but the RGB mode does not inherently have a color space.

Let's say we changed the filters over the photosensitive "pixels" to optimally admit different wavelengths, particularly ones that we don't normally see, or ones closer together in a specialized color range that would provide more detail.

The question is one of what constitutes detail. If the goal is to perform spectroscopy, one should not use a normal camera but instead a spectrometer or spectrophotometer.

Each filter added reduces the overall efficiency of the sensor. An RGB camera has a net efficiency of about 20~25% over the visible band. A UV-VIS-IR camera utilizing 5 filters will have closer to 10% efficiency over that band, and the UV and IR bands have less light in them to begin with, so they will need much more gain and be noisier.

Is this already being done? If not, why not?

Yes, they're called spectrophotometers. In fact, something extremely similar to what you are talking about is done. MastCAM on the curiosity rover uses a special bayer array that bleeds significant IR light coupled with an 8 filter wheel. The camera can then do full resolution narrowband imaging in the short wave IR at 6 different wavelengths.

Is it done commonly, no. Outside of scientific inquery, this type of setup makes a very bulky camera with a more complex metadata scheme required. These are two things that are the bane of consumer products.

Answer 5

Note that you can use any 3 primaries in the visible spectrum and you'll generate an accurate image (within the limits of your recording and display devices) so long as the recording device and display device use the same primaries. For example, most cameras released over the last 10 years have sensors that capture colors that fit into the sRGB colorspace. And most monitors display in the sRGB colorspace (or something close to it).

Newer cameras (currently high end, but no doubt soon consumer cameras) are able to capture in a wider colorspace called DCI-P3. It's still considered an "RGB" colorspace because the primaries that are captured are what we would subjectively call "Red," "Green," and "Blue," though they are different from the sRGB primaries. Several LCD displays in recent computers and cell phones can now display in the DCI-P3 colorspace as well. These devices capture and display a much wider range of colors.

If you want to see what it would look like to capture with one set of primaries and display in another set, you can use a hue adjusting filter in your favorite image editor. Rotating the hue will show you the equivalent of capturing with one set of primaries and displaying with another.

Answer 6

Are there limitations in technology to what wavelengths can be captured?

There is:

• Near infrared photography (night vision),
• Mid infrared photography (Thermal imagery) http://www.ipac.caltech.edu/outreach/Edu/Regions/irregions.html
• X-rays (not only to see bones with passing trhu x-rays, but some are so sensitive that you can see reflected ones) https://en.wikipedia.org/wiki/Backscatter_X-ray,
• Radio telescopes and microwave telescopes.
• Gamma ray telescopes.
• Uv cameras, etc.

So basicly all the spectrum has being explored.

But all of theese has diferent systems. Somethings to consider are the relationship between wavelength and the matter, the ambiance and more specific the sensor.

Take a look at why we see "visible light" If the wavelength in particular do not pass trhu the upper atmosphere, there would be no lightsource, aka sunlight: https://en.wikipedia.org/wiki/File:Atmospheric_electromagnetic_opacity.svg the other passing light it is the radio but it is too long that passes through our body.

The diferences in wavelengths are exponential, so yes, there are some technological problems related on which electromagnetic waves something can percive, with our eyes or instruments.

What exists in terms of consumer technology to take images this way?

Infrared

A simple question is you can have near infrared film and filters for you to experiment, and you could adapt your dlsr: https://photo.stackexchange.com/search?q=infrared

There are some night vision cameras and lenses.

You could buy a far infrared thermal camera but it is not a "consumer" product because they are expensive.

UV I doubt it is legal to fire a more enrgetic lightbeam at people. Remember some long exposures to UV light can burn, first of all your retina. so you need a low light environment to use a low power UV. "Blacklight" images are UV induced reflections so yea, you can also do that. https://en.wikipedia.org/wiki/Ultraviolet_photography

I've seen ultraviolet and infrared photography before, but it is usually black / white or black

If you can not see it it is an interpretation. Night vision googles are normaly green because our eyes are more sensitive to green, and when a soldier removes the lens his eyes adapt more easily to the darkness. If you have a black and white vision the time for the eye to adapt to the darkness would be much longer.

Why use one dimension?

The "3D imensionality" of the "primary" colors is just because the way our brains percive light. Magenta is not in the visible espectrum, it has no wavelength asociated with it. Our brain interprets it as magenta.

In reality the electromagnetic wavelength spectrum is unidimensional. It is bimensional if we use intensity as a the second dimension to produce images.

Why not stretch the spectrum?

We have to strech the spectrum. Or we see it or we do not. A black and white image is actually a re-compression of a wavelength we do not see into the limited spectrum we see.

Of cours you could make an Xray digital machine to display the colors magenta, I had an old CTR monitor that did that on its own. But this is more a psicological aspect than a technical one.

But in some fields like thermal imagery the color spectrum is used to detect the diferences in temperature, so it is currently done.

Regarding why not tweeking the visible light spectrum or not, I think it is totally an artistic interpretation, so you can do whatever you want.

But

But on the other hand would be interesting to have a Tetrachromacy simulator of the few people that have it, simmilar on how we have color blindness simulators like this: http://www.color-blindness.com/coblis-color-blindness-simulator/

Answer 7

I am reading a really interesting book called "Vision and Art, the Biology of Seeing" by Margaret Livingstone. I am not yet done with it, but the chapters I have read so far talk about how the eye perceives color, how colors are blended (both light and pigments) and what the limitations are and why. It may help answer some of your questions on how the eye works, and what the limits are on photo capabilities.

Answer 8

Last things first:

But why must we restrict a camera to capturing and recording light the way humans see it?

We don't always restrict our cameras to capturing and recording light that way. The recently launched Webb space telescope, for example, will image infrared wavelengths. Of course the images it captures will be rendered in colors we can perceive. These will be what we call false color images. If we created displays to emit color accurate output of the Webb telescope we'd need displays that emit infrared light. When we tried to view images we'd see a blank screen, because our eyes are not sensitive to the wavelengths of infrared light the Webb telescope is going to record.

On the other hand, if we want a camera to produce images that will be perceived to be the same colors as we perceive the things that we used the camera to take an image of, then the closer to the peak sensitivities of human retinal cones we can match the filters on our Bayer masks the more successful our results will be.

---

The following is the TL:DR portion of our answer to the question; Why are Red, Green, and Blue the primary colors of light?

Do primary colors really exist in the real world?

No.

There are no primary colors of light, in fact there is no color intrinsic in light at all (or any other wavelength of electromagnetic radiation). There are only colors in the perception of certain wavelengths of EMR by our eye/brain systems.

Or did we select red, green, and blue because those are the colors that human eyes' cones respond to?

We use three-color reproduction systems because the human vision system is trichromatic, but the primary colors we use in our three-color reproduction systems do not match each of the three colors, respectively, to which each of the three types of cones in the human retina are most responsive.

For a much more comprehensive discussion regarding human perception of color, please see our complete answer to the question Why are Red, Green, and Blue the primary colors of light?

---

There's a rather lengthy answer to this tangentially related question that you might find helpful. That answer draws quite a bit from this online color course hosted at Memcode.

---

We'll give some background here that informs the answer at the very top. This is all covered in much greater detail in the two answers linked above.

First, let's make sure we understand how our eyes and cameras actually produce color. There are a few assumptions reflected in the question that aren't entirely correct. As Roger Cicala says in one of his excellent blog entries at lensrentals.com:

"Some things I wrote about years ago, I now realize are, um, well, less correct than I would have liked."

The understanding reflected in the question of the sensitivities of the cones in the human retina is flawed. They're not most sensitive to the wavelengths we perceive as Red, Green, and Blue. Long before we had the ability to precisely measure this, it was assumed that each type of retinal cone was most sensitive to red, green, and blue. We later learned, from the work of Gunnar Svaetichin, this was

"... um, well, less correct than (we) would have liked."

It wasn't until the late 20th Century that we were able to nail down exactly at what wavelengths each of the three types of retinal cones are most sensitive.

Our M cones are most sensitive pretty close to green, our S cones are pretty close to blue (with a bias towards violet), but our L cones are nowhere near most sensitive to red. They're actually most sensitive to a color somewhere between lime green and yellow.

Note the three dots near the bottom of the graphic above: Those are the three colors to which each type of our retinal cones are most sensitive.

Here's another representation that makes it even clearer with the lines in the actual color to which each type of cone is most sensitive:

The Bayer masks in our digital cameras are not actually red, green, and blue, either, all of the cute little red, green, and blue drawings on the internet notwithstanding.

It can vary from one manufacturer to another, and even from one model to another, but the colors of the filters on Bayer masks are usually about the same as the peak sensitivities of out S cones and L cones while the "red" filter tends to be a yellow-orange color around 600nm that is about halfway between the lime-green peak sensitivity of our L cones at 564 nm and the "Red" that our transmissive displays emit at about 640nm.

The above image is a microscopic capture of a sensor that has had part of the Bayer mask removed. As you can see, the "red" filters are actually yellow-orange and the "blue" filters are as much violet as they are blue.

Out transmissive displays (TVs, monitors, etc.), however, do emit three color bands more or less centered on what we call 'Red', 'Green', and 'Blue'.

It can vary significantly from one design to the next. Some higher end displays have even narrower bands of emission for each color.

For further reading:

Why are Red, Green, and Blue the primary colors of light?
RAW files store 3 colors per pixel, or only one?
Why don't mainstream sensors use CYM filters instead of RGB?
Filter for RGB separation and its effect on the image
What does an unprocessed RAW file look like?