What would happen if a camera used entirely different primary colors?

Question

So, as many people know, humans have three cones cells, enabling us to see three distinct "primary" colors, which may combine to form the entire spectrum that we are capable of seeing. Meanwhile, many other animals have four or more cone cells, enabling them to see an even broader, or more well-defined, spectrum.

Now, digital cameras typically record the light using an array of photosensitive "pixels". The pixels are generally arranged in groups of four, with two specialized (using filtering materials) for green, one for red, and one for blue. The intensities detected by each pixel and then converted to an RGB file using some algorithm. The intensities recorded by each specialized pixel can be mapped to the hue spectrum below.

This is what we generally want, as the resulting image makes perfect sense to our eyes and is sufficient to record a scene for most intents and purposes. But why must we restrict a camera to capturing and recording light the way humans see it?

Let's say we changed the filters over the photosensitive "pixels" to optimally admit different wavelengths, particularly ones that we don't normally see, or ones closer together in a specialized color range that would provide more detail. From there, we could stretch the hue spectrum, with 0/360 being the first color, 120 being the second color, and 240 being the final color.

I'm very curious to see what the result of this would be, if for example we picked the wavelengths of 800 nm, 400 nm, and 200 nm in order to see a little bit more into the infrared and ultraviolet. Or, if we had a collage of something that appeared blue, we could choose the wavelengths of 450 nm, 475 nm, and 500 nm in order to distinguish similar shades more easily. Another possibility would be to detect four different wavelengths and map these onto the hue spectrum. This would allow for something like "tetrachromatic" photography.

Here is a mockup of what one might expect (changed to better reflect the question):

Here are some things to answer:

Is this already being done? If not, why not? (I've seen ultraviolet and infrared photography before, but it is usually black / white or black / magenta. Why use one dimension and why not stretch the spectrum?)

What exists in terms of consumer technology to take images this way?

Are there limitations in technology to what wavelengths can be captured?

Answer 1

Color photography is indeed based on the tri-color theory. The world saw the first color picture in 1861 made using red, green, and blue filters by James Clark Maxwell. Today’s color photography is based on his method. In 1891, Gabriel Lippmann demonstrated full color images using a single sheet of black & white film, no filters, no colored dye or pigment. This process fell by the wayside because the beautiful images could not be copied or duplicated. In the 1950’s Dr. Edwin Land of the Polaroid Corporation demonstrated that he could make beautiful color pictures using just two colors (579 & 599 nanometers). This too fell by the wayside.

Imaging engineers, long ago wanted to image using the non-visual portion of the spectrum. It was quickly discovered that ordinary photo plates and film imaged only recording violet and blue light as well as ultraviolet (4 nanometers to 380 nanometers). They discovered that films record X-Ray and infrared.

What other portions of the spectrum can be imaged? Astronomers image via radio frequencies Weathermen and the aviation industry, image via radar. The optical microscope is limited to about 1000X, however the electron microscope images molecules and atoms.

We image the human body using sound waves (ultrasound). We image the human body using radio waves (magnetic resonance imaging, MRI).

There are countless other ways to image. At first images made using the non-visual portion of the spectrum were presented only in black & white. After all, we can’t see via this radiation, so any graphic image we present will be an incorrect presentation.

Now doctors looking at X-rays are looking for subtle changes in shades of gray. With computer logic we can change black & white tones into false colors to better differentiate. Thus the modern X-ray and sonogram are displayed with false colors. The other imaging disciplines of science follow suit. False color images made from the non-visual portions of the spectrum are routine.

Answer 2

Is this already being done?

Sure. The Hubble Space Telescope senses the near IR, visible, and near UV spectrum. Any images you see from Hubble that contain information outside of the visible spectrum are false color images.

Similarly, images from Chandra, which observes the X-ray spectrum, can only be visualized by mapping its "tones" to the visible light spectrum.

In the non-astronomical domain, the millimeter-wave scanners at airports map the, well, millimeter-range signals, into the visual domain.

What exists in terms of consumer technology to take images this way?

FLIR cameras, for one.

Are there limitations in technology to what wavelengths can be captured?

That question is overly broad (there's always limits in technology).

Answer 3

Some general use photographic cameras actually record outside the visible spectrum, so there is some experience with that. Leica M8 was notoriously known for recording IR. The extended range had bad impact on color accuracy and Leica had to give customers IR/cut filters for their lenses to resolve that.

Extending to UV is difficult as glass in the lenses blocks UV.

The effect of capturing wider spectrum at once - at least as seen with the Leica or modified cameras - is not particularly pleasant, interesting or useful. Even if you manage to process the data in some interesting way, you will get a single trick ponny.

There are companies that will remove the filters from the sensor, if you are interested. You could use color filters with different spectra on top of your lens, create three exposures with different filters and blend them in software.

Answer 4

The intensities recorded by each specialized pixel can be mapped to the hue spectrum below.

The Bayer matrix does not map to any color. The image is interpolated to yield a full-color-per-pixel image, where each pixel has an R, G, and B component. These RGB components can be mapped to a color space, such as sRGB or adobeRGB, but the RGB mode does not inherently have a color space.

Let's say we changed the filters over the photosensitive "pixels" to optimally admit different wavelengths, particularly ones that we don't normally see, or ones closer together in a specialized color range that would provide more detail.

The question is one of what constitutes detail. If the goal is to perform spectroscopy, one should not use a normal camera but instead a spectrometer or spectrophotometer.

Each filter added reduces the overall efficiency of the sensor. An RGB camera has a net efficiency of about 20~25% over the visible band. A UV-VIS-IR camera utilizing 5 filters will have closer to 10% efficiency over that band, and the UV and IR bands have less light in them to begin with, so they will need much more gain and be noisier.

Is this already being done? If not, why not?

Yes, they're called spectrophotometers. In fact, something extremely similar to what you are talking about is done. MastCAM on the curiosity rover uses a special bayer array that bleeds significant IR light coupled with an 8 filter wheel. The camera can then do full resolution narrowband imaging in the short wave IR at 6 different wavelengths.

Is it done commonly, no. Outside of scientific inquery, this type of setup makes a very bulky camera with a more complex metadata scheme required. These are two things that are the bane of consumer products.

Answer 5

Note that you can use any 3 primaries in the visible spectrum and you'll generate an accurate image (within the limits of your recording and display devices) so long as the recording device and display device use the same primaries. For example, most cameras released over the last 10 years have sensors that capture colors that fit into the sRGB colorspace. And most monitors display in the sRGB colorspace (or something close to it).

Newer cameras (currently high end, but no doubt soon consumer cameras) are able to capture in a wider colorspace called DCI-P3. It's still considered an "RGB" colorspace because the primaries that are captured are what we would subjectively call "Red," "Green," and "Blue," though they are different from the sRGB primaries. Several LCD displays in recent computers and cell phones can now display in the DCI-P3 colorspace as well. These devices capture and display a much wider range of colors.

If you want to see what it would look like to capture with one set of primaries and display in another set, you can use a hue adjusting filter in your favorite image editor. Rotating the hue will show you the equivalent of capturing with one set of primaries and displaying with another.

Answer 6

Are there limitations in technology to what wavelengths can be captured?

There is:

• Near infrared photography (night vision),
• Mid infrared photography (Thermal imagery) http://www.ipac.caltech.edu/outreach/Edu/Regions/irregions.html
• X-rays (not only to see bones with passing trhu x-rays, but some are so sensitive that you can see reflected ones) https://en.wikipedia.org/wiki/Backscatter_X-ray,
• Radio telescopes and microwave telescopes.
• Gamma ray telescopes.
• Uv cameras, etc.

So basicly all the spectrum has being explored.

But all of theese has diferent systems. Somethings to consider are the relationship between wavelength and the matter, the ambiance and more specific the sensor.

Take a look at why we see "visible light" If the wavelength in particular do not pass trhu the upper atmosphere, there would be no lightsource, aka sunlight: https://en.wikipedia.org/wiki/File:Atmospheric_electromagnetic_opacity.svg the other passing light it is the radio but it is too long that passes through our body.

The diferences in wavelengths are exponential, so yes, there are some technological problems related on which electromagnetic waves something can percive, with our eyes or instruments.

What exists in terms of consumer technology to take images this way?

Infrared

A simple question is you can have near infrared film and filters for you to experiment, and you could adapt your dlsr: https://photo.stackexchange.com/search?q=infrared

There are some night vision cameras and lenses.

You could buy a far infrared thermal camera but it is not a "consumer" product because they are expensive.

UV I doubt it is legal to fire a more enrgetic lightbeam at people. Remember some long exposures to UV light can burn, first of all your retina. so you need a low light environment to use a low power UV. "Blacklight" images are UV induced reflections so yea, you can also do that. https://en.wikipedia.org/wiki/Ultraviolet_photography

I've seen ultraviolet and infrared photography before, but it is usually black / white or black

If you can not see it it is an interpretation. Night vision googles are normaly green because our eyes are more sensitive to green, and when a soldier removes the lens his eyes adapt more easily to the darkness. If you have a black and white vision the time for the eye to adapt to the darkness would be much longer.

Why use one dimension?

The "3D imensionality" of the "primary" colors is just because the way our brains percive light. Magenta is not in the visible espectrum, it has no wavelength asociated with it. Our brain interprets it as magenta.

In reality the electromagnetic wavelength spectrum is unidimensional. It is bimensional if we use intensity as a the second dimension to produce images.

Why not stretch the spectrum?

We have to strech the spectrum. Or we see it or we do not. A black and white image is actually a re-compression of a wavelength we do not see into the limited spectrum we see.

Of cours you could make an Xray digital machine to display the colors magenta, I had an old CTR monitor that did that on its own. But this is more a psicological aspect than a technical one.

But in some fields like thermal imagery the color spectrum is used to detect the diferences in temperature, so it is currently done.

Regarding why not tweeking the visible light spectrum or not, I think it is totally an artistic interpretation, so you can do whatever you want.

But

But on the other hand would be interesting to have a Tetrachromacy simulator of the few people that have it, simmilar on how we have color blindness simulators like this: http://www.color-blindness.com/coblis-color-blindness-simulator/

Answer 7

I am reading a really interesting book called "Vision and Art, the Biology of Seeing" by Margaret Livingstone. I am not yet done with it, but the chapters I have read so far talk about how the eye perceives color, how colors are blended (both light and pigments) and what the limitations are and why. It may help answer some of your questions on how the eye works, and what the limits are on photo capabilities.