Can we improve phone cameras using separate sensors for each of RGB?

Question

So I was reading all about the multi camera system on the new Huawei P20 Pro which really got me thinking about how we can use multiple cameras to improve image quality, especially in such a compact form such as a phone.

I looked into the Bayer filter and noticed that when each photosite collects the intensity of one of the RGB components, we also discard the intensities of the two other RGB components. And so we have to approximate the discarded intensities using demosaicing.

Instead of resorting to demosaicing, what if we create a camera system that has 3 cameras, each responsible for only one of RGB? No more demosaicing required. Wouldn't it drastically improve the quality of the final image since there's no more color approximations?

Would it be really expensive, given that the P20 Pro already seems to be ok with 3 cameras?

Answer 1

Your idea is very interesting. But there are some very serious problems.

1. Parallax. Because the sensors will be shifted to each other they will "see" different picture. And the process of creating one photo from those 3 will be serious challenge and will need a lot of computing power
2. Space. In mobile phones you do not have much space so adding one more sensor, optics, cables can be unfeasible
3. Price. Adding more censors and optics will increase the price. And especially this market is very sensitive in this direction. At the end phone is mainly for talking (IMHO)
4. Market needs. How many people will want even better photos from the phone?
5. Cleaning. Now you (maybe) clean one lens, then you will clean 3. Right, they are small, but you keep you phone (usually) in the pocket

Sure this list can be extended, but hope this is enough

EDIT: Related to point 1 if sensors are in one plane this will additionally "distort" the images captured by them. And this will mean on close-up you should have 3 different focusing systems.

Answer 2

Modern smartphone image quality is limited by their tiny sensor size, lens quality and diffraction, not the demosaicing process.

Therefore, the idea does not seem to be useful.

Note that aforementioned P20 features a monochrome camera, introducing the same kind of problems with image registration as the hypothetical separate R, G and B cameras (parallax, inconsistent images between channels). And the goal was increasing the low light sensitivity rather than color resolution. Because the color resolution is not a problem and nobody cares about it.

Answer 3

Many video cameras already do just what you propose. But they use a beam splitting system between the lens and the sensors which essentially cuts the amount of light reaching each sensor by roughly the same proportion as using a Bayer masked filter does.

The biggest problem with doing it with three different lenses while maintaining a very high degree of precision is the complexity and cost of creating sets of perfectly matched lenses. This is not unsolvable, but the solution is likely more expensive and computationally complex than using a Bayer mask and demosaicing.

Instead of resorting to demosaicing, what if we create a camera system that has 3 cameras, each responsible for only one of RGB? No more demosaicing required. Wouldn't it drastically improve the quality of the final image since there's no more color approximations?

Each of those three cameras would need a single-color filter in front of it in order to use the comparative differences from each to produce a "color" image. This means the same proportion of light that is absorbed/reflected by the tiny filters on a Bayer masked sensor would still be absorbed/reflected by the large single color filters in front of each of the three independent sensors. Without a different colored filter in front of each one they would all be identical monochrome cameras producing the same data.

Here's the thing about "color": there's no such thing as "color" in nature. Light only has wavelengths. Electromagnetic radiation sources on either end of the visible spectrum also have wavelengths. The only difference between visible light and other forms of electromagnetic radiation, such as radio waves, is that our eyes chemically react to certain wavelengths of electromagnetic radiation and do not react to other wavelengths. Beyond that there is nothing substantially different between "light" and "radio waves" or "X-rays". Nothing.

The cones in our retinas are made up of three different sizes that are each most responsive to a different wavelength of electromagnetic radiation. In the case of our "red" and "green" cones there is very little difference in the response to most wavelengths of light. But by comparing the difference and which has a higher response, the red or the green cones, our brains can interpolate how far and in which direction towards red or towards blue, the light source is strongest.

Color is a construct of our eye brain system that compares the relative response of the three different sized cones in our retinas and creates a perception of "color" based on the different amounts each set of cones responds to the same light. There are many colors humans perceive that can not be created by a single wavelength of light. "Magenta", for instance, is what our brains create when we are simultaneously exposed to red light on one end of the visible spectrum and blue light on the other end of the visible spectrum.

Answer 4

This technology exists in the form of a 3ccd / 3-chip camera. There would be some issues with fitting the prism into the space available in a phone. I think you would end up with something at least as thick as a gen1 iPhone but something closer to the thickness of a deck of cards might be needed. I suppose the market might be strong enough to support such a device. There was enough interest for pre-orders to sell out for the $1200 hydrogen one. I know I almost bought one.