7

So I was reading all about the multi camera system on the new Huawei P20 Pro which really got me thinking about how we can use multiple cameras to improve image quality, especially in such a compact form such as a phone.

I looked into the Bayer filter and noticed that when each photosite collects the intensity of one of the RGB components, we also discard the intensities of the two other RGB components. And so we have to approximate the discarded intensities using demosaicing.

Instead of resorting to demosaicing, what if we create a camera system that has 3 cameras, each responsible for only one of RGB? No more demosaicing required. Wouldn't it drastically improve the quality of the final image since there's no more color approximations?

Would it be really expensive, given that the P20 Pro already seems to be ok with 3 cameras?

  • 1
    You don't actually lose as much as you think with a Bayer masked sensor. The response of the sensels under each color filter still record some of the other colored light. It is much like using colored filters with B&W film to get three different tonal ranges. It's the difference between how each grid with a filter centered on a different color responds that allows "color" information to be interpolated (which is also exactly how human vision "creates color"). For more, please see: RAW files store 3 colors per pixel, or only one? – Michael C May 19 '18 at 7:48
  • you lose about 2/3 of the light - no, not because there are 3 channels and they absolutely must cut 1/3 of the spectrum to make it look nice for mathematicians. the filters try to mimic the human vision and maximize the efficiency at the same time and the most common compromise is to filter out about 1/2 to 2/3 of the light, which can be exactly calculated by measuring the area below the filter transmission curves. for those not trusting the soulless numbers ;-) here some brave guy is physically removing the bayer filters, increasing the sensitivity of the exposed area: goo.gl/tbCdMA – szulat May 19 '18 at 13:03
  • @szulat for some of the best sensors, which have efficiency in the upper 50s range, it's actually less than 1/2 the light that is lost, not 2/3. – Michael C May 20 '18 at 5:27
  • @MichaelClark no, because the reported QE usually means peak efficiency for the "best" single wavelength (e.g. green). the bayer filter transmission across wavelengths needs to be added on top of that. – szulat May 20 '18 at 11:02
  • 1
    @MichaelClark as for colors - yes! you could make a "diluted" bayer filter, filtering very little, allowing a lot of light into sensor and increasing the sensitivity. but the color information would need to be calculated by amplifying the channel difference, which means amplifying noise. that's one of the reasons why the foveon sensor or CMY sensors fail so badly in low light - despite having a theoretical advantage of using more light than a regular bayer sensor. so, in the end, everyone is using the good old regular bayer. – szulat May 21 '18 at 10:26
6

Your idea is very interesting. But there are some very serious problems.

  1. Parallax. Because the sensors will be shifted to each other they will "see" different picture. And the process of creating one photo from those 3 will be serious challenge and will need a lot of computing power
  2. Space. In mobile phones you do not have much space so adding one more sensor, optics, cables can be unfeasible
  3. Price. Adding more censors and optics will increase the price. And especially this market is very sensitive in this direction. At the end phone is mainly for talking (IMHO)
  4. Market needs. How many people will want even better photos from the phone?
  5. Cleaning. Now you (maybe) clean one lens, then you will clean 3. Right, they are small, but you keep you phone (usually) in the pocket

Sure this list can be extended, but hope this is enough

EDIT: Related to point 1 if sensors are in one plane this will additionally "distort" the images captured by them. And this will mean on close-up you should have 3 different focusing systems.

  • Parallax: many phone cameras already combine images from multiple sensors just fine such as P20 Pro. – Baseer May 19 '18 at 7:46
  • 2
    But not split by colour... – Romeo Ninov May 19 '18 at 12:58
5

Modern smartphone image quality is limited by their tiny sensor size, lens quality and diffraction, not the demosaicing process.

Therefore, the idea does not seem to be useful.

Note that aforementioned P20 features a monochrome camera, introducing the same kind of problems with image registration as the hypothetical separate R, G and B cameras (parallax, inconsistent images between channels). And the goal was increasing the low light sensitivity rather than color resolution. Because the color resolution is not a problem and nobody cares about it.

5

Many video cameras already do just what you propose. But they use a beam splitting system between the lens and the sensors which essentially cuts the amount of light reaching each sensor by roughly the same proportion as using a Bayer masked filter does.

The biggest problem with doing it with three different lenses while maintaining a very high degree of precision is the complexity and cost of creating sets of perfectly matched lenses. This is not unsolvable, but the solution is likely more expensive and computationally complex than using a Bayer mask and demosaicing.

Instead of resorting to demosaicing, what if we create a camera system that has 3 cameras, each responsible for only one of RGB? No more demosaicing required. Wouldn't it drastically improve the quality of the final image since there's no more color approximations?

Each of those three cameras would need a single-color filter in front of it in order to use the comparative differences from each to produce a "color" image. This means the same proportion of light that is absorbed/reflected by the tiny filters on a Bayer masked sensor would still be absorbed/reflected by the large single color filters in front of each of the three independent sensors. Without a different colored filter in front of each one they would all be identical monochrome cameras producing the same data.

Here's the thing about "color": there's no such thing as "color" in nature. Light only has wavelengths. Electromagnetic radiation sources on either end of the visible spectrum also have wavelengths. The only difference between visible light and other forms of electromagnetic radiation, such as radio waves, is that our eyes chemically react to certain wavelengths of electromagnetic radiation and do not react to other wavelengths. Beyond that there is nothing substantially different between "light" and "radio waves" or "X-rays". Nothing.

The cones in our retinas are made up of three different sizes that are each most responsive to a different wavelength of electromagnetic radiation. In the case of our "red" and "green" cones there is very little difference in the response to most wavelengths of light. But by comparing the difference and which has a higher response, the red or the green cones, our brains can interpolate how far and in which direction towards red or towards blue, the light source is strongest.

Color is a construct of our eye brain system that compares the relative response of the three different sized cones in our retinas and creates a perception of "color" based on the different amounts each set of cones responds to the same light. There are many colors humans perceive that can not be created by a single wavelength of light. "Magenta", for instance, is what our brains create when we are simultaneously exposed to red light on one end of the visible spectrum and blue light on the other end of the visible spectrum.

  • yes, it is good to note that similar systems already exist. but the "biggest problem" you mention is not a real problem anymore, now when everything can be easily corrected in software. as proved by phones like the P20 pro, or even more extravagant "Light 16". actually, the perfectly matched lens can't even exist because there is always parallax and such images would have to be heavily processed every single time. – szulat May 19 '18 at 12:29
  • 3
    And one other problem in sense of phone will be the size. Such split system require some depth which is usually not available in phones – Romeo Ninov May 19 '18 at 13:00
  • 2
    The depth issue for the beam splitter should go without saying. But the main point is that the beam splitter also reduces the amount of light reaching each sensor. So the number of photos making it down a pixel well or the overall sensitivity of the system with a one lens/three sensor system doesn't increase over a Bayer masked sensor. – Michael C May 19 '18 at 16:45
3

This technology exists in the form of a 3ccd / 3-chip camera. There would be some issues with fitting the prism into the space available in a phone. I think you would end up with something at least as thick as a gen1 iPhone but something closer to the thickness of a deck of cards might be needed. I suppose the market might be strong enough to support such a device. There was enough interest for pre-orders to sell out for the $1200 hydrogen one. I know I almost bought one.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.