15
\$\begingroup\$

Why is it impossible for camera sensors to function the way the human eye can? What I mean is, why does a certain portion of the image have to be over/underexposed if we compensate for dark and light areas respectively when taking a photo and deciding on the aperture and shutter speed settings.

I understand that the light getting in depends on aperture and shutter speed but since DSLRs are digital, can't there be a technology that would enable each sensor cell use of its own metering and therefore they wouldn't all be subjected to the same amount of light but depending on the metering, a CPU of the camera would shut off certain cells as not to overexpose them.

I hope I'm not saying nonsense. It sure seems to me like a plausible idea.

\$\endgroup\$
6
  • \$\begingroup\$ It sounds like you are asking why a sensor cannot have an unlimited dynamic range? Or maybe another path would be "how does a camera sensor work". Interesting question regardless. \$\endgroup\$
    – dpollitt
    Commented Jan 3, 2012 at 16:25
  • \$\begingroup\$ Yeah, you would say that the question is about unlimited dynamic range but it actually seems pretty weird to me that since each sensor cell has the ability to "record" or batter say capture any amount of light (disregard the clipping of white/black), why can't it do it on the single cell basis but capture the same amount throughout the whole sensor instead? Perhaps it would demand a bit more CPU power and whatnot but I am interested whether it was even considered by any DSLR manufacturer... \$\endgroup\$
    – user7264
    Commented Jan 3, 2012 at 16:36
  • 2
    \$\begingroup\$ Too much wiring is my guess. You would need non-trivial additional circuitry per photosite. The closest to date are cameras that read half of pixels partway through the exposure. \$\endgroup\$
    – Itai
    Commented Jan 3, 2012 at 16:54
  • \$\begingroup\$ Do you have a link to the article? So the only answer to my original question is that it is, in fact, too complicated at this point in DSLR technology history? \$\endgroup\$
    – user7264
    Commented Jan 3, 2012 at 17:08
  • 1
    \$\begingroup\$ That's the kind of question which is a bit outside the box. It's very sensible to ask it. I am sure that one day cameras will work in a very similar way to your suggestion. ... Although personally, I find that technical limitations feed the artistic. \$\endgroup\$
    – AJ Finch
    Commented Jan 10, 2012 at 16:16

8 Answers 8

20
\$\begingroup\$

Who decides which pixels get how much gain? Much of what goes on in the human visual system happens in the cortex, not the eye, and depends on what we think is important to see based on a combination of intellectual decision and the (somewhat override-able) instinctual drive for self-preservation. While it's true in one sense that we see what's there, it is equally true in another sense that we see what we want to (or need to) see.

It would be almost trivial to create a relatively low pixel density sensor with large photosites that allow for an enormous dynamic range and (assuming a CCD-type technology, since the current CMOS sensor tech can't work this way) a per-pixel electronic shutter in addition to the mechanical shutter. So what would that get you? A flat image with a lot of bit depth and very low local contrast (if the entire bit depth is converted as-is for display or print) along with a number of pixels that are almost, but not quite, clipped by the sensor saturation (although they are, in fact, clipped by the limiting action of the electronic shutter just before the point of saturation). Let's say for the sake of argument, though, that this sensor and its associated computer could record the clipping data (the reason why it stopped recording at that sensel, which could be as simple as recording the actual exposure duration at that site). That would allow the camera's electronics to reconstruct what the numbers would have been if the photosite could have stayed in the game until the final whistle. So now we have an even flatter image with greater bit depth. And where do you draw the line? 32 bits? 64?

Now comes the hard part -- turning this flat, high-dynamic-range image data into a compelling photograph. The simplest approach is to take the eight bits (or whatever the output bit depth would be) that represent the primary metered image and throw away the rest. It would probably be not much more difficult to fit the data to an S-curve, compressing the extreme shadows and/or highlights -- which is more or less what the extended dynamic range settings on newer cameras already do. But there are only so many output bits available per pixel, and most of the extended highlight values are going to round up to white (or at least a 254 and 255 mix). So you've gained very little by dramatically complicating the system.

But there is still one option open -- selective area mapping. Why not bring the sky, say, or just the clouds in that sky, down in value so it can retain detail, while preserving the desired contrast in the foreground? This is where the hard problem lives. What's important? Should the camera decide for you? If the camera decides, then we have a big advance in machine vision and artificial intelligence to get around to first. If not, then do you really want to make this level of post-capture decision for every picture you shoot? Yes, I know there will be some photo-techno-weinies who really do want to be that hands-on, but can we accept that it's a pathological condition, and that professionals interested in turn-around time and the vast majority of consumers aren't like that?

So you need a new sensor, vastly more complicated electronics around the sensor, an enormous image file for projected raw data (which necessitates larger cards and longer write times/slower frame rates), all to gather data that is going to be thrown away most of the time so that you can occasionally shoot one-shot HDR images that require a lot of human intervention in post (or a huge leap in MV/AI). You could probably sell a few of these, but I'd expect the market to look an awful lot more like the medium format market than the existing 35mm/APS-C market. That is, you'd sell to a select group of well-heeled photographers who either actually need the capabilities for professional reasons or to fulfill their fine art vision, and a few who just get a big enough kick out of post-processing to pay the technology tax.

\$\endgroup\$
8
  • 1
    \$\begingroup\$ This is a great analysis (and I'm voting it up as such). But in the end I find the conclusions unconvincing. After all, it wouldn't be hard to create a "dumb" matrix of sensels, some of which are masked by, say, a six-stop ND filter. (Use one of the G's in the Bayer filter for this, for instance.) No significant change in electronics would be needed; nothing about the cameras, lenses, etc., would change; but in return for a tiny loss in sensitivity you would obtain the ability to distinguish (coarse) gradations within blown-out pixels. That's close to what the OP asks for. \$\endgroup\$
    – whuber
    Commented Jan 3, 2012 at 20:44
  • 2
    \$\begingroup\$ A better simple approach already exists in Fuji's EXR sensor. It has different-sized sensels; a small sensel and a large one are paired at each source pixel site on the sensor. While that can provide for a wider capture range, it's still limited in what it can do -- you can still blow out the highlights if you want to get all of the shadows. (And Fuji's current implementation uses too many pixels on too small a sensor.) Assume a perfect implementation with multiple sensor sizes -- that still leaves the processing problem—and that, not the capture, is the real issue. \$\endgroup\$
    – user2719
    Commented Jan 3, 2012 at 20:54
  • \$\begingroup\$ Thanks for the EXR reference. It shows my suggestion is not just plausible, but real. Now hasn't HDR software addressed the processing problem? If you have managed to capture a wide dynamic range with an integrated sensor, then in effect you already have a perfectly registered HDR stack. The limitations of this technology will be the limitations of HDR, to be sure. \$\endgroup\$
    – whuber
    Commented Jan 3, 2012 at 21:42
  • \$\begingroup\$ Again, straight processing will give a crappy image if the source is HDR. The limitation is not in the capture, but in the processing—how many haloed images do you need to see before you come to the conclusion that HDR processing isn't simple? Well-meaning people screw it up royally, and they know what they want the image to "say". Then the storage and write speed problems are still there, too, and there's another use compromise. A camera can be all things to all people, but at what cost in both price and ergonomics? 14-bit capture and DR compression are here now; 16-bit won't be far behind. \$\endgroup\$
    – user2719
    Commented Jan 3, 2012 at 22:00
  • 1
    \$\begingroup\$ ... And both of them still get blown highlights. You need to extend substantially further to avoid the issue altogether, and at some point the practicality breaks down, at least at the mass-market level. Studio shooters don't need it; most consumers don't need it. I do mostly environmental portraiture, and I can't think of too many times I needed it (overexposure has made many of my images). The overhead just wouldn't be worth it to most people, so that's going to create an expensive niche product. \$\endgroup\$
    – user2719
    Commented Jan 3, 2012 at 22:10
4
\$\begingroup\$

There's one thing that only a few people mention, and that's that a scene would not look the same if different areas where being exposed differently to other areas. A scene looks the way it does because there is variation. Yes, it could be a way to save highlights and boost shadows, but in the end, what you really want is a larger dynamic range that can capture the dynamic range in the scene using one exposure setting.

Our eyes are great at providing us a far greater dynamic range than current consumer camera technology. I can look around quickly and perceive accurate detail in shadowed areas and bright sunlit areas at the same time.

One of the ways camera manufacturers are getting around this problem is by using both high and low sensitivity pixels in the one sensor then combining the result per pixel. RED's newer digital cinema cameras have a matrix of normal and low sensitivity sensor pixels called HDRx. The small, low sensitivity sensor pixels are combined into the highlights of bright pixels to increase the dynamic range.

\$\endgroup\$
1
  • \$\begingroup\$ You are correct, it would result in a bland photo but obviously it would have to have some limitations. I believe that letting camera decide on which areas should be brighter and darker would be too much. I wasn't thinking right... \$\endgroup\$
    – user7264
    Commented Jan 4, 2012 at 14:30
3
\$\begingroup\$

First, human eye dynamic range is not that great. It only appears to be better than our current high end cameras, because our brain constantly merges "snapshots" taken using different exposure settings. Our eyes cannot register extremely bright and extremely dark objects simultaneously (even though the brain can). True wonder of image processing, but only mediocre optics/imaging device.

There are several proposals/prototypes showing how the dynamic range of image sensors could be greatly improved:

\$\endgroup\$
2
\$\begingroup\$

You are missing some basic physics here. The main problem is that real scenes have large contrast ratios. Our eyes have evolved to deal with that by perceiving light levels logarithmically instead of linearly. Unfortunately, the current sensor technology inherently measures light linearly. Or put more precisely, the noise is fixed on a linear light scale.

With current technology, the maximum contrast limit is basically a function of the noise level. For sake of argument, let's use a 0-1000 light scale, meaning a sensor can tell you the light level from 0 to 1000. What's the highest ratio it can therefore measure? It depends on the noise level. The noise level is basically what you get instead of true black, which would be 0 in this example. Roughly, if the noise level is 2, then you get about 1000:2 = 500:1 brightness ratio. As long as the scene doesn't exceed that (almost all would though, in reality 500:1 isn't that much), you can do whatever logarithmic mapping you want later.

So the current strategy, given that the current sensors are inherently linear, is to try to increase the signal to noise ratio, and to provide enough bits so that quantization noise is below the inherent random noise. The lower noise the sensor, the wider dynamic range scenes you can capture without either clipping the highlights or mudding out the shadows.

There is a totally different sensor technology that does inherently measure the log of the brightness. Sometimes these are called "CMOS" sensors, because they are a lot like CMOS dynamic RAMs with the lid taken off (I oversimplify, but the first test in the lab was actually done this way). You get a voltage proportional to the log of the light, but these currently have much lower signal to noise ratios. Mitsubishi was the first to commercialize these sensors, but they are nowhere near good enough yet for high end cameras.

There are no doubt going to be advances on multiple fronts, and I'm sure we're going to be seeing steady progress for years yet to come. However, there are good reasons why things are as they are now, not just because nobody can imagine something better. If someone had a technology that could measure a wide dynamic range accurately and at a price people are willing to pay for, they'd be out there getting rich.

\$\endgroup\$
1
\$\begingroup\$

I believe that it's just too complicated.

There basically would be two possible approaches; either each photosensor could keep track of the time and turn itself off, or the CPU could keep track of the data from the photosensors and turn them off.

For the first approach it would mean that each photosensor would need a clock signal and a counter, so that it could keep track of how long time it took until it shut itself off. That's a lot more circuitry to fit on the chip, and a lot more electricity needed to run it, which increases the signal noise. Probably so much that the increased dynamic range would be pointless.

For the second approach the CPU would need to read all the data from the sensor about once a 1/10000 second. That is about 1000 times faster than the current technology can accomplish, so that's decades into the future, if at all possible.

Also, there are other complications with such a solution, like that each pixel would get a different exposure time. You would get pretty strange artifacts if you photograph anything that moves.

\$\endgroup\$
1
  • 1
    \$\begingroup\$ I am not that convinced that the "selective motion blur" would be such a big problem. When shooting a moving subject, the pixels that would be exposed the longest would be those that stayed in the dark the whole time. These would be inside mostly uniformly dark areas, where some blur would not be very noticeable. Pixels on edges between dark and light area, in the case the subject was moving, would get higher exposure, and would shut down faster, resulting in less blur. \$\endgroup\$ Commented Jan 3, 2012 at 21:43
0
\$\begingroup\$

While it is true that DSLRs are digital, lenses are not. All the cell sensors will be subject to the same aperture no matter how smart the DSLR body becomes, because aperture is set at the lens. So I think varying the aperture per sensor cell is out, at least with current lens technology.

As far as shutter speed, that is controlled by the camera, but if we imagine a camera that can vary the shutter speed on different parts of the picture to control over/under exposure you will end up with uneven motion blur. The darker parts of the scene will have to be exposed longer and will be more blurry than the brighter parts. I think a solution that changes shutter speed will not work for this reason.

So the only thing left is ISO. Varying the ISO would mean different noise levels in different parts of the picture. This does not sound too bad, considering that you would get a much increased dynamic range in return. I don't know much about how sensors work, but I would imagine that the ISO setting is implemented in sensors as a sort of "tuning" towards a specific subset of the brightness scale. It sounds to me it would be prohibitively expensive to have independent metering and ISO control at every sensor cell, but maybe the picture can be divided up in areas, and each area metered separately. Then the camera will have to have some sort of algorithm to blend the differently exposed areas, sort of what "enblend" does when it assembles a panorama where each picture has a different exposure. This sounds doable to me.

Another option would be to have a camera with multiple sensors, each configured to a different ISO. In video technology there are 3 CCD cameras, where each CCD records one of red, green and blue. I don't see why there couldn't be a similar concept for DSLRs, where multiple sensors take the picture at different ISO levels, producing an HDR picture.

\$\endgroup\$
0
\$\begingroup\$

I cannot find the information at the moment, but I seem to recall reading a description of a similar technology. The idea was roughly this: the only thing that needs to be taken care of is overblown highlights. If you can prevent those, dark areas can be taken care by increased exposure to the whole picture. So, in order to prevent overblown highlights, each sensor will keep track of its accumulated light, and if that gets close to maximum, the sensor will shut off. That by itself would not improve anything, it would actually make matters worse, instead of having few bright white overblown highlights, one would end up with even more slightly darker highlights. So instead of just shutting off, the cell would also shut off cells in some neighborhood, which would preserve some detail in bright areas. The problem with that was that it created light halos around bright objects on dark background.

As I wrote, I cannot find the text now, but somehow it is in my mind associated with HP digital cameras.

\$\endgroup\$
1
  • \$\begingroup\$ If you do find it, please link to the article. An interesting idea indeed. \$\endgroup\$
    – user7264
    Commented Jan 4, 2012 at 14:27
0
\$\begingroup\$

It can be done mathematically (theoretically): http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.64.9692&rep=rep1&type=pdf

\$\endgroup\$
1
  • 1
    \$\begingroup\$ Could you summarize that article? Nice to have a link, but would be helpful if you described what it's about briefly \$\endgroup\$
    – MikeW
    Commented Aug 10, 2013 at 23:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.