Could a "universal exposure" setting be practically possible?

Question

Not sure how uneducated this question is, but I'm interested in learning, so thanks in advance for your indulgence.

Film physically changes over the period of time it's exposed. A digital sensor, though, doesn't; it's just reading data. Is there any reason the camera couldn't be made to "remember" what the sensor readings were at every exposure point? It's just data. It might be a lot of data, but there are times one might want to do that, no? Giving far more flexibility in post processing.

If data storage were not an issue, is there any reason this couldn't be the norm, at least for professional and art photography?

Answer 1

A digital sensor isn't really best described as "reading data". A much better way to describe it is "collecting photons" that are then converted into data by measuring the microscopic electrical charges they produce once the collection period is over. They do not have the capability to continuously record the changing state of each pixel well as they collect light. And depending on how little or how much light is falling on the sensor it might take a long time for enough photons to strike the sensor before anything more than random data is generated. On the other hand, in very bright light sometimes all of the pixel wells can fill up so fast that any additional photons falling on the sensor are lost.

In the first scenario not enough photons are collected to create a discernable pattern through the "noise" generated by the energy flowing through the sensor that is used to collect the voltages created by the photons falling into the pixel wells. Thus no usable information is collected. Your entire photo is dark with random spots of color and light.

In the second scenario so many photons are collected that every pixel is read out at the same maximum value, called full saturation, and since every pixel in the image has the same value no usable information has been preserved. Your entire photo is solid bright white.

It is only when enough photons strike a sensor that the areas with more photons per unit of time have a higher readout value than areas with fewer photons striking them per unit of time. Only then has the sensor collected meaningful information that can differentiate between areas of varying brightness.

Imagine setting out a number of water buckets in your yard to collect raindrops. Imagine that they all have some water in them but you dump it out before you place them. Some are placed under the eaves of your house's roof. Some are placed under large trees in your yard. Some are placed out the open. Some are placed under the spout that dumps the water from your gutters into the yard. Then it starts to rain.

Let's say that it only rains for a very short time: 15 seconds. There are a few drops of water in each bucket. But there's not enough water in each bucket to be able to tell if each bucket may have had more rainwater fall in it or if it may have just had a few more drops left in the bucket when you dumped the water out before you put the buckets in the yard. Since you don't have enough data to be able to determine how much rain fell on which parts of the yard, you dump all of the buckets out and wait for it to rain again.

This time it rains for several days. By the time it stops raining every bucket in the yard is overflowing. Even though you are fairly certain some buckets filled up faster than other buckets, you have no way of knowing which buckets filled up quickest and which buckets filled up last. So you need to dump the buckets out again and wait for more rain.

On your third attempt it rains for three hours and then stops raining. You go out to the yard and inspect your buckets. Some are almost full! Some have barely any water in them at all! Most have varying amounts of water in between the two extremes. Now you can use the location of each bucket to determine how much rain fell on each area of your yard.

The reason we alter exposure in digital cameras is to attempt to collect enough light that the brightest areas are almost, but not quite, saturated. Ideally this occurs with the camera at base ISO sensitivity. Sometimes, though, there isn't enough light to do this. Even at the largest aperture available we can't collect enough light in the longest amount of time we dare leave the shutter open (due to motion of our subjects). What we do in this case is adjust the ISO setting in our camera so that all of the values coming off the sensor are multiplied at a factor that brings the highest values to a point where they are almost, but not quite saturated. Unfortunately, when we amplify the signal (the voltages created by photons landing in pixels wells) we also amplify the noise (the random uneven voltages produced by the current used to collect the voltages from each pixel well). This results in a lower signal-to-noise ratio which decreases the amount of detail we can create from the data we have collected from the sensor.

There are other technical limitations that prevent cameras from keeping a "running total" of the number of photons collected at various intervals while the shutter is open. Throw enough money at the problem and some of those limitations can be overcome, at least partially. But either the laws of physics would need to change or we need to completely change the way sensors count photons before others of those limitations could be overcome. Eventually the technology in some or all of these devices might replace the way we currently capture very high quality images, but we're nowhere near there yet.

Answer 2

We already have some of the technology for this. Our term for remembering the sensor readings at each exposure point is "video", and what you are asking for is reconstruction of an optimal still image from multiple video frames.

For an overview of Microsoft Research work on this, start here: http://research.microsoft.com/en-us/um/redmond/groups/ivm/multiimagefusion/

For an available example, see the Synthcam app, which can be used for reducing noise in low light by combining video frames taken with a phone camera: https://sites.google.com/site/marclevoy/

This is a long way from practical for everyday photography, but it's conceivable that future cameras will shoot many frames of high-definition, high frame-rate video allowing the photographer to achieve their desired result by selecting and combining later.

Late 2016 update: When I wrote the original answer, this was some way from the market. In late 2016 it seems a lot closer. Marc Levoy's "See In The Dark" app integrates multiple video frames with stabilisation on a consumer smartphone to produce usable images from moonlight. See also the Light L16 camera, which integrates multiple small sensors into a single image.

Answer 3

The original question is based on incorrect assumption (about digital sensor not changing state during the exposure) but the concept is related to the Quanta Image Sensor (QIS) idea researched by Eric Fossum.

http://engineering.dartmouth.edu/research/advanced-image-sensors-and-camera-systems/

The QIS is a revolutionary change in the way we collect images in a camera that is being invented at Dartmouth. In the QIS, the goal is to count every photon that strikes the image sensor, and to provide resolution of 1 billion or more specialized photoelements (called jots) per sensor, and to read out jot bit planes hundreds or thousands of times per second resulting in terabits/sec of data.

Such device would (quoting the question)

"remember" what the sensor readings were at every exposure point

and having the complete data set we could for example "change" the effective exposure time after the "photograph" was captured.

Today this can approximated by recording a video and combining frames in postprocess to simulate longer exposure times (limited by camera performance, video mode resolution and shutter speed, but it shows the idea)

If the QIS works as promised, it would also introduce other cool features, like better low light performance, increased dynamic range, no aliasing, completely customizable sensitivity (e.g. film-like), no ISO settings, adjustable resolution vs noise

Recent announcement: http://phys.org/news/2015-09-breakthrough-photography.html

Answer 4

Film physically changes over the period of time it's exposed. A digital sensor, though, doesn't; it's just reading data.

That really depends on the type of sensor. The kind of CMOS sensors that are used in today's DSLR's accumulate an electrical charge in each pixel over time, so they do, in fact, change over time much like film does. If they didn't work that way, the image would exist only for as long as the shutter was open. CCD sensors (the other common technology for image sensors in cameras) also work this way, accumulating light over time.

Is there any reason the camera couldn't be made to "remember" what the sensor readings were at every exposure point?

That's exactly what the camera does when it records an image. I think that what you mean, though, is that if the sensor could read the instantaneous light intensity, then you could adjust the exposure after the fact to whatever value you want. As explained above, that's not really how most image sensors work. On the other hand, we can and often do adjust exposure quite a bit in post-processing.

If data storage were not an issue, is there any reason this couldn't be the norm, at least for professional and art photography?

As far as "remembering" the data from the sensor, it is the norm for many photographers. Most cameras let you record images in "RAW" format, and this is pretty much the data as it's read from the sensor plus a bit more data about what the camera settings were at the time. RAW images take up a lot more space than other formats like JPEG, but they give the photographer the freedom to re-interpret the data later, so you can easily change settings like color temperature and white balance in post-processing.

Answer 5

Others have already explained why this won't work, technically. I want to touch on why it wouldn't work practically.

If data storage were not an issue, is there any reason this couldn't be the norm, at least for professional and art photography?

Consider the magnitude of different lighting conditions that we may want to take photographs of. Even ignoring extremes such as astrophotography (where you are often photographing small speckles of light surrounded by nearly total black), you still have evening or night terrestrial photography, and brightly lit snow-covered winter landscapes. I'm going to use the latter two as examples.

Also, I'm going to assume that in order to accurately recreate any desired exposure, we have to expose the sensor to the point of full saturation.

Also, I'm going to assume that we can read the sensor values in a non-destructive fashion. (This is probably one of those problems that fall into the category of "throw enough money at the problem and it might be solvable".)

In the night photography case, we would need to expose the sensor for a very long time to saturate all pixels, which means that any photo, no matter what we actually want a picture of, is going to take absurdly long to take. The classic tourist picture of dancers at an outdoors bar becomes nearly impossible because, well, you might be able to snap a few of those during an entire evening. Not good. So we can't expose to saturation, at least not indiscriminately. (Exposing to some percentage of pixels being saturated is equally useless, but for different reasons; try getting the exposure exactly right when taking a photograph of a fireplace with a fire burning in it. That's almost impossible; no matter how hard you try, some pixels will be overblown or huge swaths of the image will be horribly underexposed.)

When photographing a brightly lit snow-covered landscape, such as a winter vista during daytime when the sun is out, the exposure that the camera's automatic exposure system aims for ("18% gray") is woefully inadequate. This is why you often see photos of snow that are dark, and where the snow appears more of a light gray than white. Because of this, we often use a positive exposure compensation setting that results in the snow being exposed as a nearly saturated white. However, this means that we can't rely on the camera's AE system to determine when to end the exposure: if we do, such pictures will invariably be underexposed.

In other words, exposure to full saturation is impractical in many cases, and exposure to make the AE system happy is inadequate in many cases. This means that the photographer will still have to make some sort of choice, and at that point, we are at least just as well off staying with what we have and photographers are used to, making the AE systems better and giving the photographer easy (easier?) access to exposure compensation settings. By increasing the sensor's practically usable dynamic range, we can allow (even) greater latitude in exposure changes in post-processing; the original digital SLRs were horrendeously expensive, yet truly horrible in this regard compared to today's even entry-level models.

All of which can be done fully within the framework of what we already have. This is not to say that dramatically improving the usable dynamic range of the sensor is easy, but it's probably a lot easier than what you are proposing, and it's a problem vendors have experience working on.

Professionals, almost by definition, know how to use the equipment of their trade. It isn't really any different if they are photographers or space shuttle pilots. Especially when it can be done without causing information overload, it's usually better to give the user full control of professional equipment. In my opinion, current high-end DSLRs are pretty good about hitting the sweet spot on this.

Answer 6

Let's simplify the problem to understand why we will always have to make compromises.

Let's invent the camera you want, but with only one monochrome pixel. It needs to be able to reliably receive and notify the processor of the reception of a single photon. It also needs to be able to receive and notify the processor of the reception of, practically speaking, uncountably infinite photons.

The first case in a situation where there's no light. The second in the case of even a moderate amount of light.

The main issue is that we simply don't have the technology to create a sensor with such a wide dynamic range. We're always going to have to compromise, and right now we're compromising by selecting a higher range where the sensor can accept nearly infinite photons and give us a reading that suggests a relative amount of light hitting the sensor. It doesn't count them at all, but acts like our eyes do - they merely give an output that's relative to the amount of photons hitting them, without attempting to count photons.

This is further complicated by the fact that this is collected over time.

An ideal sensor would actually be more like a geiger counter - measuring the time between photons to give us a nearly instantaneous measurement of the amount of light falling on the sensor, assuming that the photons are relatively evenly spaced (which isn't true, but is a convenient assumption, and why geiger counters average over time just like cameras do).

Quantum sensors would have essentially the same problem. Sure, they can sense an individual photon, but at some point they're coming fast enough that you simply can't measure the time between them, or even count how many are coming per exposure period.

So we have this compromise that requires we either take several images of several exposures, or add multiple images of the same high exposure together to tease out the low light areas, or split the incoming light into two or more paths with different sensors of different dynamic range, or build sensors that can group pixels together or stack light sensors, or, or, or - there are literally thousands of ways photographers have overcome this basic problem over the decades with a wide variety of media.

It's a physics limitation that isn't likely to be overcome. We're not ever going to have a camera* with no input from the photographer that allows all the decisions to be made in post processing.

*_{Of course, if you change the definition of camera, then you might be satisfied with some other process's results, but this is largely subjective. The reality is that if you image a scene with your camera, then show the scene to a person, then the image you shot, they will perceive differences due to inherent differences between their eyes, your image sensor, and the process you used to print the image. Photography is as much about interpretation and art as it is about capturing light, and so a fanatical focus on the "perfect camera" is probably not very useful.}