Spring 2012

Spring 2012
by ani

Submit your Photo
Hall of Fame

Please participate in Meta
and help us grow.

Photography Stack Exchange is a question and answer site for professional, enthusiast and amateur photographers. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Not sure how uneducated this question is, but I'm interested in learning, so thanks in advance for your indulgence.

Film physically changes over the period of time it's exposed. A digital sensor, though, doesn't; it's just reading data. Is there any reason the camera couldn't be made to "remember" what the sensor readings were at every exposure point? It's just data. It might be a lot of data, but there are times one might want to do that, no? Giving far more flexibility in post processing.

If data storage were not an issue, is there any reason this couldn't be the norm, at least for professional and art photography?

share|improve this question
    
Intriguingly, that is almost exactly what our eyes do. HDR is designed to try to reconstruct the algorithms done within our retina cells and the visual cortex, and emulate them as best as we can. – Cort Ammon Jan 11 at 18:59
    
@CortAmmon And even then, most advanced simple eyes have the additional feature of the iris, which expands or contracts to allow more or less light through the pupil as an additional component in intensity regulation. (Of course, eyes do not adjust immediately either, as easily shown when going from an area of bright light to one with little light or, more painfully, vice versa.) – JAB Jan 11 at 20:52
    
A digital sensor is also physically changing as it's exposed to light (discussions about "physical" vs. "chemical" vs. "electrical" can be made, but I can argue they're same thing). A "perfect" camera would record the position/time/angle of each photon; fairly difficult (truly perfect is impossible because of Heisenberg) unless you make tradeoffs: dealing with extremely few photons, in a specific band (two-photon excitation, neutrino detectors) – Nick T Jan 11 at 23:28
    
@Cort Ammom I think you are on the correct track with mention of HDR, that uses the concept of multiple exposure settings integrated back to a single high definition picture. To get that functionality in real time without any need for a change in settings, I think would require to have a camera that has multiple lenses -- perhaps 5 would suffice for decent HDR level pictures. – A.Danischewski Jan 12 at 6:57
    
Is it just what our eyes do, Cort? My understanding is that our eyes scan a scene in front of us, adjusting focus and exposure (very quickly, but far from instantaneously) as they move. And that our brains constitute a picture from the result. HDR is closer - effectively exposure-bracketing the scene and then combining the best-exposed parts. In fact, very close - in a high dynamic range situation, the eye probably is often temporarily exposing badly, but we "forget" those moments in constituting our picture of the scene. – Wombat Pete Mar 12 at 15:24

A digital sensor isn't really best described as "reading data". A much better way to describe it is "collecting photons" that are then converted into data by measuring the microscopic electrical charges they produce once the collection period is over. They do not have the capability to continuously record the changing state of each pixel well as they collect light. And depending on how little or how much light is falling on the sensor it might take a long time for enough photons to strike the sensor before anything more than random data is generated. On the other hand, in very bright light sometimes all of the pixel wells can fill up so fast that any additional photons falling on the sensor are lost.

In the first scenario not enough photons are collected to create a discernable pattern through the "noise" generated by the energy flowing through the sensor that is used to collect the voltages created by the photons falling into the pixel wells. Thus no usable information is collected. Your entire photo is dark with random spots of color and light.

In the second scenario so many photons are collected that every pixel is read out at the same maximum value, called full saturation, and since every pixel in the image has the same value no usable information has been preserved. Your entire photo is solid bright white.

It is only when enough photons strike a sensor that the areas with more photons per unit of time have a higher readout value than areas with fewer photons striking them per unit of time. Only then has the sensor collected meaningful information that can differentiate between areas of varying brightness.

Imagine setting out a number of water buckets in your yard to collect raindrops. Imagine that they all have some water in them but you dump it out before you place them. Some are placed under the eaves of your house's roof. Some are placed under large trees in your yard. Some are placed out the open. Some are placed under the spout that dumps the water from your gutters into the yard. Then it starts to rain.

Let's say that it only rains for a very short time: 15 seconds. There are a few drops of water in each bucket. But there's not enough water in each bucket to be able to tell if each bucket may have had more rainwater fall in it or if it may have just had a few more drops left in the bucket when you dumped the water out before you put the buckets in the yard. Since you don't have enough data to be able to determine how much rain fell on which parts of the yard, you dump all of the buckets out and wait for it to rain again.

This time it rains for several days. By the time it stops raining every bucket in the yard is overflowing. Even though you are fairly certain some buckets filled up faster than other buckets, you have no way of knowing which buckets filled up quickest and which buckets filled up last. So you need to dump the buckets out again and wait for more rain.

On your third attempt it rains for three hours and then stops raining. You go out to the yard and inspect your buckets. Some are almost full! Some have barely any water in them at all! Most have varying amounts of water in between the two extremes. Now you can use the location of each bucket to determine how much rain fell on each area of your yard.

The reason we alter exposure in digital cameras is to attempt to collect enough light that the brightest areas are almost, but not quite, saturated. Ideally this occurs with the camera at base ISO sensitivity. Sometimes, though, there isn't enough light to do this. Even at the largest aperture available we can't collect enough light in the longest amount of time we dare leave the shutter open (due to motion of our subjects). What we do in this case is adjust the ISO setting in our camera so that all of the values coming off the sensor are multiplied at a factor that brings the highest values to a point where they are almost, but not quite saturated. Unfortunately, when we amplify the signal (the voltages created by photons landing in pixels wells) we also amplify the noise (the random uneven voltages produced by the current used to collect the voltages from each pixel well). This results in a lower signal-to-noise ratio which decreases the amount of detail we can create from the data we have collected from the sensor.

There are other technical limitations that prevent cameras from keeping a "running total" of the number of photons collected at various intervals while the shutter is open. Throw enough money at the problem and some of those limitations can be overcome, at least partially. But either the laws of physics would need to change or we need to completely change the way sensors count photons before others of those limitations could be overcome. Eventually the technology in some or all of these devices might replace the way we currently capture very high quality images, but we're nowhere near there yet.

share|improve this answer
2  
It's not a quote. It is my own illustration. The block quote formatting was used to set it apart from the rest of the answer. I chose to do it that way because I felt that, among the formatting options available, it was the best choice. – Michael Clark Jan 11 at 15:13
1  
One thing to note is that we could potentially capture shorter exposures from a longer exposure, if we develop sensors that we can poll fast enough - eg we could use a 1 second exposure but capture 1/250, 1/125, 1/60, 1/30 etc from that by reading the sensor repeatedly during that time and keeping a "running total" of sorts. We can already kind of do this, but it's not very well developed and has limits to how short an exposure we can capture – Jon Story Jan 11 at 15:51
1  
Readout takes too long on high res sensors. And many shots don't stay put for one second. Sometimes 1/1000 isn't fast enough. – Michael Clark Jan 11 at 15:59
2  
That's nothing more than glorified bracketing. Which is hard to do when you're shooting world class athletes at 10+ fps. With the latitude raw files currently give us any photographer worth her salt should be able to expose well enough to get the desired result. And remember, with a focal plane shutter any Tv shorter than sync speed still takes the same amount of time as sync speed for the two curtains to transit the sensor. And then there's the readout time needed before then sensor can be cleared for the next frame. – Michael Clark Jan 11 at 16:41
1  
As we pointed out in our answer, there are some ways around some of the physical limitations, but they are expensive to execute and leave a lot to be desired for certain uses. They are certainly not yet practical, which is what the question addresses. – Michael Clark Jan 11 at 16:43

We already have some of the technology for this. Our term for remembering the sensor readings at each exposure point is "video", and what you are asking for is reconstruction of an optimal still image from multiple video frames.

For an overview of Microsoft Research work on this, start here: http://research.microsoft.com/en-us/um/redmond/groups/ivm/multiimagefusion/

For an available example, see the Synthcam app, which can be used for reducing noise in low light by combining video frames taken with a phone camera: https://sites.google.com/site/marclevoy/

This is a long way from practical for everyday photography, but it's conceivable that future cameras will shoot many frames of high-definition, high frame-rate video allowing the photographer to achieve their desired result by selecting and combining later.

share|improve this answer
1  
Which kind of falls into the "Throw enough money at the problem" category. Imagine what 4,000 fps at 50MP would cost! – Michael Clark Jan 11 at 12:45
    
I worked on 1000fps about five years ago, and it weighed a few kg. I'm currently carrying a 120fps camera in my pocket. This might be practical in 5 to 10 years. Certainly not for mainstream photography now, and you can always find a dark situation where there just aren't enough photons for anything to work with. – Adrian Cox Jan 11 at 12:51

The original question is based on incorrect assumption (about digital sensor not changing state during the exposure) but the concept is related to the Quanta Image Sensor (QIS) idea researched by Eric Fossum.

http://engineering.dartmouth.edu/research/advanced-image-sensors-and-camera-systems/

The QIS is a revolutionary change in the way we collect images in a camera that is being invented at Dartmouth. In the QIS, the goal is to count every photon that strikes the image sensor, and to provide resolution of 1 billion or more specialized photoelements (called jots) per sensor, and to read out jot bit planes hundreds or thousands of times per second resulting in terabits/sec of data.

Such device would (quoting the question)

"remember" what the sensor readings were at every exposure point

and having the complete data set we could for example "change" the effective exposure time after the "photograph" was captured.

Today this can approximated by recording a video and combining frames in postprocess to simulate longer exposure times (limited by camera performance, video mode resolution and shutter speed, but it shows the idea)

If the QIS works as promised, it would also introduce other cool features, like better low light performance, increased dynamic range, no aliasing, completely customizable sensitivity (e.g. film-like), no ISO settings, adjustable resolution vs noise

Recent announcement: http://phys.org/news/2015-09-breakthrough-photography.html

share|improve this answer
    
"If the QIS works as promised, it would also introduce other cool features, like better low light performance, increased dynamic range, no aliasing, completely customizable sensitivity (e.g. film-like), no ISO settings, adjustable resolution vs noise" - That's what I was thinking, it's exactly why I asked the question in the first place! – Wombat Pete Jan 11 at 21:41
1  
Terabits per second? Being generous, that's 1 GB for a 1/125 second exposure, or 2 GB for a 1/60 second exposure. (You think a 25-50 MB RAW file from a reasonably high-end DSLR, mosty irrespective of exposure time, is big?) At 1000 MB/s write performance, which is quite a ways up there, you are about 100x too slow. I think at these levels, storage is a very real consideration. – Michael Kjörling Jan 12 at 8:54

Film physically changes over the period of time it's exposed. A digital sensor, though, doesn't; it's just reading data.

That really depends on the type of sensor. The kind of CMOS sensors that are used in today's DSLR's accumulate an electrical charge in each pixel over time, so they do, in fact, change over time much like film does. If they didn't work that way, the image would exist only for as long as the shutter was open. CCD sensors (the other common technology for image sensors in cameras) also work this way, accumulating light over time.

Is there any reason the camera couldn't be made to "remember" what the sensor readings were at every exposure point?

That's exactly what the camera does when it records an image. I think that what you mean, though, is that if the sensor could read the instantaneous light intensity, then you could adjust the exposure after the fact to whatever value you want. As explained above, that's not really how most image sensors work. On the other hand, we can and often do adjust exposure quite a bit in post-processing.

If data storage were not an issue, is there any reason this couldn't be the norm, at least for professional and art photography?

As far as "remembering" the data from the sensor, it is the norm for many photographers. Most cameras let you record images in "RAW" format, and this is pretty much the data as it's read from the sensor plus a bit more data about what the camera settings were at the time. RAW images take up a lot more space than other formats like JPEG, but they give the photographer the freedom to re-interpret the data later, so you can easily change settings like color temperature and white balance in post-processing.

share|improve this answer
2  
I'd say your answer is the only one that contains the key point: Sensors don't produce a continuous readout which would be required for what the OP desired. If one wanted continuous exposure, one would require continuous data output at very small time steps (smaller than "Live View" and similar). – DetlevCM Jan 11 at 11:42
2  
@DetlevCM Have you carefully read my answer? It includes that information in both the first and last paragraphs. – Michael Clark Jan 11 at 12:16
1  
@MichaelClark Indeed - it does contain it when I re-read it now (no edits recorded). So I guess it got lost in noise when I looked over the answers... – DetlevCM Jan 11 at 12:20
    
I've edited it to make it even more clear. – Michael Clark Jan 11 at 12:20
    
"If one wanted continuous exposure, one would require continuous data output at very small time steps (smaller than "Live View" and similar)" ...which is the "glorified bracketing" notion in the discussion above, right? The new question, as I'm coming to understand this, goes to practicality/availability of this kind of thing. – Wombat Pete Jan 11 at 21:39

Others have already explained why this won't work, technically. I want to touch on why it wouldn't work practically.

If data storage were not an issue, is there any reason this couldn't be the norm, at least for professional and art photography?

Consider the magnitude of different lighting conditions that we may want to take photographs of. Even ignoring extremes such as astrophotography (where you are often photographing small speckles of light surrounded by nearly total black), you still have evening or night terrestrial photography, and brightly lit snow-covered winter landscapes. I'm going to use the latter two as examples.

Also, I'm going to assume that in order to accurately recreate any desired exposure, we have to expose the sensor to the point of full saturation.

Also, I'm going to assume that we can read the sensor values in a non-destructive fashion. (This is probably one of those problems that fall into the category of "throw enough money at the problem and it might be solvable".)

In the night photography case, we would need to expose the sensor for a very long time to saturate all pixels, which means that any photo, no matter what we actually want a picture of, is going to take absurdly long to take. The classic tourist picture of dancers at an outdoors bar becomes nearly impossible because, well, you might be able to snap a few of those during an entire evening. Not good. So we can't expose to saturation, at least not indiscriminately. (Exposing to some percentage of pixels being saturated is equally useless, but for different reasons; try getting the exposure exactly right when taking a photograph of a fireplace with a fire burning in it. That's almost impossible; no matter how hard you try, some pixels will be overblown or huge swaths of the image will be horribly underexposed.)

When photographing a brightly lit snow-covered landscape, such as a winter vista during daytime when the sun is out, the exposure that the camera's automatic exposure system aims for ("18% gray") is woefully inadequate. This is why you often see photos of snow that are dark, and where the snow appears more of a light gray than white. Because of this, we often use a positive exposure compensation setting that results in the snow being exposed as a nearly saturated white. However, this means that we can't rely on the camera's AE system to determine when to end the exposure: if we do, such pictures will invariably be underexposed.

In other words, exposure to full saturation is impractical in many cases, and exposure to make the AE system happy is inadequate in many cases. This means that the photographer will still have to make some sort of choice, and at that point, we are at least just as well off staying with what we have and photographers are used to, making the AE systems better and giving the photographer easy (easier?) access to exposure compensation settings. By increasing the sensor's practically usable dynamic range, we can allow (even) greater latitude in exposure changes in post-processing; the original digital SLRs were horrendeously expensive, yet truly horrible in this regard compared to today's even entry-level models.

All of which can be done fully within the framework of what we already have. This is not to say that dramatically improving the usable dynamic range of the sensor is easy, but it's probably a lot easier than what you are proposing, and it's a problem vendors have experience working on.

Professionals, almost by definition, know how to use the equipment of their trade. It isn't really any different if they are photographers or space shuttle pilots. Especially when it can be done without causing information overload, it's usually better to give the user full control of professional equipment. In my opinion, current high-end DSLRs are pretty good about hitting the sweet spot on this.

share|improve this answer
    
Thank you for pointing out the practical problems with this idea. I imagine a camera that cleverly and independently averages every pixel to create a balanced exposure would produce rather grey images. – HamishKL Jan 13 at 4:52

Let's simplify the problem to understand why we will always have to make compromises.

Let's invent the camera you want, but with only one monochrome pixel. It needs to be able to reliably receive and notify the processor of the reception of a single photon. It also needs to be able to receive and notify the processor of the reception of, practically speaking, uncountably infinite photons.

The first case in a situation where there's no light. The second in the case of even a moderate amount of light.

The main issue is that we simply don't have the technology to create a sensor with such a wide dynamic range. We're always going to have to compromise, and right now we're compromising by selecting a higher range where the sensor can accept nearly infinite photons and give us a reading that suggests a relative amount of light hitting the sensor. It doesn't count them at all, but acts like our eyes do - they merely give an output that's relative to the amount of photons hitting them, without attempting to count photons.

This is further complicated by the fact that this is collected over time.

An ideal sensor would actually be more like a geiger counter - measuring the time between photons to give us a nearly instantaneous measurement of the amount of light falling on the sensor, assuming that the photons are relatively evenly spaced (which isn't true, but is a convenient assumption, and why geiger counters average over time just like cameras do).

Quantum sensors would have essentially the same problem. Sure, they can sense an individual photon, but at some point they're coming fast enough that you simply can't measure the time between them, or even count how many are coming per exposure period.

So we have this compromise that requires we either take several images of several exposures, or add multiple images of the same high exposure together to tease out the low light areas, or split the incoming light into two or more paths with different sensors of different dynamic range, or build sensors that can group pixels together or stack light sensors, or, or, or - there are literally thousands of ways photographers have overcome this basic problem over the decades with a wide variety of media.

It's a physics limitation that isn't likely to be overcome. We're not ever going to have a camera* with no input from the photographer that allows all the decisions to be made in post processing.

*Of course, if you change the definition of camera, then you might be satisfied with some other process's results, but this is largely subjective. The reality is that if you image a scene with your camera, then show the scene to a person, then the image you shot, they will perceive differences due to inherent differences between their eyes, your image sensor, and the process you used to print the image. Photography is as much about interpretation and art as it is about capturing light, and so a fanatical focus on the "perfect camera" is probably not very useful.

share|improve this answer
    
if you are trying to count nearly inifinite number of photons then you are already dead ;-) (burned by the powerful light source), in other cases we can always invent some technology to handle this, for example --> dpreview.com/articles/5923827506/… – szulat Jan 12 at 20:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.