1

A typical digital camera sensor operates by capturing the analog state of each pixel after it has been active for some period of time. If some pixels would saturated in less time than it takes other pixels to change measurably, this will result in the pixels that receive more light being "blown out", making it impossible to distinguish between pixels that received enough light to saturate quickly, versus those that received just enough light to saturate eventually.

What would be the practical effect of augmenting or replacing each pixel with a "has saturation yet been reached" circuit and a counter, and measuring the time between when the circuit is allowed to start charging or discharging, and when it reaches saturation? If one were to try to photograph the moon behind some hills, the pixels seeing the brightest parts of the moon might reach saturation in less than a millisecond, while those seeing darker parts of the hills might take dozens of milliseconds to capture any sort of useful image, but if in a "1/20 second" exposure the camera determined that some parts of the image reached saturation after 1,037 microseconds, some reached saturation in 38,197 microseconds, and then for each pixel the camera would log either the time to reach saturation (if saturation was reached), or the sensor value (if it wasn't), that would seem like a fairly easy way to augmnent the dynamic range of the sensor by many f-stops. While a 16-bit counter per pixel would take up some space on the silicon, I would expect that a 16-bit LFSR-based counter would take up far less space than most kinds of analog sensor.

The biggest downside I could see to such a design is that motion artifacts might look weird. If, during a 1/20 second exposure, some bright objects were moving through the frame in front of backgrounds of differing intensity, the lengths of streaks left by the objects would vary based upon the intensity of the backgrounds behind them. Otherwise, though, I would think such an approach could offer a much wider dynamic range than would typically be practical.

Do any commercial or experimental camera designs make use of such techniques? While I am aware of some cameras that try to use automatic bracketed exposures, or incorporate sensing elements with differing levels of sensitivity in their matricies, bracketing exposures results in images that are captured during two non-overlapping time intervals, and any mix of elements that are more or less sensitive would need to be determined when a sensor is built, rather than adjusting according to a scene. Even if motion artifacts would look horrible when filming moving subjects, I would think a camera with time-sensing pixels would offer sufficient advantages for still photography as to be highly desirable. Do any exist, or are there technical issues I haven't seen?

12
  • "What would be the practical effect of augmenting or replacing each pixel with a "has saturation yet been reached" circuit" This just doesn't seem in any way practical to me, you can't put all that electronics on each of 20 million+ pixels.
    – Philip Kendall
    Nov 10 at 19:31
  • The circuit density required to hold counts in a circuit layer underneath the light-sensing surface wouldn't seem unreasonably high by modern standards. It's been decades since I took a VLSI design course, but a circuit that could accommodate fast counting with pixel dimension less than 10x the minimum feature dimension would seem achievable, and slower counting approaches could be made more compact than that.
    – supercat
    Nov 10 at 21:25
  • @supercat I think the trouble's going to be interconnect - One of my recent projects involved a process that supported image sensors, and when the image sensor layers were in play the stackup got weird; I can imagine that the higher complexity of this counting circuit could put unacceptable amounts of metal in the light path (but I would need to look at the stackup to be sure)
    – nanofarad
    Nov 10 at 21:29
  • @nanofarad: I suspect the biggest shortcomings in my 1990s understanding have to do with three-dimensional stacking issues. Two metals, one poly, and two diffusion can altogether be treated as essentially "flat" when using 500nm processeses. To minimize total circuitry area, both under the sensor and elsewhere, a simple approach would be an SRAM array where the bits associated with each pixel are gated by a threshold sensor. To capture an image, write a sequence of count values to all memory cells in gray-code order, and then read back the array. Various techniques...
    – supercat
    Nov 10 at 21:45
  • ...could allow some memory bits to be moved to the top and bottom of the array in exchange for a possible reduction in count speed, For example, if one kept four bits in each pixel, and added RAM to the top and bottom of the sensor area, a 3072-pixel-high sensor would need to have about 256 rows worth of count data read out for each time the counter advances. Having the coutner advance every microsecond with such a design might be difficult, but having it read out every ten would seem reasonable (25.6 million reads/second per column).
    – supercat
    Nov 10 at 21:50

2 Answers 2

3

If you want to use the logged time as an indicator of exposure (relative brightness) you would also need to incorporate another way of converting that data compared to the relatively simple ratiometric voltage conversion the ADC does in current designs. And that conversion would have to happen in parallel with the ADC (secondary ADC?) and be combined afterwards digitally as they are two separate measures/values/units. So you are also increasing the processing load and introducing a secondary data stream/pipeline.

The current approach is to use dual gain architecture in bright scenarios where photosites are more likely to saturate (secondary capacitor in parallel)...

I.e. when a low ISO is selected the photosite's FWC is increased so that the pixels are less likely to reach saturation. The slight disadvantage this has is that the larger FWC also has a higher conversion gain requirement (it's somewhat less sensitive to lower light levels).

The other current direction is increasing ISO invariance and smaller pixels (i.e. reducing the minimum conversion gain requirement and reducing/offsetting camera generated noise levels). In this way you can simply underexpose an image by using a lower ISO and prevent any pixels from saturating in the first place; and the resulting image/data can be digitally manipulated (brightened in camera/post) with no negative effect.

I don't really see where your idea provides significant benefits over the current directions/processes...

9
  • I didn't want to get into excessive details discussing the purpose of such a design when asking if anything similar had been done, but the goal would be for a sensor to have an s-curve-like response somewhat like that of film over a total dynamic range spanning 12 or more f-stops. If some light level would take 1000 counts for the sensor to reach the trigger threshold, then a light level 100x as high would still take 10 counts to reach the threshold.
    – supercat
    Nov 11 at 17:25
  • The current dynamic range of modern cameras which are (nearly/essentially) ISO invariant is already approaching 14 stops measurable. Nov 11 at 18:27
  • Do such cameras use a single integration period common to all pixels, or do they all have some means of shortening the sampling period of pixels which saturate? Fourteen f-stops would seem difficult with "conventional" sampling methods.
    – supercat
    Nov 13 at 20:11
  • They use a single integration time for all pixels. The key is reducing the minimum electron (photon) requirement for a usable signal (above noise/read error levels). For a photosite with a FWC of 66Ke-, if the minimum signal required were only 400e-, then the DR capability would be 16 stops. For instance, the 46MP D850 has a FWC of ~ 61Ke- and a measurable pixel level DR of 13.55 stops. FW capacities well in excess of 100Ke- exist in current lower MP FF sensors. Nov 13 at 22:14
  • But I should note that "measurable" (engineering) DR is generally about 2 stops more than "useful/photographic" DR... the first couple stops being relatively obscured by noise visually. Nov 13 at 22:17
1

As for experimental cameras, some ideas avoiding the fundamental saturation problem are already being researched, for example:

  • MIT Modulo Camera - pixels reset to zero (and start accumulating again) instead of saturating
  • Quanta Image Sensor - no analog/digital conversion, count individual photons instead
4
  • "Could a practical sensor measure time until pixel reaches threshold?" The Modulo and Quanta designs are far from practical if practical means commercially feasible for a consumer level product.
    – Michael C
    Nov 13 at 18:21
  • 1
    "Do any commercial or experimental camera designs make use of such techniques?" I guess practical could also mean experimental cameras that were actually constructed and tested, as opposed to purely theoretical design ideas.
    – szulat
    Nov 13 at 19:14
  • @MichaelC: There's a difference between "immediate" practicality and potential long-term practicality. If someone were to construct a camera which used individual fiber-optic light pipes from an imagine plane to a humongous set of discrete sensors, I'd regard that as impractical even if someone were to spend millions of dollars constructing a megapixel sensor using that technique, since I can't imagiene any plausible means of constructing such a thing cheaply. On the other hand, if a technique would require experimentation to figure out how to put a sensor circuit on top of...
    – supercat
    Nov 13 at 20:20
  • ...other circuitry withiout affecting its flatness, but would then ease the demands placed on the sensor circuitry, then such a design could be practical even if people haven't yet figured out a cheap way of making the chip under the sensor area acceptably flat, since it would seem plausible that cheaper ways of making the top of a particular layer flat might be developed if there were any perceived need to do so.
    – supercat
    Nov 13 at 20:23

Not the answer you're looking for? Browse other questions tagged or ask your own question.