A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.
2 added 1468 characters in body
source | link

There are a number of factors. First, the sensor's read speed would have to be less than the reciprocal of the frame rate: if you want 30 fps video, you'd have to be able to read the entire 20Mp sensor in less than 1/30s. Second, the camera would also need a data bus that could support that rate. I don't actually know what the data bus speed is in a modern DSLR, but for 30 fps 20Mp data, you'd need a 24-bit bus running around 30*20,000,000=60MHz, which is probably faster than you'd find in most cameras. Third, the camera itself needs some time to process the data -- even RAW files are processed and compressed to some degree. Fourth, you'd need a place to store all that data. Even a large SD or CF card wouldn't hold more than a few minutes' worth of 20Mp video. Fifth, the write speed of flash memory is already a limiting factor; cameras typically cache images while they're being written to storage, and most cameras only have enough memory to cache a a few dozen or fewer full-resolution RAW images taken in burst mode. Sixth, most people have no practical way to view 20Mp video -- even 4K UHD video is only about 8.3Mp per frame. There's an 8K standard, so it's clearly not impossible to read data from a high resolution sensor quickly, but for all the reasons listed above, it's not surprising that you don't find that capability in your current DSLR.

I haven't really looked into reading individual pixels from an image sensor, but you'd have to assume that doing so would slow things down a lot because you'd have to send the pixel coordinates of every pixel you wanted to read to the sensor. Reading data from the sensor in a pattern that the sensor already knows about would save a lot of work (and probably time). If you look at the data sheet for an image sensor (this one for example), you'll see that there are a lot of registers that control what data to read and how that data should be formatted, so there's certainly some control, but reading the entire image in random order probably isn't practical.


From a comment:

CMOS sensors are usually able to take higher FPS video at lower resolutions, do you know what enables them to do so?...Does the CPU downsample or the sensor itself does that?

Looking at the datasheet linked above might help. You can glean a lot even if you don't fully understand it. For example, we can tell just from the specs that the sensor itself must limit the data it sends for different formats, because it can transfer more frames at lower resolutions. Among the key features is this:

maximum image transfer rate:
QSXGA (2592x1944): 15 fps 1080p: 30 fps
1280x960: 45 fps
720p: 60 fps
VGA (640x480): 90 fps
QVGA (320x240): 120 fps

Digging in, you'll find information about subsampling and windowing. Subsampling is where the sensor outputs only some of the pixels, like every second pixel or every fourth pixel. This effectively reduces the resolution of the sensor. Windowing limits the rectangular region of the sensor from which pixels are output. These functions (and many more) are performed by the sensor itself and are configurable.

Look at the datasheets for other sensors, too. I found one for an ON Semiconductor sensor(PDF) that's quite informative. You'll find more information here about subsampling and windowing, and also a number of timing diagrams that give some idea of the sync issues I talked about above.

There are a number of factors. First, the sensor's read speed would have to be less than the reciprocal of the frame rate: if you want 30 fps video, you'd have to be able to read the entire 20Mp sensor in less than 1/30s. Second, the camera would also need a data bus that could support that rate. I don't actually know what the data bus speed is in a modern DSLR, but for 30 fps 20Mp data, you'd need a 24-bit bus running around 30*20,000,000=60MHz, which is probably faster than you'd find in most cameras. Third, the camera itself needs some time to process the data -- even RAW files are processed and compressed to some degree. Fourth, you'd need a place to store all that data. Even a large SD or CF card wouldn't hold more than a few minutes' worth of 20Mp video. Fifth, the write speed of flash memory is already a limiting factor; cameras typically cache images while they're being written to storage, and most cameras only have enough memory to cache a dozen or fewer full-resolution images taken in burst mode. Sixth, most people have no practical way to view 20Mp video -- even 4K UHD video is only about 8.3Mp per frame. There's an 8K standard, so it's clearly not impossible to read data from a high resolution sensor quickly, but for all the reasons listed above, it's not surprising that you don't find that capability in your current DSLR.

I haven't really looked into reading individual pixels from an image sensor, but you'd have to assume that doing so would slow things down a lot because you'd have to send the pixel coordinates of every pixel you wanted to read to the sensor. Reading data from the sensor in a pattern that the sensor already knows about would save a lot of work (and probably time). If you look at the data sheet for an image sensor (this one for example), you'll see that there are a lot of registers that control what data to read and how that data should be formatted, so there's certainly some control, but reading the entire image in random order probably isn't practical.

There are a number of factors. First, the sensor's read speed would have to be less than the reciprocal of the frame rate: if you want 30 fps video, you'd have to be able to read the entire 20Mp sensor in less than 1/30s. Second, the camera would also need a data bus that could support that rate. I don't actually know what the data bus speed is in a modern DSLR, but for 30 fps 20Mp data, you'd need a 24-bit bus running around 30*20,000,000=60MHz, which is probably faster than you'd find in most cameras. Third, the camera itself needs some time to process the data -- even RAW files are processed and compressed to some degree. Fourth, you'd need a place to store all that data. Even a large SD or CF card wouldn't hold more than a few minutes' worth of 20Mp video. Fifth, the write speed of flash memory is already a limiting factor; cameras typically cache images while they're being written to storage, and most cameras only have enough memory to cache a a few dozen full-resolution RAW images taken in burst mode. Sixth, most people have no practical way to view 20Mp video -- even 4K UHD video is only about 8.3Mp per frame. There's an 8K standard, so it's clearly not impossible to read data from a high resolution sensor quickly, but for all the reasons listed above, it's not surprising that you don't find that capability in your current DSLR.

I haven't really looked into reading individual pixels from an image sensor, but you'd have to assume that doing so would slow things down a lot because you'd have to send the pixel coordinates of every pixel you wanted to read to the sensor. Reading data from the sensor in a pattern that the sensor already knows about would save a lot of work (and probably time). If you look at the data sheet for an image sensor (this one for example), you'll see that there are a lot of registers that control what data to read and how that data should be formatted, so there's certainly some control, but reading the entire image in random order probably isn't practical.


From a comment:

CMOS sensors are usually able to take higher FPS video at lower resolutions, do you know what enables them to do so?...Does the CPU downsample or the sensor itself does that?

Looking at the datasheet linked above might help. You can glean a lot even if you don't fully understand it. For example, we can tell just from the specs that the sensor itself must limit the data it sends for different formats, because it can transfer more frames at lower resolutions. Among the key features is this:

maximum image transfer rate:
QSXGA (2592x1944): 15 fps 1080p: 30 fps
1280x960: 45 fps
720p: 60 fps
VGA (640x480): 90 fps
QVGA (320x240): 120 fps

Digging in, you'll find information about subsampling and windowing. Subsampling is where the sensor outputs only some of the pixels, like every second pixel or every fourth pixel. This effectively reduces the resolution of the sensor. Windowing limits the rectangular region of the sensor from which pixels are output. These functions (and many more) are performed by the sensor itself and are configurable.

Look at the datasheets for other sensors, too. I found one for an ON Semiconductor sensor(PDF) that's quite informative. You'll find more information here about subsampling and windowing, and also a number of timing diagrams that give some idea of the sync issues I talked about above.

1
source | link

What is preventing me from sampling it at a higher rate?

Every discrete step in the operation takes time. Changing the value of inputs to a logic gate takes time. Once those inputs have stabilized, getting a stable output from the gate takes time. Placing a new value on a data bus takes time, and then signaling the other device(s) that the data is ready takes time.

In a complicated device like an image sensor, you might have hundreds or thousands of signals changing in parallel, and interacting with each other, and so there's a need for synchronization signal -- something that starts all those changes at once, and then allows enough time for the analog signals to stabilize in all the affected parts. That synchronization comes from a clock signal, which is sort of like the heartbeat of the system. The clock's period -- the time between one tick and the next -- has to be long enough to accommodate the slowest component.

Will it burn out or just start giving me garbage?

Garbage, most likely. The "values" that we're talking about here are voltages. In digital electronics, the digital value of a given signal can only be 0 or 1, on or off. But those values are represented by analog voltages, for example voltages between 0.0V and 0.5V might represent a logical 0 or false value, and voltages between 2.4V and 3.3V might represent a 1 or true value. If you try to read a signal before it has stabilized, though, it might be anywhere between 0V and 3.3V.

However, there is also a risk of permanent damage to a device if you try to run it so fast that it can't dissipate the power it's using. Heat is the enemy of electronics -- materials generally have lower resistance and work faster when they're cool.

What is the reason that a chip capable of taking 20MP images can't take 20MP video?

There are a number of factors. First, the sensor's read speed would have to be less than the reciprocal of the frame rate: if you want 30 fps video, you'd have to be able to read the entire 20Mp sensor in less than 1/30s. Second, the camera would also need a data bus that could support that rate. I don't actually know what the data bus speed is in a modern DSLR, but for 30 fps 20Mp data, you'd need a 24-bit bus running around 30*20,000,000=60MHz, which is probably faster than you'd find in most cameras. Third, the camera itself needs some time to process the data -- even RAW files are processed and compressed to some degree. Fourth, you'd need a place to store all that data. Even a large SD or CF card wouldn't hold more than a few minutes' worth of 20Mp video. Fifth, the write speed of flash memory is already a limiting factor; cameras typically cache images while they're being written to storage, and most cameras only have enough memory to cache a dozen or fewer full-resolution images taken in burst mode. Sixth, most people have no practical way to view 20Mp video -- even 4K UHD video is only about 8.3Mp per frame. There's an 8K standard, so it's clearly not impossible to read data from a high resolution sensor quickly, but for all the reasons listed above, it's not surprising that you don't find that capability in your current DSLR.

Also, do you usually have control over being able to sample from the pixel array or are you forced to do it in predetermined patterns? Can you for example just sample from half the array 2x faster? Can that half be random?

I haven't really looked into reading individual pixels from an image sensor, but you'd have to assume that doing so would slow things down a lot because you'd have to send the pixel coordinates of every pixel you wanted to read to the sensor. Reading data from the sensor in a pattern that the sensor already knows about would save a lot of work (and probably time). If you look at the data sheet for an image sensor (this one for example), you'll see that there are a lot of registers that control what data to read and how that data should be formatted, so there's certainly some control, but reading the entire image in random order probably isn't practical.