There are a number of factors. First, the sensor's read speed would have to be less than the reciprocal of the frame rate: if you want 30 fps video, you'd have to be able to read the entire 20Mp sensor in less than 1/30s. Second, the camera would also need a data bus that could support that rate. I don't actually know what the data bus speed is in a modern DSLR, but for 30 fps 20Mp data, you'd need a 24-bit bus running around 30*20,000,000=60MHz, which is probably faster than you'd find in most cameras. Third, the camera itself needs some time to process the data -- even RAW files are processed and compressed to some degree. Fourth, you'd need a place to store all that data. Even a large SD or CF card wouldn't hold more than a few minutes' worth of 20Mp video. Fifth, the write speed of flash memory is already a limiting factor; cameras typically cache images while they're being written to storage, and most cameras only have enough memory to cache a a few dozen or fewer full-resolution RAW images taken in burst mode. Sixth, most people have no practical way to view 20Mp video -- even 4K UHD video is only about 8.3Mp per frame. There's an 8K standard, so it's clearly not impossible to read data from a high resolution sensor quickly, but for all the reasons listed above, it's not surprising that you don't find that capability in your current DSLR.
I haven't really looked into reading individual pixels from an image sensor, but you'd have to assume that doing so would slow things down a lot because you'd have to send the pixel coordinates of every pixel you wanted to read to the sensor. Reading data from the sensor in a pattern that the sensor already knows about would save a lot of work (and probably time). If you look at the data sheet for an image sensor (this one for example), you'll see that there are a lot of registers that control what data to read and how that data should be formatted, so there's certainly some control, but reading the entire image in random order probably isn't practical.
From a comment:
CMOS sensors are usually able to take higher FPS video at lower resolutions, do you know what enables them to do so?...Does the CPU downsample or the sensor itself does that?
Looking at the datasheet linked above might help. You can glean a lot even if you don't fully understand it. For example, we can tell just from the specs that the sensor itself must limit the data it sends for different formats, because it can transfer more frames at lower resolutions. Among the key features is this:
maximum image transfer rate: QSXGA (2592x1944): 15 fps 1080p: 30 fps 1280x960: 45 fps 720p: 60 fps VGA (640x480): 90 fps QVGA (320x240): 120 fps
Digging in, you'll find information about subsampling and windowing. Subsampling is where the sensor outputs only some of the pixels, like every second pixel or every fourth pixel. This effectively reduces the resolution of the sensor. Windowing limits the rectangular region of the sensor from which pixels are output. These functions (and many more) are performed by the sensor itself and are configurable.
Look at the datasheets for other sensors, too. I found one for an ON Semiconductor sensor(PDF) that's quite informative. You'll find more information here about subsampling and windowing, and also a number of timing diagrams that give some idea of the sync issues I talked about above.