Keep in mind the data rate involved in photos is very high, even by memory standards.  Say a camera can save 24 megapixel raw files at 30mb each and can take a shot at 1/8000 of a second shutter speed.  That means it needs to be able to save 30mb of information in 1/8000 of a second.  That's 240gb per second of transfer rate while it is held for processing by the processing circuitry.  

Even if the buffer is an intermediate dump from the initial frame buffer, you're still talking about several hundred megabytes per second on the faster burst modes.  The cache memory has to be extremely fast and also extremely low power.  This doesn't leave a lot of options for how it is produced and can greatly increase the cost and space requirements.

In addition, your still base limited by the rate you can write to the card since eventually the buffer has to empty.  Using less extra fast cache allows for the cache to be writing to the card while information is still being dumped from the sensor and worked on by the processor.  This is why a faster card will extend the maximum burst length significantly in a good camera (to the point that shooting jpegs can go almost indefinitely on higher end models.)