CSI image resolution enhance: How real is it?

Question

So I watched the following 1 minute long Youtube clip from CSI New York. In the clip, using what seems to be the recording from a standard bank camera, they zoom in at least 100, and see the image of the culprit in the reflection of the eye of the girl.

Now, I thought this was completely ridiculous, so much so that I thought it was actually really funny.

However, my friend argued that there are very good tricks for image enhancement, such as "super resolution" a procedure where multiple frames of a video to produce a much higher single resolution image. He did think the show bends the truth quite a bit, but how much?

Honestly, I don't actually know anything about these things, so my question is:

How good is modern image resolution enhancement? Also, how far off are the CSI television programs?

Thank you,

Remark: This is cross posted on the Skeptics site. I was told I might receive better answers here.

Answer 1

Short answer: you can obtain some very good results, but only under certain conditions and absolutely not even close to what is shown in the linked video clip.

My company, Amped Software, develops image and video processing software for forensic and intelligence applications, so basically we are the real world counterpart of the CSI software.

With reference to the general problem of quality enhancement, I can tell you that for our market it is a huge problem to live up to the expectations created by TV series and Hollywood movies. You can see on our samples page that sometimes the results we are able to get are really amazing, but it is important to understand that we can obtain them only under some conditions: if there is information that is covered by disturbs, but it is there, we are able to recover it. If there is no information, we can't and we must not recreate it. In this particular application is essential not only getting the results from a visual point of view, but also following a scientific workflow that must be accepted by the court.

Last year I presented a research describing issues and results on almost 200 cases I've worked on and the final result was the following:

• in more than 50% of the cases there is nothing to do (for example recovering a license plate that is 5x2 pixels is completely impossible with any software on the world);
• in about 30% of the cases we can get some little result (for example restoring some letter of a license plate or improve the overall appearance of a face);
• in 10% of the cases you get good results (you get most of the license plate, for example).

Please note that all these cases had severe quality issues. If their quality was good, we weren't asked to work on them.

For what regards specifically resolution enhancement:

• when you zoom on an image you are interpolating missing pixels: from a single image you can improve visually the appearance of the image but you will not add any real detail;
• super resolution techniques may yield good results under certain conditions: you should have enough frames, shifted by a non integer amount of pixels and preferably with few compression artifacts. In the best case you can expect good results within 2x and 3x zoom.

What is shown in the video clip can be possible only if the original video has been shoot at several megapixels and then you will have the resolution to zoom very close (more or less like you do on Google Maps). Of course, at that point there still would be other problems, like the right focus, low light condition, the fact that the perspective of the eye is different from that of the whole subject in the video, just to mention a few.

Answer 2

You can't make something out of nothing, you have to have (or guess) some information in order to be able to enhance an image in any way. For example if you know the properties of the blurring function (and there is no image noise) then you can actually unblur a photo. However you rarely know the blur function and noise is always present so that severely limits what you can recover (Adobe recently demonstrated an unblur filter but their demo was with synthetic blur).

In short, CSI is almost pure fiction - the gains that are possible in real life are marginal, nothing like the 5x increase in resolution that is presented on TV.

Sanity check: if they could do all that, people wouldn't be paying tens of thousands for 40+ MP Hasselblad cameras, it would be cheaper to simply duplicate the software!

edit: I somehow didn't notice the original question mentioned super-resolution from video. Multiple image super-resolution is possible in reality but only up to the limitations of the sensor. It works by using a set of images with sub pixel shifts. This gives information of the values in between pixels allowing you to build up a higher resolution image. Super-resolution from video works because a moving subject creates the same sort of shifts, however the appearance of objects mustn't change that much between frames. The technique at best is only giving you the results of a higher resolution sensor, you can't overcome the limits of lens resolving power, which would be quite limited with CCTV spec lenses.

Here is an example of superresolution under good conditions:

image from photoacute.com

an improvement in resolution, yes, but still nowhere near CSI level performance.

To reference my comment above, the latest Hasselblads actually implement sensor shift super-resolution, under the name "multi-shot capture" so you still can't beat the medium format manufacturers at their own game using clever software...

Answer 3

So this is what I have so far:

The area of the eye where they see the Basketball is about 1 square millimeter. Based on the height of the girl, we can confidently estimate that that should be at most 1 millionth of the total number of pixels on the screen. (The width and height of the image, while not well defined for photographs since some thing may be closer, look roughly about 3 x 3 meters, which would mean 10 million square millimeters. The eye however might be closer to the actual camera lense so I am roughly estimating and divide by $10$.)

If the recording was HD quality, that would still only be 2 megapixels, so where we see the basketball should be the size of a single pixel.

I think that reasoning undoubtedly shows it is not true, but I am still left wondering, what are the upper limits on image enhancement?

Answer 4

There is some commercially available software out there, with Super Resolution. I haven't tried any of these myself, but the advertising material is pretty good. The software is pitched towards surveillance, security and armed forces but I guess some forensic units will have access to this stuff.

Two examples are: Ikena from MotionDSP and TacitView from 2d3