32
\$\begingroup\$

So I watched the following 1 minute long Youtube clip from CSI New York. In the clip, using what seems to be the recording from a standard bank camera, they zoom in at least 100, and see the image of the culprit in the reflection of the eye of the girl.

Now, I thought this was completely ridiculous, so much so that I thought it was actually really funny.

However, my friend argued that there are very good tricks for image enhancement, such as "super resolution" a procedure where multiple frames of a video to produce a much higher single resolution image. He did think the show bends the truth quite a bit, but how much?

Honestly, I don't actually know anything about these things, so my question is:

How good is modern image resolution enhancement? Also, how far off are the CSI television programs?

Thank you,

Remark: This is cross posted on the Skeptics site. I was told I might receive better answers here.

\$\endgroup\$
12
  • 11
    \$\begingroup\$ I almost spit out my coffee when she nodded so seriously and said "corneal imaging". The whole tone of that scene is so patently ridiculous — if this were possible at this level, it would be routine and obvious, not an amazing clever plot twist as it appears to be presented as in that clip. \$\endgroup\$
    – mattdm
    Nov 10, 2011 at 13:51
  • 2
    \$\begingroup\$ LOL, about as real as the satellite imagery on demand zoom-ins in action flicks. \$\endgroup\$ Nov 10, 2011 at 15:10
  • 1
    \$\begingroup\$ @Bob not to the degree done in movies, where they can identify specific persons in a single movie frame from space (or from an aircraft flying 10km up). From a drone flying a few hundred feet high, using a still frame, with some luck, maybe. \$\endgroup\$
    – jwenting
    Nov 11, 2011 at 6:21
  • 1
    \$\begingroup\$ I think you'd be surprised. Not that I'm an expert, but supposedly the SR-71 can spot parking lot lines from 25km. Granted you probably wont get a face, but you could identify them using other tidbits of intel, like their entourage/vehicles en.wikipedia.org/wiki/Reconnaissance_aircraft \$\endgroup\$
    – Jane Panda
    Nov 11, 2011 at 14:59
  • 1
    \$\begingroup\$ 6" resolution from a film frame about the size of a legal pad, yes, not 0.01" resolution from a frame the size of a postage stamp :) \$\endgroup\$
    – jwenting
    Nov 15, 2011 at 13:31

5 Answers 5

44
\$\begingroup\$

Short answer: you can obtain some very good results, but only under certain conditions and absolutely not even close to what is shown in the linked video clip.

My company, Amped Software, develops image and video processing software for forensic and intelligence applications, so basically we are the real world counterpart of the CSI software.

With reference to the general problem of quality enhancement, I can tell you that for our market it is a huge problem to live up to the expectations created by TV series and Hollywood movies. You can see on our samples page that sometimes the results we are able to get are really amazing, but it is important to understand that we can obtain them only under some conditions: if there is information that is covered by disturbs, but it is there, we are able to recover it. If there is no information, we can't and we must not recreate it. In this particular application is essential not only getting the results from a visual point of view, but also following a scientific workflow that must be accepted by the court.

Last year I presented a research describing issues and results on almost 200 cases I've worked on and the final result was the following:

  • in more than 50% of the cases there is nothing to do (for example recovering a license plate that is 5x2 pixels is completely impossible with any software on the world);
  • in about 30% of the cases we can get some little result (for example restoring some letter of a license plate or improve the overall appearance of a face);
  • in 10% of the cases you get good results (you get most of the license plate, for example).

Please note that all these cases had severe quality issues. If their quality was good, we weren't asked to work on them.

For what regards specifically resolution enhancement:

  • when you zoom on an image you are interpolating missing pixels: from a single image you can improve visually the appearance of the image but you will not add any real detail;
  • super resolution techniques may yield good results under certain conditions: you should have enough frames, shifted by a non integer amount of pixels and preferably with few compression artifacts. In the best case you can expect good results within 2x and 3x zoom.

What is shown in the video clip can be possible only if the original video has been shoot at several megapixels and then you will have the resolution to zoom very close (more or less like you do on Google Maps). Of course, at that point there still would be other problems, like the right focus, low light condition, the fact that the perspective of the eye is different from that of the whole subject in the video, just to mention a few.

\$\endgroup\$
0
13
\$\begingroup\$

You can't make something out of nothing, you have to have (or guess) some information in order to be able to enhance an image in any way. For example if you know the properties of the blurring function (and there is no image noise) then you can actually unblur a photo. However you rarely know the blur function and noise is always present so that severely limits what you can recover (Adobe recently demonstrated an unblur filter but their demo was with synthetic blur).

In short, CSI is almost pure fiction - the gains that are possible in real life are marginal, nothing like the 5x increase in resolution that is presented on TV.

Sanity check: if they could do all that, people wouldn't be paying tens of thousands for 40+ MP Hasselblad cameras, it would be cheaper to simply duplicate the software!

edit: I somehow didn't notice the original question mentioned super-resolution from video. Multiple image super-resolution is possible in reality but only up to the limitations of the sensor. It works by using a set of images with sub pixel shifts. This gives information of the values in between pixels allowing you to build up a higher resolution image. Super-resolution from video works because a moving subject creates the same sort of shifts, however the appearance of objects mustn't change that much between frames. The technique at best is only giving you the results of a higher resolution sensor, you can't overcome the limits of lens resolving power, which would be quite limited with CCTV spec lenses.

Here is an example of superresolution under good conditions:


(source: wikimedia.org)

image from photoacute.com

an improvement in resolution, yes, but still nowhere near CSI level performance.

To reference my comment above, the latest Hasselblads actually implement sensor shift super-resolution, under the name "multi-shot capture" so you still can't beat the medium format manufacturers at their own game using clever software...

\$\endgroup\$
2
  • \$\begingroup\$ Seems you have a couple broken sentences: "...lens resolving power, which would ??" and "?? an improvement in resolution, yes, ..." \$\endgroup\$
    – jrista
    Nov 15, 2011 at 1:10
  • \$\begingroup\$ @jrista Thanks, I've fixed the first one, the second one is supposed to be the continuation of the sentence above the image. \$\endgroup\$
    – Matt Grum
    Nov 15, 2011 at 10:14
3
\$\begingroup\$

So this is what I have so far:

The area of the eye where they see the Basketball is about 1 square millimeter. Based on the height of the girl, we can confidently estimate that that should be at most 1 millionth of the total number of pixels on the screen. (The width and height of the image, while not well defined for photographs since some thing may be closer, look roughly about 3 x 3 meters, which would mean 10 million square millimeters. The eye however might be closer to the actual camera lense so I am roughly estimating and divide by $10$.)

If the recording was HD quality, that would still only be 2 megapixels, so where we see the basketball should be the size of a single pixel.

I think that reasoning undoubtedly shows it is not true, but I am still left wondering, what are the upper limits on image enhancement?

\$\endgroup\$
1
  • 1
    \$\begingroup\$ my guestimate is that inventing more than 50% of pixels will create total fantasy, and long before that your image will become so bad as to be useless for identifying specific people or items except by general shape. \$\endgroup\$
    – jwenting
    Nov 11, 2011 at 6:23
2
\$\begingroup\$

There is some commercially available software out there, with Super Resolution. I haven't tried any of these myself, but the advertising material is pretty good. The software is pitched towards surveillance, security and armed forces but I guess some forensic units will have access to this stuff.

Two examples are: Ikena from MotionDSP and TacitView from 2d3

\$\endgroup\$
3
  • 3
    \$\begingroup\$ It should be noted that superresolution requires multiple source images to feed the algorithm more data than it would otherwise have. This works with video, since your continually capturing an ongoing sequence of frames, where each subsequent frame is usually mostly similar to the prior. Additionally, the more source resolution, the more food for the algorithm to munch on. Such image enhancement wouldn't really be possible with a single static image, or images from a low-framerate, low-resolution camera like the youtube video seemed to indicate. \$\endgroup\$
    – jrista
    Nov 15, 2011 at 1:14
  • \$\begingroup\$ That's true, and the answer form @Matt-Grum explains this in a bit more detail. There is also good information on Wikipedia about Super Resolutino and Speckle imaging (also known as video astronomy). Both works on series of pictures, but use different techniques to create the finished product. \$\endgroup\$ Nov 15, 2011 at 9:10
  • \$\begingroup\$ Interesting, hadn't heard of the term Speckle imaging before, although I have heard of "stacking", as it is commonly referred to in astrophoto groups. \$\endgroup\$
    – jrista
    Nov 15, 2011 at 17:34
1
\$\begingroup\$

Image/video enhancing to the level suggested in TV shows is simply not possible, and is actually limited by the image capturing device. That's the technology that would need to evolve first.

It is impossible to gain information from a collection of 10 pixels into a recognisable object. At the pixel level, that's the final amount of information provided in the image. You only have 100 blocks of colour in that finite zoom. You can increase a 10x10 pixel area into 100x100 pixels via interpolation, but the information in the 10x10 pixels is all there is to go by, and interpolation relies on educated guesses based on those 10 x 10 pixels by the software. The result would be a 100 x 100 pixel blur. Alternatively, take a 2000x2000 pixel image, and interpolate it up 4000x4000 pixels, and some blurs not as clear in the original image may look clearer as possible objects in the larger image purely through pareidolia - but even then that's a guess or supposition. The interpolated image would give the "illusion" of more detail, but still relies on the detail of the original 2000x2000 image capture (or 4,000,000 finite pixels of information).

Image enhancement relies solely on the maximum amount of information captured via the original imaging source, and zooming into pixels is all there is. Some cameras can take very good resolution images, but no piece of software can bring out detail that is not recorded in the original image.

Now, to go back to the CSI shows, most of them grab your standard surveillance camera footage, which to begin with is not particularly high in resolution anyway in real life, so when I see this sort of enhancement on shows it just makes me laugh - as it is impossible even with the most advanced form of image editing software (and PhotoShop is well and truly at the upper end of the advanced scale). Image enhancement and zooming relies 100% on the information captured by the imaging device - so a surveillance camera capable of capturing say a reflection of a person's face on the side of someone's eye cornea, would need to be super powerful, and would be outrageously expensive. The footage would need to have an outrageously high pixel definition per frame (say 100 megapixels or approx 9.5 terabytes to store 1 second of footage), which would blow out the file size of the footage to the point that each surveillance camera would need a small server farm to hold 24 hrs worth of footage. Very expensive. The resulting enhancement technology would rely on image capturing devices able to do this level of detailed video and image capturing first, to store the data in a convenient fashion, and be at a point that this is inexpensive to roll out across cities. The file size would be so enourmously large on the video footage (remember about 9.5 terabytes per second) that the software able to do this would need an extremely powerful supercomputer (by today's standards) to run. With technology increasing at the speed that it is, this may be possible at some point in the future, probably in our lifetime. Only then could they do it, but the images wouldn't be enhanced, they would only be zooming on super detailed imagery. I know this as I work in digital imagery for a full time living.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.