How does Super Resolution work?

Question

Picking from this question I read the Wikipedia article but I can't understand why stacking copies of the same image over each other should enhance the image resolution?

Answer 1

The process is complicated but this should give an intuition into what's going on. Imagine you have a regular camera, but with motors to move the sensor half a pixel in any direction.

Instead of taking one image this camera takes four: one centred, one shifted half a pixel right, one shifted half a pixel down, and one shifted half a pixel right and half a pixel down.

We can then take the centred image, make it double the size, spacing out the pixels like so:

xxxx               x x x x 
xxxx      ____\    
xxxx          /    x x x x 
xxxx               
                   x x x x

                   x x x x

Then we can fill in the gaps, using the other shifted images, 1, 2, and 3:

x1x1x1x1
23232323
x1x1x1x1
23232323
x1x1x1x1
23232323
x1x1x1x1
23232323

Leaving us with a image of twice the resolution. Interestingly enough there are cameras than employ this technique - such as the Hasselblad H4D-200MS (sorry if you have to ask how much you can't afford one).

Superresolution with a standard camera is a bit more complex as when you have uncontrolled camera or subject motion you don't get anywhere near an exact half pixel shift, but unless you are extremely unlucky your shifted image will be some amount offset from the original. By combining enough images you will get a very irregularly sampled image (with pixel samples that don't fall onto a grid) but one which can be interpolated (by tracing lines between samples to guess a result that does fall on an exact gridline) into a regular image.

Answer 2

Consider that the sensor is not a perfect capture device. Each pixel will be recorded with some amount of error. For example, if the most accurate value of a pixel is N, the sensor will record a value that is in a range N-E to N+E for a given E. For a good sensor E is small, a bad sensor will have a larger E.

Also note that on each exposure a given pixel will have a different error, the cells in the sensor have no memory, so a pixel that came out low one time may come out high in the next one.

When you take several exposures of the same subject and average them together you are effectively reducing E. For our example pixel above, you will be averaging a bunch of different values that are all around an unknown N, so the average will bring you closer to that ideal N.

Answer 3

The following is as I understand things. People should feel free to point out any misconceptions so that we are all edified, but hopefully will actually point out any they spot and not just mumble in their beer. (or beards or ...).

Put simply and simplistsically, there is slightly different information in the different photos and various methods are used to detect and extract this extra information and combine it in a consistent additive manner.

It's worth noting that the system is not guaranteed to work in all cases.
The [Wikipedia Super-Resolution page] notes:

• In the most common SR algorithms, the information that was gained in the SR image was embedded in the LR images in the form of aliasing.

  This requires that the capturing sensor in the system is weak enough that aliasing is actually happening. A diffraction-limited system contains no aliasing, nor does a system where the total system Modulation Transfer Function is filtering out high-frequency content.

Aliasing is the ability of the system to properly present data of the frequencies concerned. See "explanation" as end.

If I understand them correctly (and I may or may not) their phrase"is weak enough" means that their is extra information that the sensor cannot itself resolve that is usually considered bad so it is normally suppressed where possible VBUT that this "aliased" extra information is needed by the SR system. The Nikon D800r has no antialisasing optical filter on the sensor whereas the std D800 and almost all other DSLRs do have such a filter.

MTF is effectively the ability of the lens to produce contrast OR to produce "sharpness" (the two being tightly interrelated. MTF is usally best near the lens middle and, with a rectangular image, falls off towards edges and usually more so in image corners. They are saying that the ability of the system to produce a super resolution image depends on its ability to render contrast and sharpness - ie on its quality. ie the lens needs to be at least about as good as the lens which would oroduce the super rsolutionj iomage directyly if sensor and process capabilities were improved.

---

Aliasing is what happens when an information stream is sampled so slowly that some of the high frequency information changes more rapidl;y than the sampling rate and "wraps around" and appears as if it is really a lower frequency component. In a limiting system the sampling rate needs to be at least double the highest information rate present but in practice somewhat higher rates than this are required.

Simple example:

• Consider the sequence 0 1 2 3 4 5 6 5 4 3 2 1 0 1 2 3 4 5 6 5 4 3 2 1 0 1 2 3 4 5 6 ...

  Clearly there is a pattern which repeats every 12 units.
  It's a triangle ave which increases for 6 cycles and decreases for another 6 cycles and then repeats, with period = 12 units.

  Now sample the sequence every 11th time only. We get
  0 1 2 3 4 5 6 5 4 3 2 1
  This is exactly the same pattern BUT it changes 11 times more slowly - a triangle wave with period 11 x 12 = 132 units.

  Sample the same sequence every 8th time and you get 0 4 4 0 4 4 0 4 4
  ie it looks like a 1:2 square wave with period = 24 units.

  Any sampling period greater than 6 units of time = half a cycle will result in such aliasing errors.