How does Super Resolution work?

Picking from this question I read the Wikipedia article but I can't understand why stacking copies of the same image over each other should enhance the image resolution?

The process is complicated but this should give an intuition into what's going on. Imagine you have a regular camera, but with motors to move the sensor half a pixel in any direction.

Instead of taking one image this camera takes four: one centred, one shifted half a pixel right, one shifted half a pixel down, and one shifted half a pixel right and half a pixel down.

We can then take the centred image, make it double the size, spacing out the pixels like so:

xxxx               x x x x
xxxx      ____\
xxxx          /    x x x x
xxxx
x x x x

x x x x


Then we can fill in the gaps, using the other shifted images, 1, 2, and 3:

x1x1x1x1
23232323
x1x1x1x1
23232323
x1x1x1x1
23232323
x1x1x1x1
23232323


Leaving us with a image of twice the resolution. Interestingly enough there are cameras than employ this technique - such as the Hasselblad H4D-200MS (sorry if you have to ask how much you can't afford one).

Superresolution with a standard camera is a bit more complex as when you have uncontrolled camera or subject motion you don't get anywhere near an exact half pixel shift, but unless you are extremely unlucky your shifted image will be some amount offset from the original. By combining enough images you will get a very irregularly sampled image (with pixel samples that don't fall onto a grid) but one which can be interpolated (by tracing lines between samples to guess a result that does fall on an exact gridline) into a regular image.

• It's important to remember that the movement doesn't have to happen inside the camera, it can be done by moving the camera itself. That's why super resolution works very well with video, since the camera almost always will have tiny movements between frames. Jun 28, 2012 at 10:42
• This drizzle or dithering is used extensively with digital astrophotography. Many individual images are stacked to help reduce the noise inherent in the image while at the same time there is the opportunity to recover significant detail and resolution. Jun 28, 2012 at 20:35
• and how stacking multiple copies of the same photo can achieve that? I'm confused because after stacking you'll have to align and this will cancel the shift between pixels which looks like the core idea behind super resolution
– K''
Jun 28, 2012 at 20:35
• @AkramMellice - stacking of the same doesn't do this. Software can be used to examine and potentially "fill in" the info based on other information in the image, but it would be less than perfect... At any rate, it would be incredibly awesome if Pentax added this feature, something entirely possible given their anti-shake is on sensor and it's a variation on how they do the astrotrace with the GPS unit. Jun 29, 2012 at 0:04

Consider that the sensor is not a perfect capture device. Each pixel will be recorded with some amount of error. For example, if the most accurate value of a pixel is N, the sensor will record a value that is in a range N-E to N+E for a given E. For a good sensor E is small, a bad sensor will have a larger E.

Also note that on each exposure a given pixel will have a different error, the cells in the sensor have no memory, so a pixel that came out low one time may come out high in the next one.

When you take several exposures of the same subject and average them together you are effectively reducing E. For our example pixel above, you will be averaging a bunch of different values that are all around an unknown N, so the average will bring you closer to that ideal N.

• It's not just reducing error - in fact in this case it's not MAINLY reducing error - it's getting MORE information by capturing the nominally the same data repeatedly and looking for extra information that is present with a resolution finer than the sensor pitch. Addidng some mechanicl dither to the sensor between shots and then realihning them would aid this. // The A77 Sony DSLR and some other Sony's has a multi shot low noise aggregating mode that is NOT just an averaging program. You can move the camera substantially as it takes 6 frames and still get an improved image. Jun 28, 2012 at 9:26
• This explains how image stacking for noise reduction works. Whilst excessive noise can prevent you from resolving details, superresolution is different as it requires at least some camera movement to increase resolution. Jun 28, 2012 at 9:44
• As it has already been pointed out, what you're describing is not super resolution, but image stacking. This does not increase the resolution, but reduces the noise. Jun 28, 2012 at 10:39
• While the title of the question mentions super-resolution, the question in the body asks about 'stacking copies of the same image'. What I explained is, in fact, one of many ways to increase the resolution of your pictures, and it can be done with any camera. Jun 28, 2012 at 15:08

The following is as I understand things. People should feel free to point out any misconceptions so that we are all edified, but hopefully will actually point out any they spot and not just mumble in their beer. (or beards or ...).

Put simply and simplistsically, there is slightly different information in the different photos and various methods are used to detect and extract this extra information and combine it in a consistent additive manner.

It's worth noting that the system is not guaranteed to work in all cases.
The [Wikipedia Super-Resolution page] notes:

• In the most common SR algorithms, the information that was gained in the SR image was embedded in the LR images in the form of aliasing.

This requires that the capturing sensor in the system is weak enough that aliasing is actually happening. A diffraction-limited system contains no aliasing, nor does a system where the total system Modulation Transfer Function is filtering out high-frequency content.

Aliasing is the ability of the system to properly present data of the frequencies concerned. See "explanation" as end.

If I understand them correctly (and I may or may not) their phrase"is weak enough" means that their is extra information that the sensor cannot itself resolve that is usually considered bad so it is normally suppressed where possible VBUT that this "aliased" extra information is needed by the SR system. The Nikon D800r has no antialisasing optical filter on the sensor whereas the std D800 and almost all other DSLRs do have such a filter.

MTF is effectively the ability of the lens to produce contrast OR to produce "sharpness" (the two being tightly interrelated. MTF is usally best near the lens middle and, with a rectangular image, falls off towards edges and usually more so in image corners. They are saying that the ability of the system to produce a super resolution image depends on its ability to render contrast and sharpness - ie on its quality. ie the lens needs to be at least about as good as the lens which would oroduce the super rsolutionj iomage directyly if sensor and process capabilities were improved.

Aliasing is what happens when an information stream is sampled so slowly that some of the high frequency information changes more rapidl;y than the sampling rate and "wraps around" and appears as if it is really a lower frequency component. In a limiting system the sampling rate needs to be at least double the highest information rate present but in practice somewhat higher rates than this are required.

Simple example:

• Consider the sequence 0 1 2 3 4 5 6 5 4 3 2 1 0 1 2 3 4 5 6 5 4 3 2 1 0 1 2 3 4 5 6 ...

Clearly there is a pattern which repeats every 12 units.
It's a triangle ave which increases for 6 cycles and decreases for another 6 cycles and then repeats, with period = 12 units.

Now sample the sequence every 11th time only. We get
0 1 2 3 4 5 6 5 4 3 2 1
This is exactly the same pattern BUT it changes 11 times more slowly - a triangle wave with period 11 x 12 = 132 units.

Sample the same sequence every 8th time and you get 0 4 4 0 4 4 0 4 4
ie it looks like a 1:2 square wave with period = 24 units.

Any sampling period greater than 6 units of time = half a cycle will result in such aliasing errors.