I'll be brief: How do digital photo sharpening tools restore edges to photos? Why do they only sometimes work? Why don't they restore the image perfectly i.e what are the limits of the tools?
Sharpening tools are basically faking sharpness by exaggerating the contrast between the two sides of an edge.
E.g. if you have a dark surface against a bright background, you can make the edge appear sharper by darkening the pixels on the dark side of the edge, and brightening the pixels on the bright side of the edge.
Top half is the original, bottom half is sharpened. Illustration from wikipedia, which also has further explanation.
Sharpening tools don't "restore" anything, they merely put a bit of makeup on whatever data we feed them.
- Software might look for dark/bright edges, and miss color edges.
- If there is too little contrast between edge and background, the edge may not be detected.
- If you turn up the sharpness too much, the "fakery" becomes visible, and can be rather distracting. See ringing.
There are large number of techniques used to sharpen and limits keep expanding (checkout the newer tools to fix shake/motion induced blur -- sort of a post processed version of IS).
The oldest techniques would generally look for areas of contrast and then increase that contrast. Moving forward from there are the edge detection algorithms that look for edges of contrast and increase the sharpness along those edges.
The limitation of all these tools is that you cannot get more information out of an image than is already there -- of course smart people are always trying to improve on how far we can get with what is there (and there is a lot of information in a shot from any semi-modern camera).
Sharpening is just one of the effects that can be achieved through convolution, a process where each pixel's value is modified according to the sum of neighboring pixels weighted according to a convolution matrix, a.k.a. a kernel.
A kernel for sharpening increases the value of the pixel being considered, and subtracts the neighboring pixels. For example, the kernel might look like this:
0 -1 0 -1 5 -1 0 -1 0
This would cause the pixel under consideration to be multiplied by 4, and the top, left, bottom, and right neighbors to each be multiplied by -1, and the diagonal neighbors to be multiplied by 0. Then all the values are added together. It's a little hard to see how that works until you remember that the same process happens for every pixel in the image.
For the sake of illustration, let's consider a grayscale image rather than an RGB one so that we don't have to worry about different components for each pixel. (The process would be the same for each component -- we'd just have to do more math.) Here are brightness values for a 5x5 image with an edge:
100 100 100 120 140 100 100 100 120 140 100 100 120 140 140 100 100 120 140 140 100 120 140 140 140
Before we start we should discuss how we'll handle edges. There are several options, but we'll just extend the image with the same value. For example, in the top right corner we'll use 120 for the top left neighbor and 140 for the top, top right, right, and bottom right neighbors.
Applying the kernel to the top left pixel in our sample image, we have:
100*0 + 100*-1 + 100*0 + 100*-1 + 100*5 + 100*-1 + 100*0 + 100*-1 + 100*0 +
which works out to
4*-100 + 5*100 = 100. So, no change for that first pixel. Let's repeat the process for all the other pixels, and the result is...
100 100 80 140 160 100 100 60 100 160 100 100 140 180 140 100 60 100 160 140 80 140 180 140 140
You can see that the values that where the neighboring pixels are similar, in the top left and bottom right corners, the operation doesn't introduce any changes. Where pixels differ, though, the differences are magnified: darker pixels become even darker and brighter pixels become even brighter.
You can try this yourself on real images if you have an image editor that lets you supply your own convolution matrix. In Photoshop, for example, the Custom Filter command gives you a 5x5 matrix -- just center our 3x3 matrix on the 5x5 and leave the unused values empty, or set them to 0 (which is the same thing).
There are 2 main different approach to sharpening:
- Amplifying Existing Details
- Restoring "Lost" Details
In practice, the tools used for one could be used for the other.
The difference is the second option requires information on the image (Either given by the user -> Deconvolution or needed to estimated -> Blind Deconvolution).
The first family, is basically "Generalized High Pass".
Namely the amplify the High Frequency Information of the image.
High Frequency Component are needed mainly to describe "Jumps" which in the Image Processing world are Edges.
These is the idea behind Unsharp Mask and High Pass Filter Sharpening.
Today there are even more modern methods to do so in images (See Topaz Detail, Know-How Wow! Frequency Equalizer, etc... which offer Multi Scale Sharpening).
The second family tries to cancel a blurring effect.
Imagine someone applied Small Radius Gaussian Blur on you image before you got it as a digital file (For instance, like the Anti Aliasing (AA) filter in Digital Camera).
Now you'd like to do the inverse operation.
Since applying Linear Spatially Constant filter is don by Donvolution, what's you're after is applying Deconvolution.
The issue sometimes you don't know what filter was applied, then the problem becomes Blind Deconvolution and it is much harder.
While the first family is judged by taste, as the end result needs only to make the picture "Nicer to the Viewer" the second family can be measured in how well the restored image approximate the original image.
The reasons it might fail are numerous:
- Bad estimation of parameters.
- Bad Signal to Noise ratio (Image is noisy).
- Wrong model.
There are some products out there who tries to tackle this (Even Smart Sharpen in Photoshop tries doing so).
Yet I haven't seen anything which is successful.