Why is a physical anti-aliasing filter still needed on modern DSLRs?

Question

I understand the purpose of the anti-aliasing (AA) filter is to prevent moire. When digital cameras first emerged an AA filter was necessary to creat enough blur to prevent moire patterns. At that time the power of in camera processors was very limited. But why is it still necessary to place an AA filter over the sensor in modern DSLR cameras? Couldn't this be accomplished just as easily by the algorithms applied when the output from the sensor is being demosaiced? It would seem that the current processing power available in-camera would allow this now much more than even a few years ago. Canon's current Digic 5+ processor has over 100 times the processing power of the Digic III processor, which dwarfs the power of the earliest digital cameras. Especially when shooting RAW files, couldn't the AA blurring be done in the post processing stage? Is this the basic premise of the Nikon D800E, even though it uses a second filter to counteract the first?

Answer 1

Aliasing is the result of repeating patterns of roughly the same frequency interfering with each other in an undesirable manner. In the case of photography, the higher frequencies of the image projected by the lens onto the sensor creates and interference pattern (moiré in this case) with the pixel grid. This interference only occurs when those frequencies are roughly the same, or when the sampling frequency of the sensor matches the wavelet frequency of the image. That is the Nyquist limit. Note...that is an analog issue...moiré occurs because of interference that occurs real-time in the real-world before the image is actually exposed.

Once the image is exposed, that interference pattern is effectively "baked in". You can use software to some degree to clean moiré patterns up in post, but it is minimally effective when compared to a physical low pass (AA) filter in front of the sensor. The loss in detail due to moiré can also be greater than that lost to an AA filter, as moiré is effectively nonsense data, where slightly blurred detail could still be useful.

An AA filter is just designed to blur those frequencies at Nyquist so they do not create any interference patterns. The reason we still need AA filters is because image sensors and lenses are still capable of resolving down to the same frequency. When sensors improve to the point where the sampling frequency of the sensor itself is consistently higher than even the best lenses at their optimal aperture, then the need for an AA filter would diminish. The lens itself would effectively handle the necessary blurring for us, and interference patterns would never emerge in the first place.

Answer 2

The physics simply doesn't work that way. Aliasing irreversibly transforms frequencies past the Nyquist limit to appear as frequencies below the limit, although those "aliases" aren't really there. No amount of processing a aliased signal can recover the original signal in the general case. The fancy mathematical explanations are rather long to get into unless you've had a class in sampling theory and digital signal processing. If you had though, you wouldn't be asking the question. Unfortunately then the best answer is simply "That't not how the physics works. Sorry, but you're going to have to trust me on this.".

To try to give some rough feel that the above might be true, consider the case of a picture of a brick wall. Without a AA filter, there will be moire patterns (which are actually the aliases) making the brick lines look wavy. You have never seen the real building, only the picture with the wavy lines.

How do you know the real bricks weren't laid down in a wavy pattern? You assume they weren't from your general knowledge of bricks and human experience of seeing brick walls. However, could someone just to make a point deliberately make brick wall so that it looked in real life (when viewed with your own eyes) like the picture? Yes they could. Therefore, is it possible to mathematically distinguish a aliased picture of a normal brick wall and a faithful picture of a deliberately wavy brick wall? No, it is not. In fact you can't really tell the difference either, except that your intution about what a picture probably represents may give you the impression that you can. Again, strictly speaking you can't tell whether the wavies are moire pattern artifacts or are real.

Software can't magically remove the wavies because it doesn't know what is real and what is not. Mathematically it can be shown that it can't know, at least by only looking at the wavy image.

A brick wall may be a obvious case where you can know that the aliased picture is wrong, but there are many more subtle cases where you really don't know, and may not even be aware that aliasing is going on.

Added in response to comments:

The difference between aliasing a audio signal and a image is only that the former is 1D and the latter 2D. The theory and any math to realize effects is still the same, just that it is applied in 2D when dealing with images. If the samples are on a regular rectangular grid, like they are in a digital camera, then some other interesting issues come up. For example, the sample frequency is sqrt(2) lower (about 1.4x lower) along the diagonal directions as apposed to the axis-aligned directions. However, sampling theory, Nyquist rate, and what aliases really are is not different in a 2D signal than in a 1D signal. The main difference seems to be that this can be harder for those not used to thinking in frequency space to wrap their mind around and project what it all means in terms of what you see in a picture.

Again, no you can't "demosaic" a signal after the fact, at least not in the general case where you don't know what the original is supposed to be. Moire patterns caused by sampling a continuous image are aliases. The same math applies to them just as it applies to high frequencies aliasing into a audio stream and sounding like background whistles. It's the same stuff, with the same theory to explain it, and the same solution to deal with it.

That solution is to eliminate the frequencies above the Nyquist limit before sampling. In audio that can be done with a simple low pass filter you could possibly make from a resistor and capacitor. In image sampling, you still need a low pass filter, in this case it's taking some of the light that would hit only a single pixel and spreading it out to neighboring pixels. Visually, this looks like a slight blurring of the image before it is sampled. High frequency content looks like fine detail or sharp edges in a picture. Conversely, sharp edges and fine detail contain high frequencies. It is exactly these high frequencies that get converted to aliases in the sampled image. Some aliases are what we call moire patterns when the original had some regular content. Some aliases give the "stair step" effect to lines or edges, especially when they are nearly vertical or horizontal. There are other visual effects caused by aliases.

Just because the independent axis in audio signals is time and the independent axes (two of them since the signal is 2D) of a image are distance doesn't invalidate the math or somehow make it different between audio signals and images. Probably because the theory and applications of aliasing and anti-aliasing were developed on 1D signals that were time-based voltages, the term "time domain" is used to contrast to "frequency domain". In a image, the non-frequency space representation is technically the "distance domain", but for simplicity in signal processing it is often referred to as the "time domain" nonetheless. Don't let that distract you from what aliasing really is. And no, it's not evidence at all that the theory doesn't apply to images, only that a misleading choice of words is sometimes used to describe things due to historical reasons. In fact, the shortcut "time domain" being applied to the not-frequency domain of images is actually because the theory is the same between images and true time-based signals. Aliasing is aliasing regardless of what the independent axis (or axes) happen to be.

Unless you are willing to delve into this at the level of a couple college courses on sampling theory and signal processing, in the end you're just going to have to trust those that have. Some of this stuff is unintuitive without a significant theoretical background.

Answer 3

You can't get the same effect in software. You can get somewhere nearby, given certain assumptions. But the AA filter spreads light so that it strikes multiple different coloured pixels giving you information that is absent from the no-AA filter sensor.

The Nikon D800E doesn't do anything at all to try and replicate the AA filter. If there are high frequency patterns in the image, you get moire and that's your problem - you have to deal with it!

Aliasing is at it's worse when the frequency of detail in the image is very close to the sampling frequency. For older cameras with low resolution sensors (and hence low frequency sampling) moire was a serious problem with lots of types of image detail so AA filters were strong (nothing to do with limited processing power). Now we have much higher sampling frequencies, it takes much higher frequency image details for moire to show up.

Eventually sampling frequencies will be so high the necessary high frequency object details wont make it past lens aberrations and diffraction effects, making the AA filter redundant. This is partly the reason that some MF backs lack an AA filter, super high resolution plus fashion photographers who like to shoot at f/32 with giant Profoto power packs proving lighting.

Answer 4

These are all good answers and good information. I have a very much simplified explanation. Let's go from 2D to 1D (same concept applies).

When a frequency hits your sensor that is higher than the "max allowed frequency", it will actually create a mirror frequency into the lower side. Once your image has been sampled you will see this lower signal but the camera or your computer doesn't know if this was an actual lower signal that was really there or if it was an alias created from a signal that was too high. This information is lost. That's the reason for the "max allowed frequency" or nyquist frequency. It says this is the highest frequency that can be sampled and above it information will get lost.

an analog to audio: let's say you have your system set up where you want a frequency range from 0hz to 1000hz. to leave a little extra room you sample at 3000hz which makes your niquist 1500hz. here is where the aa filter comes in. you don't want anything above 1500hz to enter, in reality your cut-off will begin right after 1000hz but you ensure that by the time you get to 1500hz that nothing is left.

let's assume you forget the aa filter and you allow a tone of 2500 hz to enter your sensor. it will mirror around the sample rate (3000hz) so your sensor will pick up a tone at 500 hz (3000hz - 2500hz). now that your signal is sampled you won't know if the 500hz was actually there or if it's an alias.

btw. the mirror images happen for all frequencies but are not a problem as long as you are not above the nyquist because you can easily filter them out later. example input tone is 300 hz. you will have aliases at (3000 - 300 = 2700hz [and to be correct also 3000 + 300 = 3300hz]). however since you know that you are only considering up to 1000 hz these will be easily removed. so again the problem arises when the mirror images come into the spectrum that you actually want, because you won't be able to tell the difference and that's what they mean by "baked in".

hope this helps