Alley in Pisa, Italy

by Lars Kotthoff

submit your photo


Hall of Fame
View past winners from this year

Please participate in Meta
and help us grow.

Take the 2-minute tour ×
Photography Stack Exchange is a question and answer site for professional, enthusiast and amateur photographers. It's 100% free, no registration required.

Bayer sensors use a pattern of red, green, and blue pixels, and merge those together into a final color image, with one pixel for every single color sensor. This could be done through a "naive" blending of neighbor sensors, but I've heard of more complicated approaches with names like AHD, HPHD, and AMaZE.

What are these other approaches, and what advantages do they bring? Do they have weaknesses beyond compute complexity?

I imagine that the approach used for in-camera JPEG is more tightly guarded, but clearly a lot of research and development goes into this area. Does the limited processing power available in-camera force any compromises in this area?

share|improve this question
    
I could be wrong but I was under the impression that the demosaiacing occurs in camera otherwise the image would suffer from aliasing. There was an interesting article about this in one of the recent Popular Photography talking about the Sigma (Foveon X3 sensor) which is one of the few cameras that does not have a bayer sensor. –  Jakub Apr 18 '12 at 16:06
1  
Well, a quick bit of investigation indicates AHD, or Adaptive Homogeneity-Directed Demosaicing, is the "industry standard", and seems to be used by ACR/LR (at least as of a couple versions ago...who knows if they have introduced something more advanced with ACR 6.x and LR4.x). It seems like a weighted algorithm that aims to reduce false color. –  jrista Apr 18 '12 at 16:08
4  
@Jakub: Demosaicing occurs in-camera for JPEG images. The entire point of a RAW image is that it has NOT been demoasiced yet, and is the "raw" recording of pixel data directly off the sensor without any additional processing (outside of your basic amplification to achieve the necessary ISO.) An optical low-pass filter (AA filter) eliminates aliasing by physically "blurring" spatial frequencies below the nyquist rate of the sensor. –  jrista Apr 18 '12 at 16:09
    
Right. Makes sense, thanks. –  Jakub Apr 18 '12 at 16:22
add comment

4 Answers 4

up vote 7 down vote accepted
+50

I was surprised a few months ago to find that my machine vision camera SDK used nearest neighbor "interpolation" in its built-in bayer function. It is the fastest, but worst type, giving hard edges, especially when you start doing math on the image channels for colour constancy or brightness invariance. I found this review of algorithms:

http://www.arl.army.mil/arlreports/2010/ARL-TR-5061.pdf

The next step up is bilinear and bicubic interpolations which can be computed quite fast because they amount just to convolution kernels. These give coloured saw-teeth on slanted edges - bilinear more than bicubic.

Can be seen in this paper, and with quantified quality data on 5 different algorithms:

http://research.microsoft.com/en-us/um/people/lhe/papers/icassp04.demosaicing.pdf

This is why they made edge direction based interpolations. However, these treat green as a more "important channel" (since it has the best resolution and account for most of our visual sensitivity and our eyes' resolution). And then they create blue and red as a function of the green channel, in a hue preserving fashion. This in turn makes green channel high frequency content more prone to errors. Complexity is higher as they have to detect what's going on and require multiple passes. Moire and maizes are common artifacts from these types of interpolation.

Here they show examples of Adaptive Homogeneity Demosaicing and bilinear versions with and without hue preserving and edge preserving addons:

http://math.auburn.edu/~kilgota/ahd_bayer.pdf

That paper favours AHD and doesn't show the negative part. On this page you can see the different pattern artifacts from Adaptive Homogeneity Demosaicing, Patterned Pixel Grouping, and Variable Number of Gradients (hover mouse over the names):

http://www.pages.drexel.edu/~par24/rawhistogram/40D_Demosaicing/40D_DemosaicingArtifacts.html

In summary there are a number of assumptions employed in these algorithms, and artifacts occur when the assumption doesn't hold:

  • Per channel smoothness. If the nearest neighbors are not the same, make the transition smooth. Artifact: saws/zippers, softness
  • Brightness constancy with directed edges (bilinear direction). Artifacts: high frequency texture moire, colour fringing
  • Hue constancy. if in a neighborhood the hue is the same, so if one channel changes the others have to follow. Artifacts: colour zippers on colour edges
  • Constancy can be predicted from the greens. Artifact: maizes
share|improve this answer
add comment

I started playing with this a bit, and found that the naive approach isn't all that bad. That's simply treating each color separately and interpolating to get the in between pixels. The main downside to this is if you're pixel peeping at a place where there is high contrast, you can see a little color fringing. Put another way, if you have a light gray area abutting a black area, you will see a few colored pixels at the boundary. Fortunately these average out in general, but if the edge is nearly vertical or nearly horizontal they average out over a low frequency. The same effect can be even more obvious on thin bright lines that are nearly vertical or horizontal.

Here is a example. This picture was taken deliberately as a test shot:

Note the apparent banding of the line of chrome trim. To put this in perspective, here is the full frame:

I have thought about a alternate approach but so far there have always been other things to do first. This scheme would seek to find just the brightness first. This would be the single channel of image data if the image were black and white. Each sensel contributes some to that, although the colors don't contribute equally. Once the intensity is determined, you would then interpolate the colors as in the naive method, but use the result only to set the colors in such a way as to preserve the intensity. Intensity would have higher bandwidth, or be sharper in photographic terms than the hue information. Analog TV used this trick to reduce the bandwidth requirements of a color image. They got away with it because the human visual system puts a higher importance on intensity than colors, particularly red.

Anyway, those are just some thoughts. Like I said, I haven't actually tried that yet or worked out the details. Some day.

share|improve this answer
add comment

It is my understanding that the different Process Versions in Lightroom (up to now we have 2003, 2010 and 2012) correspond among other things to different demosaicing algorithms. Another interesting software is UFRaw which offers the following (quote from the webpage):


After setting the white balance UFRaw interpolates the Bayer pattern.

  • AHD interpolation is the Adaptive Homogeneity-Directed interpolation. It is the default interpolation.
  • VNG interpolation uses threshold-based Variable Number of Gradients interpolation. This used to be the default interpolation and it is still very good.
  • VNG four color interpolation should be used if you get Bayer pattern artifacts in your photo (see DCRaw's FAQ for more details).
  • PPG interpolation stands for Patterned Pixel Grouping interpolation. It is almost as good as all of the above and much faster.
  • Bilinear interpolation is a very basic interpolation but it is much faster.

This could provide some material for experimenting. By the way, UFRaw appears to be Open source, which lets you peek at the algorithms themselves.

share|improve this answer
add comment

In astrophotography, this topic is investigated deeply, because when using a one-shot color sensor, there's a lot of lost resolution by debayering. On the upside, using a RAW file still gains access to the original data and it can be processed prior to the color application. This topic strays closely to the software side of things.

In short, if you have access to lots of images with the same subject data (which is something done to reduce sensor noise), you can trade off a single frame conversion with AHD for a drizzle approach which can recover the lost resolution. The choice depends on what kind of source data you have available. Most photographers only have a single image to use.

Some software that I've used with choices for Debayering processes are: Deep Sky Stacker and Pix Insight. There are others as well. Many are based on DCRAW.

Here's the link for the article on the Deep Sky Stacker page where they discuss some of the options: Debayering Choices

share|improve this answer
1  
This is interesting (thanks!) but I don't think it answers the question. That last link seems promising but focuses on alternatives to demosaicing at all. On the topic at hand, it says only "A lot of different interpolation methods are available producing bad to good results (linear, gradient...) but all are degrading the quality of the final picture by guessing what the missing colors should be." But it doesn't go any further into the practical details. –  mattdm Apr 18 '12 at 17:33
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.