How does automatic HDR software work?

Question

An HDR image is the combined image of varied exposures that give good detail of the subject in the photograph. How does HDR software know what part of the images to blend?

Manually, we know by looking at the image which part needs to come from which exposure, but do these software do simply show the composite image with medium exposure from all the varied exposed images?

I ask this because I use LuminanceHDR for making HDR images and the results seem very poor. It would help me to understand what the software can do automatically and where a human eye is needed.

Answer 1

There are two distinct steps to producing the images that are frequently labelled "HDR":

• Exposure blending: merging multiple low dynamic range images into one image with higher dynamic range.

• Tonemapping: processing that high dynamic range image into a low dynamic range image suitable for viewing on standard [low dynamic range] equipment (such as regular computer monitors).

There are a number of ways to perform the first step but basically you need to estimate the difference in exposure for all of the images. This enables you to take a pixel value from any image and convert it to a consistent value in your final image. For example if you have two exposures that are 3 stops apart, multiplying values in the darker image by 8 will give values consistent with the brighter image (3 stops means three doublings of brightness, or 8x).

A simple strategy for exposure blending could be to take pixel values from the brightest image unless the values get close to overexposure, in which case you switch to the next brightest image, and so on until all pixels are converted. Most software will use a more sophisticated method, perhaps by using a weighted combination of values from close exposures to minimise noise and discontinuities. Note there might not be a simple multiplicative conversion possible between images (due to nonlinear response of the camera sensor, or nonlinear tonecurves) which complicates the conversion, some alignment of images may also be necessary.

The key point is that the exposure blending part is not subjective, there ought to be one correct way to blend images together. The problem is that you would need a high dynamic range monitor in order to appreciate a high dynamic range image. If you simply scaled the values down to fit a standard display you'd end up with a very flat image as there would be little contrast between nearby tones.

The image above was produced by blending three exposures and then scaling the values in a globally linear way. As you can see there is no over or under exposure in the image, even in the bright skies and the shadows of the pillars detail is retained. However the image is dull looking overall.

This is where tonemapping comes in. It squashes the dynamic range down to fit in an locally adaptive manner, taking into account image content in order to maximise the contrast locally and retain detail. This is the subjective part, as there are many ways to reduce the dynamic range. It's also the hardest part to get right as if you vary the contrast too much over too small an area you will get artifacts such as halos around edges that are derided by critics of HDR images.

Here's the same image after tonemapping, the contrast has been boosted in the darker regions in order to make the most of your monitors limited dynamic range to show detail and texture.

This image required a lot of tweaking and I'm still not happy with the output. The problem with tonemapping is that it is very easy to push it too far and end up with huge amounts of contrast locally, and very little globally, i.e. the image is very textured but has little overall structure. Most software has some sort of adjustment for the radius of the local adjustments, the lower the radius the greater the chance of producing a fake looking result. Colour saturation can suffer and should also be reigned it carefully.

note there are pieces of software such as Enfuse that blend images directly to a low dynamic range image, negating the need for two steps. I've not used any such program so I can't tell you how they work!

Answer 2

There is a simple HDR workflow:

1. Shoot images with manual exposure, only changing exposure time.

2. If the images are in linear RAW format, you are good to go. If the images are in JPEG format, you need extra processing to undo nonlinear processing made by the camera - namely convert colors from sRGB to linear RGB - this include applying inverse gamma curve.

3. Convert all images to floating-point format (i.e. having pixel values in interval 0.0-1.0 instead of 0-255).

4. Divide each image by its exposure time (i.e. divide pixel value by 4, if the exposure time of the image was 4s).

5. Combine all the images together (every pixel value is an average of corresponding pixel values from all images).

6. Global tone mapping: Apply curve such that most detail is visible when HDR image is converted back to traditional 8 bit per pixel format. A good type of curve is a logarithmic curve, which suppresses bright tones in favor of dark and mid-tones. This is how our eye perceives light - in a logarithmic manner, not linearly like a CCD sensor.

7. Local tone mapping: Instead of 6., you can apply special tone-mapping operators to attenuate contrast in the image so that when converted to 8 bits, high-contrast details (e.g clouds vs roof) will not cause suppresion of low-contrast features (e.g. texture in shadow)

8. Convert all images back to 8 bit RGB (or 16 bit if you wish). This is done simply by linear mapping of interval 0.0-1.0 back to 0-255.

There are numerous extra steps and details that HDR software does for you, for example discrimination of overexposed/underexposed pixels that does not carry useful information and would spoil the look of final HDR image because of step 4.

Another, more simple HDR workflow is just combine all the images together like layers in photoshop. A special blending mechanism called exposure fusion creates final image as weighted average of input images, but it does so by weighting each pixel according to its saturation, contrast and "well-exposedness". The blending is done over several frequency bands to combine both small details and large features.