How do cameras undistort photos?

Question

I have an image data set consisting of pairs of JPG and RAW files. I was tasked with transforming JPG pixel coordinates (x, y) to RAW pixel coordinates (x_raw, y_raw). By comparing RAW and JPG images visually, it is clear that the device undistorts the images at some point of the image processing pipeline (RAW->JPG). Now, to undistort an image, you basically need a camera matrix and parameters of distortion (e.g. radial) that you get by calibration. This means that the device must store these parameters somehow.

My question is: how do cameras perform undistortion and how are calibration parameters derived?

Does each camera sensor get calibrated during the manufacturing process (and the parameters are written to memory of the device)? Or, perhaps, camera sensors of the same model store the same parameters that were derived by averaging parameters of many sensors?

Answer 1

Distortion is produced by the lens, not the sensor. So the distortion correction you describe is also referred to as lens correction.

There are different distortion models (equations). The parameters depend on the specific distortion models used. Different cameras and software may use different distortion models. Camera manufacturers tend to not reveal the distortion models they use. So even if you are able to extract the parameters, they would likely be of little use.

The correction parameters may be taken from a lens model (more equations) that describes the behavior of specific lenses. Or they may be found empirically, by photographing scenes and making measurements. Usually a representative copy of a lens is used. (This appears to be how Adobe and lensfun databases are built.) Results are good enough, and averaging parameters from multiple lenses does not necessarily produce desirable results.

The distortion models used by Hugin are used to perform lens correction in open-source software (eg, lensfun). Adobe appears to use different models. (See What model does Lightroom use for lens correction?)

To solve your problem (the inverse of lens correction), you should be able to use Hugin to calculate the transformation. Another option is to manually map every coordinate, but that is likely to be time consuming and unnecessary for most photographic purposes.

Answer 2

There are different types of distortion but they are always due to the lens. Geometric distortion is often corrected by image processing in the camera to compensate for compromises done during lens design. Those are known at design time for the construction of the lens and usually not by measurements on each individual lens which is why there will often be slight residual distortion, meaning that the optics might have 4% barrel distortion but the corrected image still has 0.2%, for example.

Many cameras have controls enable or disable certain correction. Often Geometric, Vignetting and Chromatic Aberrations. While I usually leave the later two on, I always turn off Geometric because correcting for this has an impact on framing. When the image is undistorted, it will produce a non-rectangular view which gets cropped so that it results in a rectangular image. If you are trying to remove the undistortion, you will be missing parts of the image!

The distortion data is often stored in the camera for each lens. For this reason, often when a new modern lens comes is released, camera makers often release a firmware to support the new lens. The lens will still work without that firmware in most cases but there will be no corrections. There are a few cases where the lens does not work but those are rare and have nothing to do with distortion, usually it is because a new feature are type of focus, control, aperture, etc has introduced.

On modern electronic platforms, it is possible that some lenses could store their own data to avoid having a F/W update needed when a new lens is introduces but that requires more communication between lens and body.

Answer 3

For my Canon 6D I have to choose which lens correction data to download into the camera, for correcting the jpg data. I can download the same data into Photo Professional, for correcting the raw data. I depend on the availability of lens data. Canon only... :-(