How to compare 2 cameras' color accuracy quantitatively with standard color chart?

Question

I need to compare the color reproduction accuracy of 2 cameras quantitatively, in RGB space or Lab space. In my set-up, I use the same standard lighting source, same lens(same focal length and same aperture), use the one-time auto white balance for the 2 cameras.No gain for both cameras. The images of a Macbeth color chart are taken by these 2 cameras, then the color values of each color patch are compared with the ground truth value.

But my question is: the exposure time for each camera is adjusted to avoid over-exposure, thus different for two cameras. The color intensity of the images from these 2 cameras depend on the exposure time. Then how can I compare the color difference ?

Answer 1

While I don’t think a minor difference in shutter time will be a major influencing factor, I suggest setting both test cameras to the same f-number and shutter speed, in manual mode. Should you observe differences in the exposure levels, mount neutral density filters on the camera that yields the greater exposure. For many years I managed an operation that made test films for the photofinishing industry. These were super accurate as to color and density. They were used worldwide to calibrate high speed photofinishing color printing machine, so I have some experience in these matters.

Use ND filters to balance the camera with the greater exposure to the other. I suggest you use Kodak Wratten Neutral Density filters. These are of high quality and as neutral as you can get. Some should be available on the web or you can procure others.
ND filters are labeled in density units. These seem cryptic however the values are exponents of base 10 logarithmic notation.

0.0 = No density change used to balance a system keeping the number of filters used the same.

0.025 = 1/12 f-stop = 1.06X

0.05 = 1/6 f-stop = 1.12X

0.10 = 1/3 f-stop = 1.26X

0.15 = 1/2 f-stop = 1.41X

0.20 = 2/3 f-stop = 1.58X

0.30 = 1 f-stop = 2.0X

Let me add -- Use constant voltage transformer to power the lighting system. The slightest voltage change will greatly effect the color output of your lighting system.

Answer 2

There is a somewhat dry ISO standard procedure for doing this experiment that you can reference. Identical exposures is not required for the test.

ISO 17321-1 Colour characterization of digital still cameras (DSCs)

"Fixed exposure settings shall be selected to provide peak output levels between 50 % and 90 % of saturation"

The reason is that in the color difference algorithm, both your colorchecker values and the ground truth values are normalized by the exposure of the white test patch. They don't need to be the same.

I didn't read the entire color difference algorithm but effectively somewhere it will normalize the values so that the CIE Y tristimulus value of the white patch is 100, and everything else is relative to that.

However, you might be interested in knowing that most color difference algorithms come with a luminance independent color error metric, that only measures chroma differences. These metrics should work even if the source colors have different luminances:

• CIE 1994: ΔE*94 , which includes Luminance L*, and ΔC*94 , which omits L*.

• CMC: ΔECMC , which includes Luminance L, and ΔCCMC , which omits L. Widely used in the textile industry for matching bolts of cloth.

• CIEDE2000: ΔE00 , which includes Luminance L*, and ΔC00 , which omits L*. This is the emerging standard as well as the most accurate color difference metric. Its acceptance has been slowed by the complexity of its formula. Although it is less familiar that the other equations, it is the best choice in the long run.

Whichever metric you choose, remember that they give different numbers. It is important to be consistent and always specify which measurement you are using.

https://www.iso.org/standard/56537.html (paid) http://www.gamutvision.com/docs/gamutvision_equations.html

Answer 3

Maybe do mot adjust anything at all. Set both cameras to manual, use very same setup and very same lens, if possible.

If you compare, say, Canon 700D to Canon 5D mark III, use one EF lens for both bodies. If you compare, say, Canon 700D to Nikon D5300, try to use vintage M42 lens with adaptors for Canon and Nikon mounts.

If you have all apperture, shutter and ISO set to same values you should have simillar images. When there are differences in resulting RAW image, there are differences in either sensor sensitivity and in-camera processing. If all the other parameters are kept the same.

With the very same postprocessing, you can directly compare the results.

Answer 4

Assuming you compare RAW files, comparing raw color rendition on two cameras makes no sense. Every sensor has a different response to color and light, and none is perfectly linear. In real life (JPGs files, photo editing softwares), the RAW values are adjusted through a calibration profile, which corrects the color deviations of the sensor, based on curves or lookup tables, to match real colors. No sensor (in cameras as in industry) is ready to work as-is, you always have to calibrate it. (Full disclosure : I'm a trained metrologist).

Assuming you compare JPEGs files, again using the same settings makes no sense (different sensors, different integrated softwares, different response). Usually, the L value of the white patch of a color chart is between 96 and 98. Just make an exposure bracketing with both cameras and retain only the files whose white patches are the closest to the expected value. In other words, compare the pictures wich have the same white level, thus the same actual exposure.

BUUUT… what you will compare will NOT be the actual color accuracy of the sensors but the accuracy of the color profiles used to calibrate the sensors, inside or outside the camera. So again, I don't know why you would need to compare the color accuracy of two cameras, but it sounds useless to me since you can make your own profiles (it's just software) and since the sensor raw data is always to be adjusted anyway.

Some other specifications might be more relevant to compare, such as the noise level in RGB channels or the color depth. More details on DXO : https://www.dxomark.com/About/Sensor-scores/Use-Case-Scores