How about high ISO (low light) performance testing of sensor with camera's auto-ISO option?

Question

I'm in the market for a camera with better high-ISO capability than my current ones. There are more than enough reviews on the internet that measure cameras' high ISO performance. They usually pit one camera against another at particular ISO.

However, I don't think such method is just. It is well known that camera bodies have different "real" ISO sensitivity than their nominal one, and some brands have tendency to inflate the ISO number. So in order to assess the high-ISO performance in real life situation, you need to consider the camera's "effective" ISO, which is also available at some review sites, like DXOMark.

This makes me wonder. Why don't people run high-ISO performance test by using the camera's own auto-ISO mode, with the aperture and shutter speed fixed at the same parameter for each camera? If camera A's ISO is so superb that it only needed 800 in the same available light while others needed 1600, it's not fair to compare camera A's ISO 1600 against other cameras' ISO 1600. You should forget the number and compare at the ISO that was necessary for each camera. This method does depend on the camera's own auto exposure metering, but I think auto exposure meter has less discrepancy between brands than the actual ISO sensitivity. This method will catch two birds with one stone too: you can see which brand tends to jack up the ISO, and you can truly see how the camera does at the given low light situation when the aperture and shutter speed are fixed and the camera is asked to do the best it can with its high ISO.

Answer 1

There are several reasons we don't do that. The primary issue when testing cameras is isolating variables and that requires making parameters which are not being tested the same. Even so, everyone's definition of the same is actually not the same! Essentially, this is the point of your question: You are questioning that for ISO sensitivity.

Not all cameras have a TAv mode and, even when they do, each camera imposes its own limitations such as the maximum sensitivity and ISO step-size which may end up using values which are more noisy than necessary. Metering systems differ greatly too. Even if two digital cameras have the same sensitivity, a different ISO may be chosen. Contrarily to your assumption, metering varies considerably more than ISO.

Answer 2

You are correct that ISO numbers are in general not comparable in most cases you can't rely on the camera metering to do the same thing between different camera brands

There really is nothing stopping a camera manufacturer adding an extra zero to all the ISO value displayed onscreen without changing anything else in the sensor, and then making the claim "our camera produces cleaner files when set to ISO 1000 than brand X does when set to ISO 160", despite the exposure times being the same between the two!

However I don't think you can easily get round the problem by using auto-ISO, as neat as that solution may be, you'll run into problems with the camera's metering system's having different behaviour.

The best testing methodology is to set the shutter speed and aperture* to the same value for all camera models and then try several different ISO settings, choosing the one that gets closest to clipping the brightest part of the scene (without overexposing). The results should then be normalised during RAW conversion to obtain identical (or close to) brightness values in the final image.

---

Going further down the rabbit hole it's worth noting that the relative sensitivity of each colour channel varies between sensors, sometimes but as much as a stop. So one camera's blue channel at ISO 100 might be the same sensitivity as a different camera's blue channel at ISO 200, even if their green channels are identical in terms of sensitivity.

The test as outlined above should be repeated three times for each colour channel. As you can see it is quite difficult providing a really fair test.

* As lens focal length and aperture values are generally only correct (or almost correct) at when the lens is focussed to infinity, the test should also be carried out using an infinitely large test scene, for example The Universe.