How relevant are the DxOMark scores and tests?

Question

To me the DxOMark sensor rating seems a bit bizarre. They recently rated the sensor of Nikon D3300 to the same score as the Canon 1Dx got which I find unlikely to reflect their performances, and which also made me really skeptical of their scoring system.

I do value their measurements of the sensor, which results in a benchmark tool for comparing different camera sensors, but are they really relevant? As far as I know there is no documentation of how the score is computed from Dynamic Range, Color Sensitivity and Low-light ISO scores. Also DxO Labs' interest lies in selling their software, not directly in giving scientific benchmarking tools and could very well be biased towards a certain manufacturer.

Just how useful is the DxOMark scores and how can you use it to compare different cameras?

Answer 1

DXOMark primary "scores" are utterly useless. IGNORE THEM. It is a futile effort to try and reduce a complex entity such as a DSLR to a single, scalar number that tells you everything about it. It's a fallacy. There are too many factors to consider, and which factors are most important for a given photographer differ. A single score entirely defeats the purpose of running measurements in the first place.

When it comes to DXO's other scores, such as low light and landscape and whatnot, take them with a hefty dose of salt. Their general scores are heavily weighted, and often based on derivations of measurements rather than actual measurements. For example, the Landscape score is based on the Print DR "measure". The problem is that DXO does not actually MEASURE Print DR, in that it is not based on samples taken from actual downsampled images. Print DR is a simple mathematical extrapolation from the TRUE measured dynamic range of the sensor.

Therefor, Print DR does not really tell you anything about the sensor. When DXO says the D800 and D600 have 14.4 stops of DR, that is Print DR, which is extrapolated from the actual hardware DR, which is 13.2 stops. Same thing goes for Canon sensors. When DXO tells you the sensor has 12 stops of DR, that really isn't the case. In reality, most Canon sensors, at a hardware level, have around 10.95 stops of DR.

The problem is worse than this, however. Much of the color depth and color sensitivity scoring information is weighted, as are many of the ISO-based scores. Weights are based on cameras achieving certain thresholds, such as the SNR at a certain ISO being higher than a certain level. This grants a certain "bonus" to the score for that camera. The moment any kind of weight-based bonus scoring enters into the game, your ability to directly compare anything by any score completely goes out the window. Your now on a non-linear playing field where you honestly don't know if that Nikon camera over there with a score of 95 has been heavily weighted relative to this Canon camera over here with a score of 80.

When it comes to actual measurements, DXO information is some of the best available. Their measures of SNR, Screen DR, color sensitivity, etc. are quite sound, as it's all taken directly from multiple RAW images samples for each camera tested. Their testing methodology is fairly rigorous, and there is nothing to indicate that part of their methodology should be doubted. Scientifically, as far as how they test and what they measure, DXO has solid practices and solid information.

DXO is really a mixed bag. They may have solid testing practices, but their scoring, given that it is often based on mathematically derived, weighted information and the fact that a couple of their scores are often given "bonus" points simply for meeting certain thresholds, completely debases the entire point of what DXO does: To produce a linear score for each camera that allows cameras to be easily compared. It was a flawed concept to start with, single-number scoring, but they made it so much worse by how they handle the actual scoring process.

Answer 2

Overall Score

I generally ignore the Overall Score, as it's way too general if you understand any of the individual scores.

The Overall Score is a function of a variety of (fairly) deterministic tests, each of which is quite informative, and most (if not all?) have clear units of measurement. But then they generate a "score" that combines these metrics, with different dimensions. It's kind of like comparing one car to another, by adding their maximum acceleration (m/s/s), the size of their fuel tank (L), their top speed (km/h) and the number of passengers they can carry. Everyone's going to want to weight the different components differently, so the overall score becomes fairly irrelevant.

Use Case Scores

As you say, the Canon 1DX got the same Overall Score as the Nikon D3300, but note some big differences, even just on the summary "Scores" page:

• "Landscape" score (aka Dynamic Range) is 11.8 vs 12.8 EVs (1 stop better on D3300)
• "Sports" score (aka Low-Light ISO) is 2786 vs 1385 ISO (1 stop better on 1DX)

These "Use Case Scores" are already much more specific, and dimensionally sensible, and make far better comparisons.

That said, they are also

1. not necessarily easy to understand, and
2. not necessarily a useful measurement for all applications

For example, the "Sports / Low-Light ISO" use case is

low-light ISO is the highest ISO setting for a camera that allows it to achieve an SNR of 30dB while keeping a good dynamic range of 9 EVs and a color depth of 18bits.

These chosen values are arbitrary, but the useful part is they're used consistently to measure all the sensors in the same way. This means that while you're only looking at a single value, you're at least able to compare apples to apples. How well do sensor A and sensor B compare, for one particular data point. It's a useful comparison because sensors all tend to perform better at lower ISOs, and all tend to have similar drop-offs as you increase ISO, etc. But you're really relying on that similar performance across all sensors for this to be a useful general purpose comparison.

Measurements

If you go into the "Measurements" section of the comparison, you'll start to see some more useful comparisons. Lots of data, at lots of different conditions. That's where you can (kind of) start to answer questions like "How much less ISO noise would I get from camera A vs camera B, at ISO6400?". Or, if you already know that you are OK with your current camera up to ISO 1600, then you can use the SNR for your camera at ISO 1600 as a baseline for comparing other cameras (in this case, there'll be a similar amount of image noise in a 1DX at ISO 3200 as there would be in a D3300 at ISO 1600). Well, even that's not quite true, since the SNR data is for 18% grey!

Comparison of performance between complex devices, with many dimensions/degrees of freedom is an inherently very difficult problem. You can often compare individual tests quite well, but the problem is finding generalisable tests that quickly and easily portray relative or absolute performance. I think the "Use Case Scores" achieve this, to a large degree, but only because the technology of most sensors is quite similar, letting you make generalisations like the one above about the 1DX being "one stop better" for noise in low light. (Imagine if sensor noise wasn't a simple function of ISO for all sensors!)

Bear in mind also that one sensor outperforming another isn't necessarily useful. The ISO performance (SNR 18%) of a D3300 and a 1DX is essentially irrelevant when shooting a JPEG of a bright, daylight scene. More useful will be things like the dynamic range (for shadow/highlight detail with harsh shadows). And even then, shooting JPEG, you won't get much more from a larger dynamic range (there's a bit of tone compression to form the JPEG but both are still capable of well over the 8-bit dynamic range of a JPEG). To repeat the car analogy, it's like comparing maximum speed & acceleration between cars for city commuting. You'll never hit the limits of any 'fast' car, so its kind of irrelevant for that application. You really need a good sense of the specific application if you want to compare two sensors in a meaningful way, and that's where you want to know:

1. The actual measurements, for comparison (not just overall scores)
2. What measurements are useful for the application
3. What the limit of usefulness is for a particular measure of performance

Summary

The Overall Score is pretty useless, except as a really general guide to someone's weighting of use-cases (not necessarily yours!)s.

The Use Case Scores are much better guides for a few general performance trends between different sensors.

The Measurements let you make your own comparisons, if you know what to compare, and have an application in mind. It's also important to know how much is "enough", or when you get "diminishing returns" for any given application.

Unless you really understand electronics, optics, physics... the absolute numbers are probably pretty meaningless.

Answer 3

I trust DXoMark about as far as I can throw them, and I'm not particularly good at shot put. They are an interesting source of information about lab conditions and can be occasionally useful when comparing cameras from the same manufacturer, but there are too many variables in there tests that depart from real world conditions to make them generally useful for comparing cameras, particularly from different manufacturers.

Just like any other camera review site, real world performance is always more important than laboratory conditions. If you know how to read the information and figure what is and isn't relevant to the real world, you can make some meaningful data out of DxOMark, but in my experience, it is best to take it with a large grain of salt.