Why does a newer camera get lower score for a specific lens than an older one on dxomark?

Question

I was looking around on DxOMark and found discrepancy in sharpness scores which seems very weird.

The lens Olympus M. Zuiko Digital ED 12mm f2.0 has a sharpness score of "10 P-Mpix" when mounted on a E-PL5, while for the newer E-PL7 it is only 5 P-Mpix.

How can a newer camera get a score which is so much lower than an old camera for the same lens? Shouldn't the sensor be better?

Seems the result is similar for other lenses when tested on these cameras.

Can I really be sure, if I buy a camera which is supposed to be good but has not been tested yet at DxOMark, that it actually is not much worse than the previous versions?

Answer 1

You are absolutely right to ask this question. Regardless of the technique used to derive the scores, the scores should be identical or very close between the two bodies using essentially the same lens. Both cameras are 16MP cameras, and the lens is the M.Zuiko 12mm. To me the 2x difference shows gross sloppiness on the part of DxO, not some horrible deficiency between the cameras.

This is but one more example why you should not use DxO test values. Instead look at the results photographers have produced in the real world with those cameras and that lens, such as on Flickr or SmugMug.

Answer 2

My understanding is that like most lens testers/reviewers other than LensRentals/Roger Cicala, DXO tests only one copy of a lens at a time. How long they keep them or whether they return or sell them I don't know. But my suspicion is that they owned different copies at the time they tested the E-PL5 and E-PL7, and the latter was a bad copy.

Answer 3

First thing that comes up in my mind is that later sensors have higher resolutions so reveal more imperfections of a specific lens.

If dxomark translates it's technical and absolute score to a more understandable MP indication you could end up with these kind of discrepancies.

That said I have no clue how dxomark gets MP scores.

Answer 4

While the "trend" is that newer gear usually yield better results, in some measurable form for DxOMarks, exceptions do exist such as the newer Nikon 24-70/2.8 with VR versus the Nikon 24-70/2.8G lens see here.

While I don't work for, nor defend DxoMarks, it should be notes that making the goal of making newer bodies/lens may not always be sharpness in my example. It could be that they wanted to add VR and in doing so, sharpness is reduced for the lens as a whole. VR is a very helpful tool and in adding it, something had to be taken away or compromised.

As other have noted, making lens is a dance of difference compromises, especially faster lens such as the f2.8 zooms or even top end primes. I can only assume that in making the E-PL7, the designers has some goal and either using a sensor that could resolve images better wasn't in the equation or even the coating on the anti-aliasing filter reduces sharpness. Perhaps a firmware update could fix it, I really can't say.

The only thing I can say is if you really want to test it, rent it or borrow it and see for yourself. If the sharpness is noticeable and you do care about it in the images, don't upgrade.