Greg's investigations of the DxO sensor test
Greg's photography pages
Greg's photo albums
Greg's photos
Greg's photo index
Groogle

I wrote this some time in 2009. Since then I have changed my opinions on a number of things, and it's possible that I no longer agree with what I have written here. But I haven't had time to check.

DxO Labs have published a web page which, amongst other things, claims to measure the accuracy of the ISO settings of cameras. It would be easy to believe it is as broken as the web site itself is, but in fact the details are quite interesting.

First, the web site breakage. Clearly DxO is a Microsoft-based shop. It seems that they bring out their software for Microsoft first and Apple only a few months later. They're apparently not interested in other operating systems at all. It shows in the rendition of their web pages. Here's firefox on FreeBSD, Safari on Apple Mac OS X, and firefox on Microsoft. For all images, clicking on them will make them progressively larger. The text boxes below the photos are shifted to the right, and they overlap the general text below:


https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/DxO-firefox-FreeBSD.gif
Image title: DxO firefox FreeBSD          Dimensions:          1011 x 823, 134 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Make this image small again Display small version of all images on this page
All images taken on Saturday, 7 November 2009, thumbnails          All images taken on Saturday, 7 November 2009, small
Diary entry for Saturday, 7 November 2009

 
https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/DxO-Safari.gif
Image title: DxO Safari          Dimensions:          1012 x 673, 127 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Make this image small again Display small version of all images on this page
All images taken on Saturday, 7 November 2009, thumbnails          All images taken on Saturday, 7 November 2009, small
Diary entry for Saturday, 7 November 2009

   
https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/DxO-firefox-Microsoft.gif
Image title: DxO firefox Microsoft
Dimensions: 769 x 631, 86 kB
Dimensions of original: 769 x 631, 86 kB
Display this image:
thumbnail    hidden   alone on page
Display all images on this page as:
thumbnails    this size
Show for Saturday, 7 November 2009:
thumbnails    small images    diary entry

This is the page that you get to when you select http://www.dxomark.com/index.php/eng/Image-Quality-Database/Compare-cameras/(appareil1)/294|0/(appareil2)/315|0/(appareil3)/267|0/(onglet)/0/(brand)/Nikon/(brand2)/Olympus/(brand3)/Canon. It isn't the page that I want to link to, but DxO's web programmers have made it impossible to link to the page I'm referring to; instead, go to this page with Safari or Microsoft-based browsers and select the tab “ISO Sensitivity”. With FreeBSD I get a page without the graph. And don't refresh the screen; it'll take you back to the previous page.

So: if DxO software were of the same quality as the web software, I wouldn't give them a second thought. This is serious breakage. But what about their measurements? I was told that my camera, an E-30, was off by nearly 1 EV at 800 ISO, and indeed, that's what the graph shows. The person who mentioned this (on a German web forum) compared the results with the results for the Olympus E-3, which the DxO analyser found correct. I tried it with cameras from other manufacturers, the Canon 50D and the Nikon D90. Result: DxO claims that they all have effectively between 0.5 and 0.8 EV inaccuracy at ISO sensitivities beyond 100 (or 21°). And they all have too high a sensitivity at ISO 100, so that the real sensitivity is the same at ISO 100 and ISO 200 (24°). Well, that's what DxO is claiming:

 
https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/DxO-analyzer.gif
Image title: DxO analyzer
Dimensions: 764 x 474, 17 kB
Dimensions of original: 764 x 474, 17 kB
Display this image:
thumbnail    hidden   alone on page
Display all images on this page as:
thumbnails    this size
Show for Saturday, 7 November 2009:
thumbnails    small images    diary entry

Only see two different colours on the graph? Yes, that's because they're all on top of each other. Here the Nikon appears to be missing; try again and one of the others may be missing instead. Now it's quite possible that all manufacturers have made the same mistake—doubtless a conspiracy?—but this kind of coincidence points to a single cause, and the obvious single cause is DxO's analyzer.

It's easy enough to test, though: take some photos. The following were taken with my E-30 with manual exposure. In each case, the aperture was f/8. In my experience, the accuracy of the aperture is lower than that of the shutter, so for the first two I set the shutter at 1/250 s, and for the third I set it to 1/500s. The first was set at ISO 100, and the other two it was set to ISO 200. If the DxO claims are correct, the first two should look the same, and the third one should be noticeably darker:


https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/ISO-100-normal.jpeg
Image title: ISO 100 normal          Dimensions:          4032 x 3024, 2451 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Make this image small again Display small version of all images on this page
All images taken on Saturday, 7 November 2009, thumbnails          All images taken on Saturday, 7 November 2009, small
Diary entry for Saturday, 7 November 2009 Complete exposure details

 
https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/ISO-200+1EV.jpeg
Image title: ISO 200+1EV          Dimensions:          4032 x 3024, 2499 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Make this image small again Display small version of all images on this page
All images taken on Saturday, 7 November 2009, thumbnails          All images taken on Saturday, 7 November 2009, small
Diary entry for Saturday, 7 November 2009 Complete exposure details

 
https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/ISO-200-normal.jpeg
Image title: ISO 200 normal          Dimensions:          4032 x 3024, 2446 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Make this image small again Display small version of all images on this page
All images taken on Saturday, 7 November 2009, thumbnails          All images taken on Saturday, 7 November 2009, small
Diary entry for Saturday, 7 November 2009 Complete exposure details

 

Clearly that's not the case, and that should enough evidence for me to discredit DxO's claims. But the truth is stranger.

First, the third one is marginally darker than the first. Is there maybe a grain of truth in their claims? These photos were taken with combined JPEG and raw images. The ones above are from the in-camera JPEG. I put the raw images through UFRaw and ran “auto adjust exposure” against them. The results:

The strange thing here is that the raw images don't match the in-camera JPEGs. There's obviously more here than meets the eye. Obviously the images converted with exposure compensation all look the same, and the histograms are pretty similar:


https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/ISO-100-normal-compensated.jpeg
Image title: ISO 100 normal compensated          Dimensions:          4100 x 3084, 2871 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Make this image small again Display small version of all images on this page
All images taken on Saturday, 7 November 2009, thumbnails          All images taken on Saturday, 7 November 2009, small
Diary entry for Saturday, 7 November 2009 Complete exposure details

 
https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/ISO-200+1EV-compensated.jpeg
Image title: ISO 200+1EV compensated          Dimensions:          4100 x 3084, 2824 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Make this image small again Display small version of all images on this page
All images taken on Saturday, 7 November 2009, thumbnails          All images taken on Saturday, 7 November 2009, small
Diary entry for Saturday, 7 November 2009 Complete exposure details

 
https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/ISO-200-normal-compensated.jpeg
Image title: ISO 200 normal compensated          Dimensions:          4100 x 3084, 2907 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Make this image small again Display small version of all images on this page
All images taken on Saturday, 7 November 2009, thumbnails          All images taken on Saturday, 7 November 2009, small
Diary entry for Saturday, 7 November 2009 Complete exposure details

 
 
https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/ISO-100-normal-adjusted-histogram.gif
Image title: ISO 100 normal adjusted histogram
Dimensions: 265 x 262, 4 kB
Dimensions of original: 265 x 262, 4 kB
Display this image:
thumbnail    hidden   alone on page
Display all images on this page as:
thumbnails    this size
Show for Saturday, 7 November 2009:
thumbnails    small images    diary entry
 
https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/ISO-200-+1EV-adjusted-histogram.gif
Image title: ISO 200 +1EV adjusted histogram
Dimensions: 262 x 265, 4 kB
Dimensions of original: 262 x 265, 4 kB
Display this image:
thumbnail    hidden   alone on page
Display all images on this page as:
thumbnails    this size
Show for Saturday, 7 November 2009:
thumbnails    small images    diary entry
 
https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/ISO-200-normal-adjusted-histogram.gif
Image title: ISO 200 normal adjusted histogram
Dimensions: 263 x 261, 4 kB
Dimensions of original: 263 x 261, 4 kB
Display this image:
thumbnail    hidden   alone on page
Display all images on this page as:
thumbnails    this size
Show for Saturday, 7 November 2009:
thumbnails    small images    diary entry

Converting the raw images without compensation gives a very different picture:


https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/ISO-100-normal-raw.jpeg
Image title: ISO 100 normal raw          Dimensions:          4100 x 3084, 2872 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Make this image small again Display small version of all images on this page
All images taken on Saturday, 7 November 2009, thumbnails          All images taken on Saturday, 7 November 2009, small
Diary entry for Saturday, 7 November 2009 Complete exposure details

 
https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/ISO-200+1EV-raw.jpeg
Image title: ISO 200+1EV raw          Dimensions:          4100 x 3084, 2815 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Make this image small again Display small version of all images on this page
All images taken on Saturday, 7 November 2009, thumbnails          All images taken on Saturday, 7 November 2009, small
Diary entry for Saturday, 7 November 2009 Complete exposure details

 
https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/ISO-200-normal-raw.jpeg
Image title: ISO 200 normal raw          Dimensions:          4100 x 3084, 2830 kB
Make a single page with this image Hide this image
Make this image a thumbnail Make thumbnails of all images on this page
Make this image small again Display small version of all images on this page
All images taken on Saturday, 7 November 2009, thumbnails          All images taken on Saturday, 7 November 2009, small
Diary entry for Saturday, 7 November 2009 Complete exposure details

 
 
https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/ISO-100-normal-raw-histogram.gif
Image title: ISO 100 normal raw histogram
Dimensions: 267 x 266, 4 kB
Dimensions of original: 267 x 266, 4 kB
Display this image:
thumbnail    hidden   alone on page
Display all images on this page as:
thumbnails    this size
Show for Saturday, 7 November 2009:
thumbnails    small images    diary entry
 
https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/ISO-200-+1EV-raw-histogram.gif
Image title: ISO 200 +1EV raw histogram
Dimensions: 261 x 265, 4 kB
Dimensions of original: 261 x 265, 4 kB
Display this image:
thumbnail    hidden   alone on page
Display all images on this page as:
thumbnails    this size
Show for Saturday, 7 November 2009:
thumbnails    small images    diary entry
 
https://lemis.nyc3.digitaloceanspaces.com/grog/Photos/20091107/big/ISO-200-normal-raw-histogram.gif
Image title: ISO 200 normal raw histogram
Dimensions: 265 x 264, 4 kB
Dimensions of original: 265 x 264, 4 kB
Display this image:
thumbnail    hidden   alone on page
Display all images on this page as:
thumbnails    this size
Show for Saturday, 7 November 2009:
thumbnails    small images    diary entry

This is the effect that I would expect from DxO's claims, so at least the shape of the graph seems to be correct: there seems to be almost no difference between ISO 100 and ISO 200. Why? Clearly the ISO ratings of raw images are only marginally of interest; the real issue is what comes out when they're converted into JPEG or other open formats, and there the results are pretty much what you'd expect. But it seems that all manufacturers do this. I checked a couple of other cameras, the Canon EOS 1Ds Mark II, the Nikon D3, the Pentax K20D and the Sony Alpha 900, and the results were similar (though the Canon starts at about 80/20° ISO). The Hasselblad H3DII 50 is given as ISO 50/18° in all cases (from 50/18° to 400/27°). Only the Leica M8 is exactly on the line. It's interesting in this connection that the default ISO rating for the E-30 is 200, and that may be related.

In summary, then, the results are interesting, but it's not clear to me what they say about the cameras or the DxO analyser.


Greg's home page Greg's diary Greg's photos Greg's links Copyright information

Valid XHTML 1.0! $Id: DxO-sensor-test.php,v 1.3 2012/08/08 02:18:28 grog Exp $