If you want to know how good a smartphone camera is, DxOMark conducts the most in-depth reviews around. But after five years of evaluating smartphone cameras, the photography lab is introducing a major update to its test suite, adding two new categories — zoom and bokeh — and ramping up the importance of low-light and motion performance.
Since 2012, DxOMark has tested every phone with the same protocol, covering categories like exposure and contrast, color, autofocus, texture, noise, artifacts, and flash. The results are explained in great detail and then the scores from each category are brought together and a proprietary algorithm reduces them down to a single number out of 100. The top spot in the mobile charts, currently held by the HTC U11 with a score of 90, is highly coveted.
“Every single mobile that has been tested between 2012 and now, you can compare, because it followed the same protocol,” Clement Viard, senior director of image quality evaluation at DxOMark told Digital Trends. “But technology has changed so rapidly in the last five years that we have had to change the protocol.”
The impact of this change can be summed up neatly by looking at the iPhone 7 and 7 Plus. Because the old protocol didn’t consider the dual camera that Apple put in the 7 Plus to achieve 2x optical zoom and artificial bokeh, via Portrait mode, the two devices effectively had the same score. Retested with the updated suite, the iPhone 7 Plus achieves a significantly higher score of 88, compared to 85 for the iPhone 7.
DxOMark believes this change to its testing was necessary because of the growing demand from the public for high-quality zoom and bokeh effects, and improvements in the technology that are making it possible. The combination of powerful processors, dual-camera systems, improved sensors, and clever software tricks is taking smartphone photography to a whole new level.
“From the beginning of mobile photography, people at Nokia, Samsung, and other manufacturers were all asking the same thing,” Etienne Knauer, DxOMark’s senior vice president for sales and marketing, business solutions, explained to Digital Trends. “We want to be able to put in optical zoom, help us, and we want to have blurred background for portrait, help us. After all these years, now it’s becoming true.”
In addition to the new zoom and bokeh sub-scores, DxOMark has placed greater emphasis on low-light performance, testing levels all the way down to just 1 Lux. The Parisian photography lab has also added a motion element into its test suite to more accurately approximate real-world conditions, where both the photographer and subject are often moving.
For each smartphone tested under the new protocol, DxOMark will shoot 1,500 photos and more than two hours of video. The result is a 100-page report that serves as the foundation for the reviews and scores you see on the DxOMark website. The old scores will remain in archived form, but every phone tested by DxOMark from today will be subject to the new testing protocol.
Only a small sample of smartphones tested on the old scale have been retested with the new protocol. DxOMark chose the HTC U11 (90), Google Pixel (90), Apple iPhone 7 (85) and 7 Plus (88), Samsung Galaxy S6 Edge (82), iPhone 6 (73), and Nokia 808 (61).
“The ones that we’ve tested that were the best phones according to the old ranking are still the best devices on the new ranking,” Viard explained. “The Google Pixel and HTC U11 were really good before, and they’re still really good. For the iPhone 7 Plus, adding bokeh and zoom has given a big jump in image quality, which we believe reflects the user experience. Older devices like the iPhone 6 and Nokia 808 are dropping significantly, because they don’t have zoom or bokeh, and the autofocus was much slower back then.”
DxOMark will be announcing a number of new scores in the coming days and weeks, but for now, HTC’s U11 remains joint top of the chart on 90 with the Google Pixel, which has climbed one point. With the new iPhone and Google’s next Pixel just around the corner, it will be interesting to see if the photography lab crowns a new device as the best mobile camera.
