How do you crown the best smartphone camera? The experts share their secrets

learn how dxomark scores smartphone cameras fugem 7

They say the best camera is the one you have with you, and most of us are never without our phones.

But how do you know which smartphone has the best camera? Every manufacturer claims to have a great camera, they throw around megapixel counts and aperture sizes, but none of this tells you how good your shots will be in the real world.

Many of us turn to independent photography lab, DxOMark, for in-depth reviews that boil down the camera quality of each flagship smartphone to a single number out of 100. Since the Parisian company began to test mobile phone cameras in 2012, it has been the gold standard. The scores it bestows are trusted enough to be widely referenced in the tech press.

“When Google achieved top marks [for the Pixel], they covered London in ads because they were proud,” Clement Viard, Senior Director, Image Quality Evaluation at DxOMark told Digital Trends.

Collectively we shared around 2.5 trillion photos online last year and 90 percent of them were captured on smartphones, according to a report from Deloitte. That number is going up every year. The Millennial generation is particularly interested in mobile photography, with 24 percent saying it’s the most important function of their smartphone, and 57 percent saying it’s very important, according to Osterman Research.

Little wonder, then that a good DxOMark is so coveted. But how does the firm arrive at these widely quoted scores?

“Today, the differentiator for the phone is probably the software more than the hardware,”

“For smartphones, we always use the auto mode. We take the phone out of the box, turn it on, start the camera app, and take pictures,” explains Viard. “Camera users are used to tweaking settings, but smartphone users expect to be able to just push a button, so the testing has to be completely different.”

Every smartphone camera is subjected to the same battery of tests, broken down into sections that cover things like exposure and contrast, color, and autofocus. Each camera is meticulously tested with a mix of specialized equipment in the lab.

“We also have to get out of the lab and test in the real-world. We go to the same spots around Paris, around the same time of day. We try to find places where the colors will always be the same, where there is dark and light to see how it handles shading.”

They take 1,500 photos with every smartphone camera that’s tested and then break them down and grade them. To account for changes in light and weather, they also snap shots with a reference device for comparison, so they can be sure they’re grading the phone and not the weather. Each photo is cross-checked by at least two experts from the 40-person team. The result is a 100-page report for each review, highlights of which are published on the DxOMark website.

Viard says they’re usually in agreement, but when there’s a difference of opinion the rest of the team will be called in for a debate. There’s no doubt they take the scoring system seriously, but separating opinion from objective fact can be tricky. You’ll often see people saying that Samsung’s Galaxy phones are oversaturated, for example, but some people prefer the way they look. Where is the line on quality?

learn how dxomark scores smartphone cameras fugem 4

“There was a lot discussion in trying to work out what would be the best way to assess color quality,” explains Viard. “We take pictures of a scene with colors and we map colors into a color space that will look at saturation and bias. What we found is that if the distance between the real color and the color from the camera is within an area, then it’s really a matter of taste. Different people can argue for weeks and they’re not going to come to the same conclusion.”

What DxOMark does in this situation is to stop factoring it into the score beyond a certain point. It does the same thing at the bottom end of the scale. There’s a cut-off point where a photo is so bad that you’ll discard it anyway.

Because DxOMark also offers consulting services and specialized equipment to help manufacturers to improve their camera performance, it has been subject to occasional claims of bias. The idea being that manufacturers who engage its consulting services will get a better score.

“The exact score calculation is confidential and the reason we keep it confidential is because we don’t want manufacturers trying to optimize for the score,” says Viard. “We are very aware that the way we design the score formula may influence some choices companies make, but the testing protocol will be exactly the same for every single phone, with all the same objective metrics, real world results, and human grading.”

DxOMark recently updated its testing protocol for the first time, to add two new categories to every score: Zoom and bokeh. The new protocol also places greater emphasis on low light conditions, and DxOMark introduced more motion into its testing, to see how cameras perform when both the subject and photographer are moving. This is all intended to better reflect what consumers want and the real-world conditions we take photos in.

learn how dxomark scores smartphone cameras fugem 6

The new testing protocol has been a long time in the making. DxOMark is keen to ensure that it only tests features that really matter to people. It considered the impact of 3D capabilities a few years ago, but decided it was a fad. It’s already testing front-facing cameras with the rise in popularity of selfies, and watching 360-degree camera technology very carefully, but it will be a while, if ever, before these make into the score.

“We always try to focus on usage rather than technology,” says Viard.

Apple’s iPhone 7 Plus bokeh effect, where the subject is in sharp focus and the background is blurred, is a good example, as it sparked a demand for bokeh and got everyone talking about it.

“There are many ways of creating the bokeh effect,” Etienne Knauer, Senior VP, Sales and Marketing, Business Solutions, DxOMark explained to Digital Trends. “But you require at least two different images, so with a single camera, you don’t know it, but actually you’re taking multiple images, even if it’s from the same sensor, or they use a dual camera. You need two to make an estimation of the depth and then software blends them.”

Under the old test protocol, there was no distinction between the single-lens iPhone 7 and the dual-lens iPhone 7 Plus, but the introduction of zoom and bokeh, enabled the 7 Plus to creep up the chart with a score of 88, while the iPhone 7 scored 85. The rescored Google Pixel managed 90 on the new scale, putting it joint top with the HTC U11 , but there are some very big phones on the horizon that might challenge the current champions.

“We only test the flagships and the very best phones,” says Viard. “We look at the specs and what the manufacturer is saying to find the phone where they’ve put most emphasis on photography and that’s the one we will pick.”

Sadly, DxOMark lacks the resources to test every phone that comes out, though Viard says they’d like to. There’s no doubt that the field has widened in the last couple of years.

“For the first few years Apple, Nokia, Sony, or Samsung was always the best. Today everybody is putting emphasis on the camera. HTC, Huawei, OnePlus, and Google have made huge improvements to the camera, so the number of companies we test has grown.”

If you’re into smartphone cameras, you’ll know that Sony makes most of the sensors that the flagships use, yet its own phones don’t tend to score as highly as other handsets with the same sensor.

“Today, the differentiator for the phone is probably the software more than the hardware,” says Viard. “Camera tuning makes such a big difference. You can have two manufacturers with exactly the same hardware, even the same chipsets, but the image quality is very different because of the algorithm.”

Even with the enormous processing power smartphone cameras now possess and all the clever software tricks, there are some things a DSLR does better. DxOMark’s bokeh test, for example, is designed to catch out these tricks. It’s a face with triangles that go out to a fine point, wires on the same plain, and lots of details and broken images in the background, to see if the camera gets confused and blurs the wrong parts, which it invariably does.

“On a DSLR, if it’s on the same plain, it will have the same sharpness,” says Viard. “Smartphone cameras have a long road before they reach DSLR level, but I think the gap is closing, the progress has been huge in the last five years.”

All the major manufacturers are investing serious resources into improving camera technology, trying to make it easier for average people to capture perfect shots. That’s great news for us, because at the end of the day, mobile photography is all about capturing precious memories. If you want to know which smartphones will give you the best chance of catching those spontaneous moments, then it’s always worth checking out what DxOMark has to say.

Product Review

The iPad Pro is the best tablet ever. But don't sell your laptop just yet

Apple has unveiled a big redesign for the iPad Pro, slimming down the bezels, adding Face ID, and the ability to attach and charge the Apple Pencil. All of this comes at a high cost however, as the iPad Pro starts at $799.
Mobile

LG G7 owners experiencing dreaded bootloop will have to wait a few days for fix

If you’ve picked up LG’s flagship and it’s not behaving itself, then you might find a solution here. We’ve rounded up the most common LG G7 ThinQ problems and tracked down workarounds and possible fixes.
Deals

Liven up the holiday party with the 10 best white elephant gifts for $20 or less

We tracked down 10 awesome gifts with the perfect mixture of humor, practicality, and most importantly, affordability. Take a look at the best white elephant gifts under $20 and get ready to get your white elephant on.
Gaming

PS4 vs. Switch: After weighing the pros and cons, which one comes out on top?

Nintendo Switch versus PlayStation 4: Which one has better overall value? We break down the pros and cons of each platform to tell you which of these consoles is truly worth the money.
Mobile

New Apple Watch begins saving lives one week after getting EKG feature

The Apple Watch Series 4 was updated almost a week ago with a new feature that allows users to take electrocardiogram tests -- and already it seems like the new feature is saving lives.
Mobile

The best smartphone stocking stuffers for a very techy Christmas

If you've got a tech-loving smartphone enthusiast to buy for, we can help you out. Here's a selection of top phone accessories that would make amazing stocking stuffers so you can have a very Merry Christmas.
Product Review

The Black Shark gaming phone takes a big bite out of your free time, but the software sinks it

The world is being treated to an ever-increasing number of high-powered gaming phones. With so many great options already out, is there room for another? The Black Shark thinks so. But is it any good? We find out.
Mobile

OnePlus's 5G phone should arrive in May 2019, may cost up to $850

OnePlus will be among the first companies to put the new Snapdragon 855 processor into a phone, and will also release a separate and more expensive 5G phone in 2019 with the help of U.K. network EE.
Mobile

Santa Claus is coming to town. Get ready with these Christmas apps

Like it or not, Christmas comes but once a year. Thankfully, we've got a list of the best Christmas apps to help get you into the holiday spirit, just in case you want to call Santa or become a Christmas elf.
Gaming

Xiaomi’s Gamepad 2.0 turns the Black Shark into a mini Switch

Chinese phone manufacturer Xiaomi has introduced the Gamepad 2.0 for its Black Shark gaming phone that launched earlier this year. The new gamepad is similar to the Switch Joy-Con, with one piece attaching to each side of the phone.
Mobile

Google makes it easy to donate to charity straight through the Play Store

As we head into the holiday season, Google announced a new feature on Google Play that makes it a little easier to donate to charity. Through the new page, you'll be able to choose from a range of charities to donate to.
Deals

REI clearance sale extends discounts on Garmin, Fitbit, and GoPro devices

Beyond the things you typically expect to find at REI — like tents, skis, and jackets — there are tons of great deals on quality tech foryour outdoor adventures. From smartwatches to action cameras, here are the best tech deals.
Wearables

These are the best smartwatches for everything from fashion to fitness

Tempted to buy a smartwatch? If so, then the growing number of great models available means you've got plenty to choose from. But which one should you pick? Here is our list of the best smartwatches.
Mobile

Sprint’s 5G rollout: Everything you need to know

Sprint is building its next-gen 5G network in preparation for a 2019 rollout, but it's taking a decidedly different approach than some of its competitors, including Verizon and AT&T. Here's everything you need to know.