Home > Apple > When objectivity gets befuddling: Now Consumer…

When objectivity gets befuddling: Now Consumer Reports loves the ‘hot’ new iPad

When-objectivity-gets-befuddling-Now-Consumer-Reports-loves-the-hot-new-iPad

Consumer Reports has given Apple’s latest iPad the top seat in its ranking of tablet devices, citing the screen as “the best we’ve seen” and indicating it will be revising its standards for excellence in displays based on the iPad’s retina display.

That’s quite a change from two weeks ago, when the organization triggered “heatgate” by reporting the new iPad can get up to 116 degrees Fahrenheit in places when running games — that’s just a smidge below the 120 degrees it usually takes to burn bare skin. Now, Consumer Reports says says the temperatures aren’t a problem and didn’t impact their overall rating of the device.

This isn’t the first time Consumer Reports has seemingly mis-stepped with Apple products or other consumer electronics devices. What’s behind these seemingly contradictory stances, and should everyday technology users start considering Consumer Reports in the same way they consider breathless advertisements from gear-makers — with a grain of salt?

About Consumer Reports

Consumer Reports 1960 television testing

Consumer Reports isn’t some tarted-up effort from a major publisher, nor is it a thinly-veiled marketing effort from product manufacturers hiding behind some allegedly-impartial trade group. Consumer Reports is published by the Consumers Union, a non-profit organization that conducts product research and engages in advocacy on issues it believes are in consumers’ general interest. And it’s not a fly-by-night outfit: The Consumers Union (and Consumer Reports) have been operating since 1936.

The magazine’s ad-free subscription model has always been a selling point. While typical publications that offer product reviews and assessments (including Digital Trends) often earn some portion of their revenue from advertising, Consumer Reports does not accept any advertising in any form. It also doesn’t permit commercial use of its reviews for selling products: If Consumer Reports says something nice about a product, the manufacturer can’t slap that all over its packaging or ads. When Consumer Reports reviews a product, its testers go out and buy it at retail like any other customer. They don’t accept loaners or free items from manufacturers, nor do they use any form of pricing discounts. This is all so Consumer Reports can claim to offer unbiased product reviews.

And Consumer Reports doesn’t just hand products to a reviewer and ask for their impression. Where possible, the magazine designs and conducts laboratory tests in an effort to objectively measure products’ qualities and meaningfully compare them to one another. This task is tougher with some things than others: It’s probably easier to quantify the quality differences between laundry detergents than, say, high-end audio gear. But Consumer Reports puts its money where it’s mouth is, reportedly spending over $20 million a year conducting its tests. A good portion of that money goes into its annual new car issue, published every April.

So about the new iPad…

ipad_thermal_images

Two weeks ago, Consumer Reports figuratively lit a fire under the new iPad, noting that in its tests the latest version of Apple’s tablet could reach temperatures as high as 116 degrees Fahrenheit in places when plugged in and running intensive games — their testers used Infinity Blade II. Consumer Reports also indicated the new iPad didn’t appear to be charging its battery when plugged in and running the game. Consumer Reports noted that the tablet felt “very warm but not especially uncomfortable” if held briefly. Of course, few gamers hold an iPad “briefly,” and sites were quick to note that 120 degrees is typically enough to cause a burn on bare skin. Consumer Reports wasn’t the first to look at the new iPad’s heat emissions — that seems to have been the Dutch site Tweakers. Apple’s previous iPad 2 apparently maxed out at about 111 degrees.

Of course, the technology media are generally comfortable dealing with Apple rumors, so a potentially scandalous story about an over-hot iPad from a source as reputable as Consumer Reports took off quickly. Apple even had to issue a terse statement that the new iPad was “operating well within our thermal specifications.”

However, now the new iPad is at the top of Consumer Reports latest survey of tablets, including rival devices from the likes of Samsung, Sony, Acer, Toshiba, and others. Other recommended tablets were the Pantech Element, Sony Tablet P, Toshiba Excite 10LE, and the Samsung Galaxy Tab 7.7. Consumer Reports concluded that the new iPad’s higher temperatures weren’t, in fact, a major issue.

“Responding to consumer comments on the new device, and to coverage from other reviewers, we also carried out further tests that confirmed the new iPad is warmer in its hottest spots than the iPad 2,” Consumer Reports wrote. “But we didn’t find those temperatures to be cause for concern.”

Consumer Reports further indicated the battery charging issue it previously reported occurred only when playing a “demanding” game with the screen at full brightness.

The iPhone 4s, iPad 2, and iPhone 4

iphone-4-death-grip

This isn’t the first time Consumer Reports has created a stir with its review of an Apple product. Despite giving the iPhone 4 the highest possible score in its smartphone category, Consumer Reports refused to list the iPhone 4 as “recommended” due to signal attenuation issues — the so-called “antennagate” that revealed signal strength to the iPhone can be significantly reduced depending how the device is held. Apple was forced to do significant spin control on the issue, holding a special event demonstrating that a variety of phones from competitors exhibited the same attenuation problems when held certain ways, and even offering free bumper cases for iPhone 4 owners who were experiencing problems. Ultimately, the issue that caused Consumer Reports to withhold its “recommended” rating from the iPhone 4 was an issue that was neither unique to the iPhone 4, nor a significant problem for most users — the iPhone 4 continues to be a top-selling device, and Consumer Reports continues to cite attenuation problems as by far the most significant issue with the device.

Consumer Reports does give its “recommended” rating to the iPhone 4S, claiming the device doesn’t suffer from attenuation issues like the iPhone 4. The iPhone 4S does include antenna-switching capability that enables the device to swap its transmit and receive antennas in the event of signal attenuation — for some folks, that may make the device somewhat more reliable than the iPhone 4.

Consumer Reports did give its “recommended” rating to the iPad 2. However, Consumer Reports also listed the Motorola Xoom, the original 7-inch Samsung Galaxy Tab, and the original iPad as “recommended.”

What does Consumer Reports want?

A tech-savvy reader probably looks at these recommendations and wonders if Consumer Reports is out of its mind. Is the Pantech Element (with Android Honeycomb, a 1,024 by 768-pixel display, and a sub-par camera) really in the same class as the new iPad? When Consumer Reports recommended the iPhone 4S, the organization went out of its way to point out they rated several Android phones more highly, including the Samsung Galaxy S II series, the Motorola Droid Bionic, and even the LG Thrill. Why? The primary criteria for ranking those Android devices higher was that they offered larger displays and (in the case of the Droid Bionic) “excellent keypad readability.”

These sorts of evaluations tend to fly in the face of the rigorous testing on which Consumer Reports prides itself. It’s hard to imagine the organization spends significant amounts of time designing and conducting quantifiable tests and comparing the results only to boils down to these phones are better because they have bigger screens, or the new iPad’s display is so great we lowered the scores for every other tablet’s display. The mixture of allegedly diligent and quantifiable testing and blatantly subjective qualities creates a disconnect. Consumer Reports seems to be stating an opinion, and promoting it to the world as a conclusion derived by scientific methods.

Bottom line

Suzuki Samurai failure test img

It’s important to remember that Consumer Reports is not written for experts. The publication’s primary audience is everyday consumers who are generally uninformed about a variety of high-cost products and services. Consumer Reports looks at everything from airlines and credit cards to healthcare, home appliances, cars and — yes — electronics. The magazine attempts to sum up a huge variety of products and product categories in neat snippets that can be quickly read and understood by users without much knowledge of the field. It’s not an easy task.

The tech savvy have only to look at Consumer Reports’ coverage of automobiles to understand the nature of the disconnect. Consumer Reports has been responsible for some recalls and redesigns of vehicles over the years — the most famous case probably remains Consumer Reports’ 1988 assessment of rollover problems in the Suzuki Samurai, but Consumer Reports auto testing has led to design changes in vehicles from BMW, Chrysler, Nissan, Toyota (via its Lexus brand), and others.

But car enthusiasts — you know, the “petrol heads” that might get featured on a segment of Top Gear — have taken issue with Consumer Reports’ automotive recommendations for years. Each case has its own set of details and points, but the overall disconnect will feel familiar to technology enthusiasts: Many car fans feel that Consumer Reports tests are unfair to particular models or products, or make recommendations based on subjective criteria rather than qualitative facts. Consumer Reports might look at a ton of details, and conclude one model of sedan is better than another because it can seat three adults in the rear seat — ignoring other features  (like a backup camera or availability of a six-speed manual transmission) that other people believe are more important.

We wouldn’t want to suggest buying a tablet is like buying a car, but the point is valid: It’s tempting to think of tablets (or phones, or cars) as one-size-fits-all solutions, but every individual’s situation is unique. Some folks want a 4.3-inch display on their smartphone, even if it means it no longer fits in their pocket; some folks can’t justify the expense of an iPad’s retina display on a device they just want to keep the kids entertained during their commute.

Consumer Reports has no idea what your particular situation might be. It’s testers are trying their best to address what research shows are key issues for many consumers. And, just like the crowd-sourcing of online product review, the conclusions may — or may not — apply to you.

[Consumer Reports 1960 television testing image © 1997 Consumers Union of U.S.]