Opinion: Performance benchmarks are worthless, here’s how to make them better

Performance-benchmarks-are-worthless-Here’s-how-to-make-them-better

AMD is getting ready to launch its next-generation architecture (code named Trinity) and invited a bunch of us out to Austin to see it. I can’t talk about the technology until it is launched, but one of the events at the show was a head-to-head comparison between this AMD technology and Intel’s top-shelf products.In each test (including productivity, video enhancement, and file compression) AMD Trinity technology wasn’t just faster, it was substantially faster.

Though the demonstration was impressive,it also reminded me why benchmarks really aren’t that useful anymore. Not only do they fail to reflect what each of us individually do, they don’t factor in cost, device size, or design, each of which might be more important than any direct performance measure.

For instance, Apple hasn’t led benchmarks in years. Side-by-side with competitors, the iPad and iPhone actually tend to appear relatively slow (they often use older networking, processor, or storage technology). They are also relatively expensive, yet lots of folks still prefer them, suggesting benchmarks as they currently exist are worthless to these buyers. They rank other things higher.

So what would a perfect benchmark look like?

How do you work?

The perfect benchmark would be derived from an ongoing analysis of how you use your hardware. We all change as we age, and even change what we do from day to night,from weekdays to weekends, and on vacation, so the capture should occur over a period of time.

It should also look for critical points, like what annoys us and what thrills us — not only in terms of what we are doing, but what we are talking about. In short, factor in our social-networking activity in things like Facebook and Pinterest.

Finally it would rank all aspects of our interest and factor in cost, not only the cost of buying the product, but the cost in time of putting the product into service, maintaining it, and our sensitivity to down time.

Analyzing the device

Since it has proven impractical to go into a store and run a benchmark on a shelved PC, and impossible to do the same thing if we want to buy online, the ideal benchmark would also need to capture the performance of systems on the market. Against this objective data, it would also capture subjective data on design, expected reliability, and time to obsolescence. While the latter two could come from historic data (much like Consumer Reports does with its ranks), the design analysis would be based on what someone similar to you in terms of personality type and taste would rank the product.

Finally, given that we live in an online “cloud” world, a major portion of the data captured would need to be on the services the device connected to, the apps it would load, and the overall end-to-end user experience.

In the end, everything would be mathematically rendered.

The result

The result would be accessible on a site where you could go, log in, and specify either the type of product you were looking for, or enter a number of products you were looking at. The system would then give you a set of choices listing the key analytical elements of each. So if you saw something that wasn’t current, or you didn’t agree with, you could change the element and thus change the ranking.

You could see an overall ranking of around 10 products with some specific ones flagged: the lowest priced, the best match to you, and the most balanced (best value for the money as defined by your unique needs and tastes). This is also somewhat similar to what Consumer Reports tries to do, but more advanced.

You would end up with a list of top choices that would be more likely to thrill you. It could also analyze products you already own to flag when performance degraded to a point that would begin to irritate you, or when the extra performance of a new system was great enough to make it worth it for you – specifically based on your needs.

Benchmarks don’t have to suck

When I first ran into benchmarks, Intel was complaining that it built systems that were betterrounded, while AMD was using benchmarks to drive people to systems they would like less. Intel tried to get the industry to drop the benchmarks, failed, and now largely optimizes for benchmarks.

If you focus on what people want to do, you’ll provide a better experience, but still likely get slammed by benchmarks. At AMD’s event, the company was pointing to the reasons benchmarks suck.

I think the answer here is to create benchmarks that don’t suck. We have online tools that capture a ton of information about us to sell to advertisers, so it doesn’t seem to be such a stretch to use some of this technology to create a tool that makes us happier consumers. Considering all this information is compiled about usand should belong to us, it would be really nice if it were used to make us happier, rather than just milk us for money. This would be a way to do that. What do you think?

[Image credit: kk-artworks/Shutterstock]

The views expressed here are solely those of the author and do not reflect the beliefs of Digital Trends.

Emerging Tech

Death from above? How we’re preparing for a future filled with weaponized drones

Drones are beginning to enable everything from search & rescue, to the delivery of medicines to hard-to-reach places. But they are also being used as cheap, and deadly flying bombs. How can we defend ourselves?
Computing

Core i9s and Threadrippers are all powerful, but should you go AMD or Intel?

The battle for the top prosumer CPUs in the world is on. In this head to head, we pit the Core i9 versus the Threadripper to see which is the best when it comes to maximizing multi-core performance on a single chip.
Computing

Leaked benchmarks suggest rumored AMD GPU could be king of midrange graphics

AMD's next GPU may not be Navi-based after all. Rumors continue to build about an RX 590 which has now shown up on 3DMark's benchmark database, delivering results that easily outstrip stock clocked GTX 1060s.
Computing

Updated Intel processor benchmarks still beat AMD Ryzen competitor, but by less

After some controversy, updated Principled Technologies testing shows the Intel i9-9900K with a reduced lead over the AMD Ryzen 2700X in benchmarks, and with the AMD Ryzen 2700 X seeing better performance. 
Computing

There’s now proof that quantum computing is superior to the classical variety

For the first time in computer science history, researchers have tangibly demonstrated how a quantum computer is better than a classical computer. A quantum computer was able to solve a math problem that a classical PC cannot.
Computing

In 2018, the rivalry between AMD and Intel has become more interesting than ever

When it comes to selecting a CPU for your PC, there's no shortage of chips for you to choose from. With Ryzen, Threadripper, and Core i9 CPUs though, the AMD vs. Intel argument is muddier than ever.
Product Review

Dell’s G3 Gaming laptop knows what gamers want, and what they can live without

Compromise and budget gaming laptops go hand-in-hand, but with the G3, Dell has figured out how to balance what gamers want with what they can live without.
Computing

Will Apple introduce a new MacBook at its Oct. 30 event? Here's everything we know

Whether it's called the MacBook Air or just the MacBook, Apple is highly rumored to introduce a new, affordable laptop in 2018. We discuss reports about upgrading displays, processors, sign-in features, and more.
Computing

Apple CEO demands Bloomberg retract its Chinese surveillance story

Apple CEO Tim Cook is calling on Bloomberg to retract a story alleging that Apple had purchased compromised servers that allowed the Chinese government to spy on Apple. Apple's investigation found no truth to the story.
Product Review

Amid a new fleet of budget laptops, the ZenBook 13 sails where others sink

It’s never been truer that you don’t need to spend over a thousand bucks to buy a good laptop. The ZenBook 13 takes we’ve always loved about its predecessor and makes enough small refinements to keep it ahead of its competitors.
Computing

Protect your digital identity with these four easy steps to online anonymity

You don't have to be a secret agent or a notorious hacktivist to care about anonymity. Consult this guide to learn tips, tricks, and best practices for staying anonymous and keeping your online activity private
Computing

Gaming on a laptop has never been better. These are your best options

Gaming desktops are powerful, but they tie you down to your desk. For those of us who prefer a more mobile experience, here are the best gaming laptops on the market, ranging from budget machines to maxed-out, wallet-emptying PCs.
Computing

How does Samsung's new 2-in-1 stack up against the tried-and-true Surface Pro 6?

We're the stacking Surface Pro 6 and Galaxy Book 2 up against each other in this head-to-head. Both of these devices share a similar form factor and design, but be it LTE connectivity or difference in pricing, each offers up its own pros…
Social Media

How to turn off Safe Mode in Tumblr

If you've joined Tumblr after hearing tales about the social network's more adult communities, you may be disappointed by how family-friendly it seems. Here's how to turn off "Safe Search" in Tumblr and delve into the site's seedy…