Opinion: Performance benchmarks are worthless, here’s how to make them better

Performance-benchmarks-are-worthless-Here’s-how-to-make-them-better

AMD is getting ready to launch its next-generation architecture (code named Trinity) and invited a bunch of us out to Austin to see it. I can’t talk about the technology until it is launched, but one of the events at the show was a head-to-head comparison between this AMD technology and Intel’s top-shelf products.In each test (including productivity, video enhancement, and file compression) AMD Trinity technology wasn’t just faster, it was substantially faster.

Though the demonstration was impressive,it also reminded me why benchmarks really aren’t that useful anymore. Not only do they fail to reflect what each of us individually do, they don’t factor in cost, device size, or design, each of which might be more important than any direct performance measure.

For instance, Apple hasn’t led benchmarks in years. Side-by-side with competitors, the iPad and iPhone actually tend to appear relatively slow (they often use older networking, processor, or storage technology). They are also relatively expensive, yet lots of folks still prefer them, suggesting benchmarks as they currently exist are worthless to these buyers. They rank other things higher.

So what would a perfect benchmark look like?

How do you work?

The perfect benchmark would be derived from an ongoing analysis of how you use your hardware. We all change as we age, and even change what we do from day to night,from weekdays to weekends, and on vacation, so the capture should occur over a period of time.

It should also look for critical points, like what annoys us and what thrills us — not only in terms of what we are doing, but what we are talking about. In short, factor in our social-networking activity in things like Facebook and Pinterest.

Finally it would rank all aspects of our interest and factor in cost, not only the cost of buying the product, but the cost in time of putting the product into service, maintaining it, and our sensitivity to down time.

Analyzing the device

Since it has proven impractical to go into a store and run a benchmark on a shelved PC, and impossible to do the same thing if we want to buy online, the ideal benchmark would also need to capture the performance of systems on the market. Against this objective data, it would also capture subjective data on design, expected reliability, and time to obsolescence. While the latter two could come from historic data (much like Consumer Reports does with its ranks), the design analysis would be based on what someone similar to you in terms of personality type and taste would rank the product.

Finally, given that we live in an online “cloud” world, a major portion of the data captured would need to be on the services the device connected to, the apps it would load, and the overall end-to-end user experience.

In the end, everything would be mathematically rendered.

The result

The result would be accessible on a site where you could go, log in, and specify either the type of product you were looking for, or enter a number of products you were looking at. The system would then give you a set of choices listing the key analytical elements of each. So if you saw something that wasn’t current, or you didn’t agree with, you could change the element and thus change the ranking.

You could see an overall ranking of around 10 products with some specific ones flagged: the lowest priced, the best match to you, and the most balanced (best value for the money as defined by your unique needs and tastes). This is also somewhat similar to what Consumer Reports tries to do, but more advanced.

You would end up with a list of top choices that would be more likely to thrill you. It could also analyze products you already own to flag when performance degraded to a point that would begin to irritate you, or when the extra performance of a new system was great enough to make it worth it for you – specifically based on your needs.

Benchmarks don’t have to suck

When I first ran into benchmarks, Intel was complaining that it built systems that were betterrounded, while AMD was using benchmarks to drive people to systems they would like less. Intel tried to get the industry to drop the benchmarks, failed, and now largely optimizes for benchmarks.

If you focus on what people want to do, you’ll provide a better experience, but still likely get slammed by benchmarks. At AMD’s event, the company was pointing to the reasons benchmarks suck.

I think the answer here is to create benchmarks that don’t suck. We have online tools that capture a ton of information about us to sell to advertisers, so it doesn’t seem to be such a stretch to use some of this technology to create a tool that makes us happier consumers. Considering all this information is compiled about usand should belong to us, it would be really nice if it were used to make us happier, rather than just milk us for money. This would be a way to do that. What do you think?

[Image credit: kk-artworks/Shutterstock]

The views expressed here are solely those of the author and do not reflect the beliefs of Digital Trends.

Computing

We tested Nvidia’s RTX 2080 and 2080 Ti. Are they a worthy upgrade?

We finally have Nvidia's newest graphics cards, the RTX 2080 and 2080 Ti, and put them through our standard suite of benchmarks and game tests to see how they performed in a standard play of Battlefield 1 and Fornite. How do they compare to…
DT Daily

Dyne sits at the intersection of fashion and technology, creative director says

While we've seen companies like Nike use technology to improve its products and push boundaries, Dyne is doing something a little different. We sat down with Chris Bevans of Dyne on this episode of DT Daily to talk fashion and technology.
Smart Home

Cloud slime, butter slime, scented slime -- we have your next DIY project covered

We researched dozens of different ingredients and tried several slime recipes. While some recipes are generally pretty safe, others are questionable at best. Which recipes should you use, and which should you avoid?
Deals

The best MacBook deals for November 2018

If you’re in the market for a new Apple laptop, let us make your work a little easier: We hunted down the best up-to-date MacBook deals available online right now from various retailers.
Computing

Want to save a webpage as a PDF? Just follow these steps

Need to quickly save and share a webpage? The best way is to learn how to save a webpage as a PDF file, as they're fully featured and can handle images and text with ease. Here's how.
Computing

Installing fonts in Windows 10 is quick and easy -- just follow these steps

Want to know how to install fonts in Windows 10? Here's our guide on two easy ways to get the job done, no matter how many you want to add to your existing catalog, plus instructions for deleting fonts in the process.
Computing

After a month of bugs, the Windows 10 October 2018 Update is finally here

After more thoroughly investigating and resolving all bugs and related issues, Microsoft is announcing the Windows 10 October 2018 is again rolling out to consumers starting today, November 13. 
Computing

Changing file associations in Windows 10 is quick and easy with these steps

Learning how to change file associations can make editing certain file types much quicker than manually selecting your preferred application every time you open them. Just follow these short steps and you'll be on your way in no time.
Emerging Tech

New simulation shows how Elon Musk’s internet satellite network might work

Elon Musk has the dream of building a network for conveying internet traffic via thousands of satellites. A new simulation created by a computer scientist looks at how feasible the idea is.
Computing

A dead pixel doesn't mean a dead display. Here's how to repair it

Dead pixel got you down? We don't blame you. Check out our guide on how to fix a dead pixel and save yourself that costly screen replacement, or an unwanted trip to your local repair shop.
Computing

Great PC speakers don't need to break the bank. These are our favorites

Not sure which PC speakers work best with your computer? Here are the best computer speakers on the market, whether you're working with a tight budget or looking to rattle your workstation with top-of-the-line audio components.
Computing

Printing to PDF in Windows is easy, no matter which method you use

Microsoft's latest operating system makes it easier than ever to print to PDF in Windows, but there are alternative methods for doing so, even if you want to forgo Adobe Acrobat. Here's how.
Computing

These are the 5 best free antivirus apps to protect your MacBook

Malware protection is more important than ever, even if you eschew Windows in favor of Apple's desktop platform. Thankfully, protecting your machine is as easy as picking from the best free antivirus apps for Mac suites.
Computing

These laptop bags will keep your notebook secure wherever you go

Choosing the right laptop bag is no easy feat -- after all, no one likes to second-guess themselves. Here are some of the best laptop bags on the market, from backpacks to sleeves, so you can get it right the first time around.