Opinion: Performance benchmarks are worthless, here’s how to make them better

Performance-benchmarks-are-worthless-Here’s-how-to-make-them-better

AMD is getting ready to launch its next-generation architecture (code named Trinity) and invited a bunch of us out to Austin to see it. I can’t talk about the technology until it is launched, but one of the events at the show was a head-to-head comparison between this AMD technology and Intel’s top-shelf products.In each test (including productivity, video enhancement, and file compression) AMD Trinity technology wasn’t just faster, it was substantially faster.

Though the demonstration was impressive,it also reminded me why benchmarks really aren’t that useful anymore. Not only do they fail to reflect what each of us individually do, they don’t factor in cost, device size, or design, each of which might be more important than any direct performance measure.

For instance, Apple hasn’t led benchmarks in years. Side-by-side with competitors, the iPad and iPhone actually tend to appear relatively slow (they often use older networking, processor, or storage technology). They are also relatively expensive, yet lots of folks still prefer them, suggesting benchmarks as they currently exist are worthless to these buyers. They rank other things higher.

So what would a perfect benchmark look like?

How do you work?

The perfect benchmark would be derived from an ongoing analysis of how you use your hardware. We all change as we age, and even change what we do from day to night,from weekdays to weekends, and on vacation, so the capture should occur over a period of time.

It should also look for critical points, like what annoys us and what thrills us — not only in terms of what we are doing, but what we are talking about. In short, factor in our social-networking activity in things like Facebook and Pinterest.

Finally it would rank all aspects of our interest and factor in cost, not only the cost of buying the product, but the cost in time of putting the product into service, maintaining it, and our sensitivity to down time.

Analyzing the device

Since it has proven impractical to go into a store and run a benchmark on a shelved PC, and impossible to do the same thing if we want to buy online, the ideal benchmark would also need to capture the performance of systems on the market. Against this objective data, it would also capture subjective data on design, expected reliability, and time to obsolescence. While the latter two could come from historic data (much like Consumer Reports does with its ranks), the design analysis would be based on what someone similar to you in terms of personality type and taste would rank the product.

Finally, given that we live in an online “cloud” world, a major portion of the data captured would need to be on the services the device connected to, the apps it would load, and the overall end-to-end user experience.

In the end, everything would be mathematically rendered.

The result

The result would be accessible on a site where you could go, log in, and specify either the type of product you were looking for, or enter a number of products you were looking at. The system would then give you a set of choices listing the key analytical elements of each. So if you saw something that wasn’t current, or you didn’t agree with, you could change the element and thus change the ranking.

You could see an overall ranking of around 10 products with some specific ones flagged: the lowest priced, the best match to you, and the most balanced (best value for the money as defined by your unique needs and tastes). This is also somewhat similar to what Consumer Reports tries to do, but more advanced.

You would end up with a list of top choices that would be more likely to thrill you. It could also analyze products you already own to flag when performance degraded to a point that would begin to irritate you, or when the extra performance of a new system was great enough to make it worth it for you – specifically based on your needs.

Benchmarks don’t have to suck

When I first ran into benchmarks, Intel was complaining that it built systems that were betterrounded, while AMD was using benchmarks to drive people to systems they would like less. Intel tried to get the industry to drop the benchmarks, failed, and now largely optimizes for benchmarks.

If you focus on what people want to do, you’ll provide a better experience, but still likely get slammed by benchmarks. At AMD’s event, the company was pointing to the reasons benchmarks suck.

I think the answer here is to create benchmarks that don’t suck. We have online tools that capture a ton of information about us to sell to advertisers, so it doesn’t seem to be such a stretch to use some of this technology to create a tool that makes us happier consumers. Considering all this information is compiled about usand should belong to us, it would be really nice if it were used to make us happier, rather than just milk us for money. This would be a way to do that. What do you think?

[Image credit: kk-artworks/Shutterstock]

The views expressed here are solely those of the author and do not reflect the beliefs of Digital Trends.

Computing

How the Google Stadia could lead to a new era of multi-GPU gaming

Google's Stadia could use more than one graphics card to deliver the high-performance visuals it's promised. If that leads to better developer support for multi-GPUs, could that mean gaming with two or more graphics cards could finally be…
Movies & TV

Quentin Tarantino’s Once Upon a Time in Hollywood: Everything we know so far

Quentin Tarantino's ninth feature film, Once Upon a Time in Hollywood, uses the infamous 1969 Manson Family murders as a backdrop to tell a story set in bohemian Los Angeles. Here's everything we know so far.
Deals

From Air to Pro, here are the best MacBook deals for March 2019

If you’re in the market for a new Apple laptop, let us make your work a little easier: We hunted down the best up-to-date MacBook deals available online right now from various retailers.
Emerging Tech

Desk lamps take on a new task by converting their light to power

What if we could charge devices using light from indoor sources like desk lamps? A group of scientists working on a technology called organic photovoltaics (OPVs) aim to do just that.
Deals

From Chromebooks to MacBooks, here are the best laptop deals for March 2019

Whether you need a new laptop for school or work or you're just doing some post-holiday shopping, we've got you covered: These are the best laptop deals going right now, from discounted MacBooks to on-the-go gaming PCs.
Computing

Still miss Windows 7? Here's how to make Windows 10 look more like it

There's no simple way of switching on a Windows 7 mode in Windows 10. Instead, you can install third-party software, manually tweak settings, and edit the registry. We provide instructions for using these tweaks and tools.
Deals

Looking for a Chromebook? The Google PixelBook just got a $200 price cut

Once relatively obscure, Chromebooks have come into their own in a big way in recent years. One of our favorites is the super-sleek Google Pixelbook, and it's on sale right now from Amazon for $200 off, letting you score this premium laptop…
Computing

Go hands-free in Windows 10 with speech-to-text support

Looking for the dictation, speech-to-text, and voice control options in Windows 10? Here's how to set up Speech Recognition in Windows 10 and use it to go hands-free in a variety of different tasks and applications within Windows.
Computing

Get the most out of your high-resolution display by tweaking its DPI scaling

Windows 10 has gotten much better than earlier versions at supporting today's high-resolution displays. If you want to get the best out of your monitor, then check out our guide on how to adjust high-DPI scaling in Windows 10.
Mobile

Got gadgets galore? Keep them charged up with the 10 best USB-C cables

We're glad to see that USB-C is quickly becoming the norm. That's why we've rounded up some of the better USB-C cables on the market, whether you're looking to charge or sync your smartphone. We've got USB-C to USB-C and USB-C to USB-A.
Computing

Nvidia’s GTX 1650 graphics card could be just a slight upgrade over the 1050 Ti

Rumors suggest Nvidia might soon launch the GTX 1650, and a leaked benchmark listing from Final Fantasy XV suggests that the new graphics card could be just a slight upgrade over last generation's GTX 1050 Ti. 
Computing

Get ready to say goodbye to some IFTTT support in Gmail by March 31

If This Then That, the popular automation service, will drop some of its support for Gmail by March 31. The decision comes as a response to security concerns and is aimed to protect user data.
Computing

Get the new Dell XPS 13 for $750 with this limited-time deal

Dell is currently running a limited time deal lasting through Thursday, March 28, where you can bring home a version of this year's new XPS 13 for around $750 with the use of a special coupon code. 
Mobile

This is the easiest way to save your iPhone data to your computer

Living in fear of losing your contacts, photos, messages, and notes on your iPhone? Fear no more -- in this guide, we'll break down exactly how to back up your iPhone to your computer using Apple's iTunes or to the cloud with iCloud.