Comments on: Opinion: Performance benchmarks are worthless, here’s how to make them better

By: Rob Enderle

Rob Enderle — Mon, 28 Jan 2013 16:24:57 +0000

Your teacher isn't a very nice person. But, to point, this is more about discussing beauty contests that are fixed. Contestants are all pretty, one who wins is sleeping with the guy who owns the contest.

By: Rob Enderle

Rob Enderle — Mon, 28 Jan 2013 16:22:22 +0000

Hard to cheat on a benchmark for an analyst, the losing company tends to call foul pretty quickly but it does happen. The bigger issue is who had the most input into setting the benchmark in the first place, and which vendor optimized for the benchmark. Either can result in a score that is higher than it should be. But in the end, you need to measure system performance as a hit processor in otherwise slow system will bottleneck at memory, magnetic storage, or network speed. Vendors do this when buyers get overly focused on one part.

By: Fair Witness

Fair Witness — Mon, 28 Jan 2013 16:13:44 +0000

Simple math. Which vendor paid the analyst more?

By: Mike Inman

Mike Inman — Mon, 28 Jan 2013 00:41:23 +0000

Kid: My teacher says beauty is on the inside
Father: That's just what ugly people say.

AMD... is the ugly people.

By: Rob Enderle

Rob Enderle — Sat, 03 Nov 2012 06:44:50 +0000

Benchmarking is a form of analysis. They aren't mutually exclusive you know. How do you think analysts compare products?

By: Chase Hawisher

Chase Hawisher — Sat, 03 Nov 2012 03:04:08 +0000

You’re missing the point of a benchmark. The ideas you presented are analyses. “For doing x, y is best.” That’s an analysis. “In running eleventy billion iterations of x, y is the fastest.” That’s a benchmark.

By: Rob Enderle

Rob Enderle — Thu, 19 Apr 2012 03:01:04 +0000

I'll give you that for the top 3% of users who still build there own systems like we do, yes I build mine as well, they can be useful. However even for us we often don't measure what we really need only make assumptions that we apply benchmarks to. In fact, even for most of us techies we tend to favor usage based benchmarks like those tied directly to games and applications we use than synthetic benchmarks because we've learned that synthetic benchmarks often don't relate closely to how an application will work or a top game will play. But for the other 97% benchmarks don't relate to what they actually do.

By: Bruce Epper

Bruce Epper — Thu, 19 Apr 2012 02:55:20 +0000

For someone that doesn’t know anything about technology, your idea has weight. But for those of us who understand the numbers and build our own systems, the performance benchmarks of each component we select is chosed because we know how all of them interact and how they will be used in each particular build. They are how we judge what will be most effective for each individual machine. Completely replacing them therefore is out of the question.

By: Rob Enderle

Rob Enderle — Thu, 19 Apr 2012 00:35:34 +0000

Replaced, performance benchmarks are generic and we humans tend not to be.

By: LenMetallica

LenMetallica — Thu, 19 Apr 2012 00:09:05 +0000

So are you saying that performance benchmarks should have no role and be completely replaced by these new ones that you suggest or can both be used together to make an informed decision.

By: Rob Enderle

Rob Enderle — Wed, 18 Apr 2012 02:15:04 +0000

Actually no, I'm asking that we use current technology to create benchmarks that are customized to how we work. We actually do that for race cars and drivers, I think we can scale that with social tech and the web. Agree it won't be easy though.

By: Matt Smith

Matt Smith — Wed, 18 Apr 2012 00:11:42 +0000

Benchmarks provide performance metrics - and nothing more. Some benchmarks are more realistic than others and all benchmarks are applicable to only some situations. Its the job of the reviewer to provide an interpretation of those benchmarks.

What you're doing is asking benchmarks to write the review by themselves and, on top of that, provide long-term data about reliability and value. Which is a bit like asking your mechanic to give you a haircut while you wait for an oil change.

By: Rob Enderle

Rob Enderle — Sun, 15 Apr 2012 21:16:29 +0000

I agree, for most it is increasingly about the eco-system what the hardware does just isn't as important as it once was.

By: jeremy_johnson

jeremy_johnson — Sun, 15 Apr 2012 07:37:54 +0000

Good points, I think another major thing to take in to account is the overall eco-system that a device works on. This is starting to be a lot more relevant with Apple having a full line of devices that all work together. I think Acer is one of the only Android-based companies even comes close to Apple's ecosystem.

Honestly though, I care much more about how all of my devices will work together, than how powerful my processor is.