Benchmarking the Benchmarks?

Kyle Bennett is no stranger to stirring the proverbial pot in the tech industry. He’s a well-known gear-head who’s not afraid to speak his mind – which is one reason he rubs so many people the wrong way. And while I may not agree with everything he says, there is one area where we do agree, and that’s with video card benchmarking.

Over at Hard|OCP, he has just posted a new article explaining their testing methodology, and goes into some depth with regards to specifics. Although we’ve not had many GPU reviews in the past, all of our testing is done manually as well, with the exception for a few timedemos for use with our CPU reviews (since timedemos rely a lot on the CPU). For our GPU reviews, we play through all of the levels manually.

This article came at a coincidental time, because I just spent the past weekend benchmarking five GPUs for upcoming reviews, with three more left on the table. I admit, playing the same level over and over and over gets tedious, but a tall Guinness or short Heineken works wonders!

The article is worth a look if you want a another opinion on why timedemos are not the way to do things. But, it will all come down to personal preference, and there is no denying that most of the time, timedemos are somewhat accurate. One interesting finding, though, is that even though most reviews for the AMD HD 3870 X2 show the card to be faster than the 8800 GTX… it turns out that real-world, that’s not the case.

That comparison also couldn’t come at a better time, since that was an identical scenario I will be dealing with later today or tomorrow. I am interested to see if the numbers all scale with his. I’ll post in the news once I have some findings.

That is not to say that synthetic and canned benchmarks do not have their places in testing, we just don’t usually find those metrics to be indicative of what the end user has in terms of actual experiences. Some website’s want to tell you the “relative performance of a graphics card” based on a timedemo that in no way represents playing the game. That is not what we want to focus on here at HardOCP.

Source: HardOCP