The Joys of Updating Our Test Suites…

In our news section a few weeks ago, I hinted at the fact that we’d soon be revising our various methodologies and application/game choices for our performance-based articles, and we’re now knee-deep in it, and hoping to wrap things up soon (it’s a time-consuming process, trust me!). As I consider our testing practices and methodologies to be the best out there to begin with, we won’t be making stark changes there, but rather some tweaks and refinements.

What is important to update is our chosen selection of applications and games we use for our processor, graphics card and even motherboard content. Over time, our selection can either begin to stale, or favor one side too heavily, and we have two such cases of that now. Making such changes isn’t a simple process though, and last time I did it, it took me two full weeks (even I think this is ridiculous), but it’s not a process of simply picking and choosing new benchmarks… we’re actually benchmarking each and every one on various architectures to make sure that we have a fair selection.

We of course don’t rule out a particular application benchmark or game if it favors one side, but it would depend on the popularity of the said application. For example, we wouldn’t give Call of Duty: World at War the boot if it happened to favor one GPU team, because it’s an ultra-popular game that people actually play. Essentially, it doesn’t make much sense to benchmark applications or games that no one plays.

That all said, I’m in the middle of revising the game selection for our graphics card content, and I’m not opposed to receiving suggestions on our selection. At this point in time, I’ve dropped all games from our previous selection except for CoD: World at War and Crysis Warhead (I think for obvious reasons). The additions would be FEAR 2, GRID, Grand Theft Auto IV and possibly Tom Clancy’s HAWX. I say possibly only because the FPS rate in that game happens to be sky-high (no pun) as it is, so it could prove to be a pointless title to include.

Check out the thread and toss in your two cents… it’d be appreciated!

Due to the sheer amount of testing involved, I usually try to put off our testing suite for as long as possible, but we clearly are due for a change. Once we have a new game suite, I don’t expect to be able to re-benchmark every card we have that quickly, but I will get as many done as quickly as possible. I’ll mainly focus on current cards though, not last-gen, if I can help it. We manually benchmark each and every card / game configuration, and that’s obviously extremely time-consuming.