The Joys of Updating Our Test Suites… Part Deux

Last week, I posted a news item regarding the various methodologies around the site that we’re currently in the process of updating, and since then, a lot has happened. We’re still a little ways off from being finished, but hey, progress is progress. That news item in particular tackled our GPU test suite, and I’m happy to report that we’ve since solidified our choices for what we’ll use going forward.

The six games include Call of Duty: World at War, Crysis Warhead, F.E.A.R. 2: Project Origin, Grand Theft Auto IV, GRID and also World in Conflict. These games were chosen for various reasons, but mostly because they’re all quite popular in today’s scheme of things, and for the most part, none of them favor either ATI or NVIDIA too heavily, so we should be good for a little while… hopefully at least until Windows 7 launches and is well underway.

So with that out of the way, next up is another important test suite… our processors! Admittedly, despite our current test suite being almost two-years-old, our selection has held up well, and I see very little reason for making drastic changes. What I’d rather do is add to the pile, to give an even more robust look at the performance between processors. Of the applications we plan to drop, however, SYSmark 2007 and ProShow Gold have a reason to shake in their boots.

I’m also in the beginning stages of something really cool I hope to add to our future CPU content, but it’s far too early to mention it, because at this point, it might not happen (it’s a wee bit complex). I hope it works out, though, because it’d give us an interesting spin on things. Once again though, and as always, we’re looking for comments, suggestions and whatever else you want to say towards our plans, so don’t hesitate to voice your opinion in our related thread (linked below). After all, we’re writing this content for you guys, not ourselves! Let’s work together to make the content the best it can be!

The first thing I want to drop is SYSmark 2007, because I find it rather useless. The results it delivers don’t scale as they really should, so it’s a bit misleading. I think it’s obvious that a Q9650 is far more powerful than an E8600, but SYSmark doesn’t tell us that. Plus, running the suite is a very patience-testing process. Not to mention, it requires a completely fresh install of Windows. That all on top of the fact that running it twice in a row, even with two iterations, could give differing results, and when we’re dealing with results that range between 1 – 250-ish, any variance can screw up the true scaling.