Techgage logo

Techgage’s Testing Methodologies

Date: August 17, 2009
Author(s): Rob Williams

Are you curious about how we handle all of the testing in our labs? Well, good news… this is the article you’re looking for. While still a work-in-progress, we so far explain how we go about installing Windows Vista, and also delve into how we benchmark for our graphics card content. In the future, all of our methodologies will be found here, in great detail.



Introduction, Installing Windows Vista

Here at Techgage, we take what we do very seriously, and that’s most true when it comes to our performance-related articles, such as our reviews for processors, graphics cards, motherboards and so forth. If you’ve been frequenting our side of the web for a while, you’re probably already well aware of this. What this article hopes to accomplish, though, is to give you guys a deep look at how we conduct our testing, so you can see just what it takes for us to deliver accurate and concise content, and also for your personal use, if you want to compare your personal results to ours.

This is an article I’ve wanted to get written for quite a while, because I feel that if everyone out there knows exactly how we handle things in our labs, you’ll be able to have utmost confidence in our results. After all, that’s the most important thing, alongside our professional analysis.

That said, this article is not yet fully complete, as most of our methodologies are not quite updated to where we want them to be. So, this effectively acts as a base, and it will be updated whenever we revise any aspect of our testing methodology, or want to add a bit more information we find important. If we do a major overhaul on one of our performance methodologies, then we’ll simply overwrite the respective page. There will be no “Part Two” to this article. Rather, it will just be continually updated when needed. Whenever we make major updates, we’ll republish the article to appear on the front-page, and use a news post to explain all of what was done.

We’ve always been open to change and ideas from our readers, and even criticism, so if you see any room for improvement in any of our methodologies, please don’t hesitate to post in the respective thread, as our staff would love to discuss anything that could further improve our content.

Installing Windows Vista

When Microsoft first introduced Vista, it became the most advanced version of Windows yet. At the same time, it also become one of the beefiest, in terms of what kind of system resources it eats up. Even with a basic installation, Vista constantly runs many services in the background that we consider rogue. Why? Because if we’re in the middle of benchmarking, and a service begins doing whatever it does best, it has the potential to affect our results to some degree.


Photograph is Japantown, San Francisco. Taken during IDF 2007. (Original Photo)

Services, coupled with other OS features, have the potential to cause variations in our results, so on the remainder of this page, we’re going to explain step-by-step exactly how we set up Vista on all of our test machines, regardless of what we’re testing.

For all of our performance-related content, we use the 64-bit version Windows Vista Ultimate. We have no explicit reason for choosing to use Ultimate over Home Premium, except that back at the OS’ launch, we anticipated that many enthusiasts would choose that as their base OS… long before we all knew of the complete lack of Ultimate-specific bonuses that were to come.

Once the OS is installed and hits the desktop for the first time, we let it detect and install whatever it needs to, and then reboot. Once back at the desktop, we perform these tasks:

After these tasks are taken care of, the machine is reboot once again to make sure no faults exist, and then Vista’s Service Pack 2 is installed. After yet another reboot, we then run Windows Update to make sure that our machine is as up-to-date as possible where feature updates and security risks are concerned. If our Windows Update list includes hardware drivers, they are changed to remain hidden, as we already make sure we have the latest factory-supplied drivers for all of our equipment.

Once this is all done, we have ourselves the most up-to-date Windows Vista possible, so we begin to install whatever applications are explicitly needed for the particular type of testing we’ll be doing. For example, in the OS installation on our graphics card machine, we’ll go ahead and install each game title we’ll be using, and also related benchmarks (like Futuremark’s 3DMark Vantage, and OCCT for stress-testing).

Applications shared between all of our performance machines is as follows:

All other applications installed are specific to the component being tested, so you can look to the respective page in this article for that information.

Once all of our required applications are installed, we move right into optimizing the OS further to help us succeed in generating the best results possible. When Vista first launched, we spent a lot of time digging around in the OS’ services panel to see what was needed, and what wasn’t. While we had no intention of going to extreme lengths, we did want to disable the beefiest services that Vista includes, while leaving whatever the OS actually needs for reliable functionality. Disabled services are as follows:

It’s at this point that the OS installation is considered complete. To ease the pain of repetitive benchmarking, there are a couple of things I personally take care of on my own test machines, which you might also find useful. That is, to create shortcuts to commonly-used Windows tools, such as the OS services screen and also the performance monitor.

I personally keep such shortcuts in the Quick Launch section of the taskbar, but you can also keep them on the desktop if you so desire. For Performance Monitor, create a shortcut that links to “perfmon.exe”, and for the Services section, link to “services.msc”. To take things one step further, I also create a shortcut for the machine’s primary network device, for quick control of enabling and disabling access to the Internet. To do this, right-click the networking icon in the system tray and click on “Network and Sharing Center”, and then access the “Manage network connections” in the left pane. From there, simply drag and drop the primary network device to the desktop, and from then on, you’ll be able to right-click that to enable or disable your network device.

Wrapping Up

From this point, a decision is made. If the OS install we just performed will be used for performance-related testing, then we get right to backing up the entire hard drive using Acronis True Image Home 2009. If the PC is not used for performance testing (cooling, chassis, et cetera), then we go ahead and install up-to-date system drivers, as these components will never change.

When preparing a machine for performance-related testing, we’ll restore our OS using the Acronis backup primarily if we are switching from one major vendor’s product to the other, namely, with graphics cards. We’ll stick to the same OS install if we are planning to benchmark more than one NVIDIA GPU in succession, for example, but if we need to begin benchmarking ATI, we’ll restore the OS and start fresh.

For our CPU testing, a Vista install is created for each processor micro-architecture, and system drivers are installed prior to creating an Acronis backup. The reason for this is that for a given architecture, the system components used other than the CPU will not change, and even if something happens to die, it will be replaced. Motherboard reviews would be different, however, since each model can vary wildly with their integrated components, which is why we never install any system drivers prior to a backup.

Lastly, before any testing occurs in any content where we deliver results, we make sure first that the OS is stable to avoid potential skews with our results. Almost always, Windows Vista takes around 7 minutes to calm itself down entirely, and to monitor this, we use Performance Monitor, built into the OS. When we reach a point similar to what’s seen in the below image, we begin our testing.

This about sums up exactly how we handle our Windows Vista installation, and we hope you agree with our methods. If you don’t, our forums exist for that very reason!

As mentioned earlier, this entire article is a work-in-progress, with our graphics card methodology being the only one complete. You can turn to the next page to see how we handle things there.

Graphics: How-To Benchmark, Games List Pt. 1

When choosing games for use in our benchmarking, we look to a few things. The most important is whether or not the game is popular, because if it’s not, its results would prove useless for most people. To help aide our search, we use sites like Game Rankings in order to see which titles are popular, and how they’ve scored in reviews around the web. Ultimately, we look to gauge a game’s worth. This especially helps when we are planning to introduce just-released games to our suite.

Past that, we also look to see whether or not the game is worth benchmarking. If it’s too mild on our hardware, then the results aren’t going to be too important. So, no World of Warcraft results from us. Lastly, we won’t choose a game that caps it’s frame-rate, unless that feature can be easily disabled. Believe it or not, some games do indeed bar this ability (Burnout Paradise, for example).

Once we’ve chosen a game to use, we first find any cheat codes that might be available, in order to skip through the game’s levels. This might seem a little odd, but where time’s concerned, it’s important that we find the perfect level suited for benchmarking right away. If games don’t offer level skips, and the time to play through is just not possible, we’ll play the game until we reach an area suitable for benchmarking. If the game does support level skips, we’ll go through each and every level to see which is the most brutal on our hardware.

Because we value results generated by real-world testing, we don’t utilize timedemos in our testing whatsoever. The exception might be Futuremark’s 3DMark Vantage. Though it’s not a game, it essentially acts as a robust timedemo. We choose to use it as it’s a standard where GPU reviews are concerned, and we don’t want to rid our readers of results they expect to see.

Below is a list of the applications we install on our graphics card benchmarking machine. Since every-single game we currently test with is available through Valve’s Steam platform, we run each through there. If you’d like to add as us a friend on Steam, feel free to visit our profile.

FRAPS is what we use to capture the FPS information from our benchmarking runs, as it’s accurate, and well, it’s pretty much the only solution available. OCCT is used both for temperature-testing and power consumption tests, as it’s brutal towards any graphics hardware and can usually push the GPU to deliver higher temps and power consumption more than any game can. GPU-Z is used to verify our card’s clocks and also record temperature information.

Here are our most-recent GPU benchmarking machine specs:

Component
Model
Processor
Intel Core i7-975 Extreme Edition – Quad-Core, 3.33GHz, 1.33v
Motherboard
Gigabyte GA-EX58-EXTREME – X58-based, F7 BIOS (05/11/09)
Memory
Corsair DOMINATOR – DDR3-1333 7-7-7-24-1T, 1.60v
Graphics (See most recent GPU review for current list.)
Audio
On-Board Audio
Storage
Seagate Barracuda 500GB 7200.11
Power Supply
Corsair HX1000W
Chassis
SilverStone TJ10 Full-Tower
Display
Gateway XHD3000 30″
Cooling
Thermalright TRUE Black 120
Et cetera
Windows Vista Ultimate 64-bit

At this time, we currently benchmark all of our games using three popular resolutions: 1680×1050, 1920×1080 and also 2560×1600. 1680×1050 was chosen as it’s one of the most popular resolutions for gamers sporting ~20″ displays. 1920×1080 might stand out, since we’ve always used 1920×1200 in the past, but we didn’t make this change without some serious thought. After taking a look at the current landscape for desktop monitors around ~24″, we noticed that 1920×1200 is definitely on the way out, as more and more models are coming out as native 1080p. It’s for this reason that we chose it. Finally, for high-end gamers, we also benchmark using 2560×1600, a resolution that’s just about 2x 1080p.

On the remainder of this page, and the next page, we’ll go into great detail about how we benchmark each game, and also provide information on how to access the same level / saved game we do. In the near-future, we’ll be providing videos of our runs, so you can see visually, in addition to our explanation, just how we pull off each run.

Call of Duty: World at War

The Call of Duty series is one that needs no introduction. Although only six years old, CoD has already become a stature where both single-player and multi-player first-person shooters are concerned. From the series’ inception, each game has delivered stellar gameplay that totally engrosses you, thanks in part to creative levels, smart AI and realistic graphics.

World at War is officially the 5th game in the series, and while some hardcore fans claim that Treyarch is simply unable to deliver as high caliber a game as Infinity Ward, the title does do well to hold everyone over until Modern Warfare 2 hits (November 10, 2009). One perk is that WaW focuses on battles not exhausted in other war games, which helps to keep things fresh.

Manual Run-through: The level chosen for our testing is “Relentless”, one that depicts the Battle of Peleliu, which has American soldiers advance to capture an airstrip from the Japanese. The level is both exciting to play and incredibly hard on your graphics hardware, making it a perfect choice for our testing.

Relentless begins with you on a moving tank, which inches towards Japanese soldiers. Within fifteen seconds, a missile is sent straight your way, leaving you feeling extremely lightheaded, and also with blurred vision for a few moments. At this point, we move ahead and up the hill to the left of the river, taking out all soldiers that enter our path. Once the area is clear, the gun is reloaded.

From here, we continue along and take out a few more soldiers, then aim across the battlefield to blow up some gas containers, effectively ridding a half-dozen men at once. We then cross the river and go up and around gunning down everything in our path. Due to all the gunfire, our ammo is low at this point, so we reload once again. Afterwards, we continue down towards the open-air trench and walk towards the ground-mounted machine gun. After taking care of yet a few more soldiers, we make our way over to the fence and stop recording our FPS.

Loading the Test Level: Before you can load our level for testing, you’ll need to enable the console via the game options section. Once that’s done, you’ll want to enable a bind to allow quick access to God mode once inside of a level. To do this, hit ~ and type in: bind o “god” with o being the letter you’ll push once in-game (choose whichever letter suits you best). To load “Relentless”, enter the console and type in: devmap pel1b.

Call of Juarez: Bound in Blood

When the original Call of Juarez was released, it brought forth something unique… a western-styled first-person shooter. That’s simply not something we see too often, so for fans of the genre, its release was a real treat. Although it didn’t really offer the best gameplay we’ve seen from a recent FPS title, its storyline and unique style made it well-worth testing.

After we retired the original title from our suite, we anxiously awaited for the sequel, Bound in Blood, in hopes that the series could be re-introduced into our testing once again. Thankfully, it could, thanks in part to its fantastic graphics, which are based around the Chrome Engine 4, and improved gameplay of the original. It was also well-received by game reviewers, which is always a good sign.

Manual Run-through: The level chosen is part of the game’s first campaign. About twenty minutes into this mission, you’re required to make use of your slow-motion Concentration mode in order to down five soldiers standing atop a fence. Our saved game begins us out just moments past this section, leaving us standing far up a hill from a large river.

Once our save is loaded, we immediately switch to our single-pistol and move forward while looking down the hill towards the river and a few men firing bullets our way. As we continue along, we fire off a few shots of our own, making our way towards a bunker where we replenish our ammo. We also pick up some dynamite, which helps us blow up a pile of lumber blocking our way.

Continuing through this newly-opened path, we finish off two soldiers blocking our way and stop at a box where we gain the ability to sneak around and fire off a couple of shots at angry soldiers. We then move towards the next box, where we are given a brief tutorial on how to peek up and over, then kill three more men. We then switch our gun back to the shotgun, and head around the corner and shoot a barrel filled with oil. Finally, we make our way towards where the explosion occurred, killing a surviving soldier if necessary and quickly looting ammo from the stash. This is where we stop recording our FPS.

Loading Test Level: Download our archived saved game file and extract it to your desktop. Once the game has been launched at least once, copy the two folders (profiles, save) to the game’s main profile folder, which can be found under your Documents folder. The proper path to copy these two folders to will be DocumentsCall of Juarez – Bound in BloodOut.

Once the folders are copied, a new profile will be found when you launch the game, called “Techgage”. The password is auto-completed, but if you accidentally delete it, it’s “techgage”, without quotes. There is only one saved game included with our archive, which you can find through the single player menu.

Crysis Warhead

Like Call of Duty, Crysis is another series that doesn’t need much of an introduction. Thanks to the fact that almost any comments section for a PC performance-related article asks, “Can it run Crysis?”, even those who don’t play computer games no doubt know what Crysis is. When Crytek first released Far Cry, it delivered an incredible game engine with huge capabilities, and Crysis simply took things to the next level.

Although the sequel, Warhead, has been available for just about a year, it still manages to push the highest-end systems to their breaking-point. It wasn’t until this past January that we finally found a graphics solution to handle the game at 2560×1600 at its Enthusiast level, but even that was without AA! Something tells me Crysis will be de facto for GPU benchmarking for the next while.

Manual Run-through: Whenever we have a new game in-hand for benchmarking, we make every attempt to explore each level of the game to find out which is the most brutal towards our hardware. Ironically, after spending hours exploring this game’s levels, we found the first level in the game, “Ambush”, to be the hardest on the GPU, so we stuck with it for our testing.

This run-through is one of the simplest on our list, as we don’t even need to fire off our gun once. The level begins us out after our team’s ship experienced a rough landing, so we follow the linear path up to the point where there is a large vehicle explosion across the bridge. There isn’t too much special in our run here, but we do make sure to look back towards a missile drop when a plane crosses our path, as we near the bridge.

Loading Test Level: The console in Crysis is automatically enabled by default, so loading the level we use for testing couldn’t be much easier. Simply open the console with ~, and type in “con_restricted 0”, without quotes, followed by “map ambush”, also without quotes. Because this game’s console supports auto-completion, after you type in “con_”, you can hit your Tab key to fill in the rest of the command. You will have to add the 0 at the end manually.

Graphics: Games List Pt. 2

F.E.A.R. 2: Project Origin

Five out of the seven current games we use for testing are either sequels, or titles in an established series. F.E.A.R. 2 is one of the former, following up on the very popular First Encounter Assault Recon, released in fall of 2005. This horror-based first-person shooter brought to the table fantastic graphics, ultra-smooth gameplay, the ability to blow massive chunks out of anything, and also a very fun multi-player mode.

Three-and-a-half years later, we saw the introduction of the game’s sequel, Project Origin. As we had hoped, this title improved on the original where gameplay and graphics were concerned, and it was a no-brainer to want to begin including it in our testing. The game is gorgeous, and there’s much destruction to be had (who doesn’t love blowing expensive vases to pieces?). The game is also rather heavily scripted, which aides in producing repeatable results in our benchmarking.

Manual Run-through: The level used for our testing here is the first in the game, about ten minutes in. The scene begins with a travel up an elevator, with a robust city landscape behind us. Our run-through begins with a quick look at this cityscape, and then we wait for the door to open, killing off two baddies as soon as we enter the room.

Heading towards the only door in the room, another enemy enters, so we finish him off and make our way outside. From here, we immediately have two more enemies to clear out, and with a few steps further, we have yet another two. The guy standing on the highest balcony conveniently has an oil-lamp sitting beside him, so we shoot at it to see some pretty flames.

After all of the enemies here are cleared, we wait a few moments for our help to open the door, and then proceed through the linear level until we reach the point you see in the screenshot below. Here, we kill off every enemy in the room and make our way over towards the door on the other side. It’s when we reach that point that we stop recording our FPS.

Loading Test Level: The way F.E.A.R. 2 handles saved games requires a fair bit of patience, so take this as a warning. Because the game keeps just one save at a time, and auto-updates fairly frequently in the game, we have to overwrite our old save in between each run. To make this process easier on yourself, you should first backup your entire F.E.A.R. 2 profile folders (DocumentsWBGames) and store it elsewhere. Once done with our save, you should move it back.

Once you’ve backed up your personal F.E.A.R. 2 folder, download our saved game file and extract it to your desktop. You’ll see a file and the folder, the latter of which can be copied straight into your Documents folder. The other file is a batch file, which aides in the restoration of our saved game file each time a benchmarking run is complete. You’ll need to edit this file with Notepad and change the username to your own, or alter the paths entirely if you are using Windows XP. If the paths in this file match up, each time you double-click it, it will restore the saved game we use for testing.

Grand Theft Auto IV

If you look up the definition for “controversy”, Grand Theft Auto should be listed. If it’s not, then that should be a crime, because throughout GTA’s many titles, there’s been more of that than you can shake your fist at. At the series’ beginning, the games were rather simple, and didn’t stir up too much passion in certain opposers. But once GTA III and its successors came along, its developers enjoyed all the controversy that came their way, and why not? It helped spur incredible sales numbers.

Grand Theft Auto IV is yet another continuation in the series, though it follows no storyline from the previous titles. Liberty City, loosely based off of New York City, is absolutely huge, with much to explore. This is so much so the case, that you could literally spend hours just wandering around, ignoring the game’s missions, if you wanted to. It also happens to be incredibly stressful on today’s computer hardware, similar to Crysis.

Manual Run-through: After the first minor mission is completed, you park your car outside an apartment building, then go in to sleep and save your game. This is where our benchmarking run begins. Once our saved game is loaded, we run through the door and back down the stairs, and through the next door to go outside. Wasting no time, we hop into our car, and hit the gas.

The majority of our run is performed from within this car. From the starting point, we drive to the end of the road and then hang a left, following this road until we meet a split to enter some raised roads. We take the right path, and then the left path when we are greeted by yet another split. We follow this until we have the option to continue on the road or go down the ramp, and do the latter.

At the bottom, we take a right turn and follow the road straight until the end. The road we turn up is, for the most part, straight ahead, but slightly on an angle, and is essentially an upwards hill. At the top of this hill, we continue along, avoiding all the cars in our way, ultimately reaching a slightly inclined road, and then jumping a set of downward roads to hit the bottom. We stop recording our FPS at this point.

Loading Test Level: F.E.A.R. 2 has a poor way of handling your saved games, but GTA IV isn’t any better. To fully create the required folders, you’ll have to enter the game and log into your Games for Windows Live account, and probably update the program. Once you have your account associated with the game, a folder will be created somewhere hidden on your computer.

In Windows Vista, the folder hierarchy will be similar to: C:UsersTechgageAppDataLocalRockstar GamesGTA IVsavegamesuser_eXXXXXXXXXX/. It’s inside that last folder that the saved game file will need to go. Download our saved game file. To avoid overwriting your own saved game files, simply change the number at the end of the filename, to have it sit at the very end of your list. For example, if you have saved games ranging from SGTA400 – SGTA409, name ours SGTA410. The saved game limit is 15, so if you have all 15 (SGTA414) already, you will need to backup one of your own saved games and name ours to replace it.

Race Driver: GRID

If you primarily play games on a console, your choices for quality racing games are plenty. On the PC, that’s not so much the case. While there are a good number, there aren’t enough for a given type of racing game, from sim, to arcade. So when Race Driver: GRID first saw its release, many gamers were excited, and for good reason. It’s not a sim in the truest sense of the word, but it’s certainly not arcade, either. It’s somewhere in between.

The game happens to be great fun, though, and similar to console games like Project Gotham Racing, you need a lot of skill to succeed at the game’s default difficulty level. And like most great racing games, GRID happens to look absolutely stellar, and each of the game’s locations look very similar to their real-world counterparts. All in all, no racing fan should ignore this one.

Manual Run-through: For our testing here, we choose the city where both Snoop Dogg and Sublime hit their fame, the LBC, also known as Long Beach City. We choose this level because it’s not overly difficult, and also because it’s simply nice to look at.

There’s not too much to say about how we proceed through the level, but it can be a little tedious at times. Off the line, our goal is to get to the front of the pack as quickly as possible, and to pull this off, the first turn in the race must be executed perfectly. It might seem counter-intuitive that we begin off in the middle of the pack (as shown below), but we do that because don’t want to avoid the car models and smoke entirely. As long as we can get to the front of the pack early in the race, we’re golden until the finish line.

The reason we choose to rid the cars as soon as possible is because if the models are in our frame for any good deal of the race, the end FPS result can vary far more than to our liking. We’re talking 5 – 10 FPS, or even higher depending on the resolution. If we do manage to rid the cars after the first or second turn, then our end result usually settles within 1 FPS… which is obviously much better.

Loading Test Level: As mentioned above, our level of choice is Long Beach. To access, enter the Race Day menu, and make sure the options chosen are as follows: Pro Tuned, Nissan Skyline GT-R Z-Tune, Long Beach, Classic Circuit, 2 and Kirby Motorsport. After accepting those choices, change the Grid Position to Middle.

World in Conflict: Soviet Assault

I admit that I’m not a huge fan of RTS titles, but World in Conflict intrigued me from the get go. After all, so many war-based games continue to follow the same story-lines we already know, and WiC was different. It counteracts the fall of the political and economic situation in the Soviet Union in the late 80’s, and instead provides a storyline that follows it as if the USSR had succeeded by proceeding with war in order to remain in power.

Many RTS games, with their advanced AI, tend to favor the CPU in order to deliver smooth gameplay, but WiC favors both the CPU and GPU, and the graphics prove it. Throughout the game’s missions, you’ll see gorgeous vistas and explore areas from deserts and snow-packed lands, to fields and cities. Overall, it’s a real visual treat for the eyes – especially since you’re able to zoom to the ground and see the action up-close.

Manual Run-through: The level we use for testing is the 7th campaign of the game, called Insurgents. Our saved game plants us towards the beginning of the mission with two squads of five, and two snipers. From the beginning, we take our entire squad and move them towards the center of the map to infiltrate the enemy.

There isn’t too much variation in our run here, as you’d expect for an RTS. What we do is basically hover the camera left to right while eyeing the action, taking any steps we need to keep our squad alive. Usually we’ll summon another squad once we’ve accrued enough points, and have them added to the battle. At one point, we zoom in on the action for about five seconds, and the run ends after the action dims down. The entire run takes between three and four minutes to run.

Loading Test Level: Download our saved game file and extract it to your desktop. Take the .WICSAVEGAME file inside the folder and copy it to the WIC saved games folder. On our machine, the path was DocumentsWorld in ConflictSavegames. To access, open the game and go to the Campaign menu. In the “Current Campaign” drop-down menu, “Deathspawner” will become available. Access it, then go to the saved games section up top, and load the “Techgage Benchmark” listed below.

Copyright © 2005-2021 Techgage Networks Inc. - All Rights Reserved.