Techgage’s Testing Methodologies

Here at Techgage, we take what we do very seriously, and that’s most true when it comes to our performance-related articles, such as our reviews for processors, graphics cards, motherboards and so forth. If you’ve been frequenting our side of the web for a while, you’re probably already well aware of this. What this article hopes to accomplish, though, is to give you guys a deep look at how we conduct our testing, so you can see just what it takes for us to deliver accurate and concise content, and also for your personal use, if you want to compare your personal results to ours.

This is an article I’ve wanted to get written for quite a while, because I feel that if everyone out there knows exactly how we handle things in our labs, you’ll be able to have utmost confidence in our results. After all, that’s the most important thing, alongside our professional analysis.

That said, this article is not yet fully complete, as most of our methodologies are not quite updated to where we want them to be. So, this effectively acts as a base, and it will be updated whenever we revise any aspect of our testing methodology, or want to add a bit more information we find important. If we do a major overhaul on one of our performance methodologies, then we’ll simply overwrite the respective page. There will be no “Part Two” to this article. Rather, it will just be continually updated when needed. Whenever we make major updates, we’ll republish the article to appear on the front-page, and use a news post to explain all of what was done.

We’ve always been open to change and ideas from our readers, and even criticism, so if you see any room for improvement in any of our methodologies, please don’t hesitate to post in the respective thread, as our staff would love to discuss anything that could further improve our content.

Installing Windows Vista

When Microsoft first introduced Vista, it became the most advanced version of Windows yet. At the same time, it also become one of the beefiest, in terms of what kind of system resources it eats up. Even with a basic installation, Vista constantly runs many services in the background that we consider rogue. Why? Because if we’re in the middle of benchmarking, and a service begins doing whatever it does best, it has the potential to affect our results to some degree.

Photograph is Japantown, San Francisco. Taken during IDF 2007. (Original Photo)

Services, coupled with other OS features, have the potential to cause variations in our results, so on the remainder of this page, we’re going to explain step-by-step exactly how we set up Vista on all of our test machines, regardless of what we’re testing.

For all of our performance-related content, we use the 64-bit version Windows Vista Ultimate. We have no explicit reason for choosing to use Ultimate over Home Premium, except that back at the OS’ launch, we anticipated that many enthusiasts would choose that as their base OS… long before we all knew of the complete lack of Ultimate-specific bonuses that were to come.

Once the OS is installed and hits the desktop for the first time, we let it detect and install whatever it needs to, and then reboot. Once back at the desktop, we perform these tasks:

Post Vista Installation Steps
• Uncheck “Run at start up” in the Welcome Center.
• Disable “User Account Control” (UAC).
• Disable Windows Sidebar.
• Disable Security Center alerts.
• Disable Windows Firewall.
• Disable Windows Defender.
• Turn off automatic updating with Windows Update.
• Disable screensaver and change power scheme to “High Performance”.

After these tasks are taken care of, the machine is reboot once again to make sure no faults exist, and then Vista’s Service Pack 2 is installed. After yet another reboot, we then run Windows Update to make sure that our machine is as up-to-date as possible where feature updates and security risks are concerned. If our Windows Update list includes hardware drivers, they are changed to remain hidden, as we already make sure we have the latest factory-supplied drivers for all of our equipment.

Once this is all done, we have ourselves the most up-to-date Windows Vista possible, so we begin to install whatever applications are explicitly needed for the particular type of testing we’ll be doing. For example, in the OS installation on our graphics card machine, we’ll go ahead and install each game title we’ll be using, and also related benchmarks (like Futuremark’s 3DMark Vantage, and OCCT for stress-testing).

Applications shared between all of our performance machines is as follows:

• CPU-Z (System information. Used for verifying clock speeds, voltages, et cetera).
• HD Tune Pro 3.5 (Storage device information. Used to periodically check up on the health of our drives).
• Everest Ultimate Edition 5.0 (System information. Used for temperature and voltage monitoring).
• PerfectDisk 10 (Storage defrag utility).

All other applications installed are specific to the component being tested, so you can look to the respective page in this article for that information.

Once all of our required applications are installed, we move right into optimizing the OS further to help us succeed in generating the best results possible. When Vista first launched, we spent a lot of time digging around in the OS’ services panel to see what was needed, and what wasn’t. While we had no intention of going to extreme lengths, we did want to disable the beefiest services that Vista includes, while leaving whatever the OS actually needs for reliable functionality. Disabled services are as follows:

Services Disabled Prior to Benchmarking
• PerfectDisk 10
• Windows Defender
• Windows Error Reporting Service
• Windows Event Log
• Windows Firewall
• Windows Search
• Windows Update

It’s at this point that the OS installation is considered complete. To ease the pain of repetitive benchmarking, there are a couple of things I personally take care of on my own test machines, which you might also find useful. That is, to create shortcuts to commonly-used Windows tools, such as the OS services screen and also the performance monitor.

I personally keep such shortcuts in the Quick Launch section of the taskbar, but you can also keep them on the desktop if you so desire. For Performance Monitor, create a shortcut that links to “perfmon.exe”, and for the Services section, link to “services.msc”. To take things one step further, I also create a shortcut for the machine’s primary network device, for quick control of enabling and disabling access to the Internet. To do this, right-click the networking icon in the system tray and click on “Network and Sharing Center”, and then access the “Manage network connections” in the left pane. From there, simply drag and drop the primary network device to the desktop, and from then on, you’ll be able to right-click that to enable or disable your network device.

Wrapping Up

From this point, a decision is made. If the OS install we just performed will be used for performance-related testing, then we get right to backing up the entire hard drive using Acronis True Image Home 2009. If the PC is not used for performance testing (cooling, chassis, et cetera), then we go ahead and install up-to-date system drivers, as these components will never change.

When preparing a machine for performance-related testing, we’ll restore our OS using the Acronis backup primarily if we are switching from one major vendor’s product to the other, namely, with graphics cards. We’ll stick to the same OS install if we are planning to benchmark more than one NVIDIA GPU in succession, for example, but if we need to begin benchmarking ATI, we’ll restore the OS and start fresh.

For our CPU testing, a Vista install is created for each processor micro-architecture, and system drivers are installed prior to creating an Acronis backup. The reason for this is that for a given architecture, the system components used other than the CPU will not change, and even if something happens to die, it will be replaced. Motherboard reviews would be different, however, since each model can vary wildly with their integrated components, which is why we never install any system drivers prior to a backup.

Lastly, before any testing occurs in any content where we deliver results, we make sure first that the OS is stable to avoid potential skews with our results. Almost always, Windows Vista takes around 7 minutes to calm itself down entirely, and to monitor this, we use Performance Monitor, built into the OS. When we reach a point similar to what’s seen in the below image, we begin our testing.

This about sums up exactly how we handle our Windows Vista installation, and we hope you agree with our methods. If you don’t, our forums exist for that very reason!

As mentioned earlier, this entire article is a work-in-progress, with our graphics card methodology being the only one complete. You can turn to the next page to see how we handle things there.