We’re Trying To Automate All The Things (Or At Least Our Benchmarking)

As of this moment, Techgage is thirteen-and-a-half years old, which means we’ve been posting performance reviews for about just as long. It seems neat to have been doing this for over a decade, but at the same time, I feel like an idiot for waiting until 2018 before I decided to explore the possibility (or reality) of introducing automation into our performance testing.

For some, it’s going to be hard to understand how much work is required to get a review out the door. That is, if you want your benchmark results to have any sort of confidence behind them. There have been times when I’d obsess so much about test results that I’d feel burned out before long, but hardware releases never stop coming, so there’s been just one option: power through.

Some media outlets have the luxury of being able to toss multiple people onto a single problem, but we at Techgage often don’t have that luxury, because not even two of us are in the same city. That means I personally handle the vast majority of benchmarking, and that means I spend way too much time doing just that. Let’s peek back to the time our Threadripper 2990WX review was posted, in August.

Results, and more results

After getting back from AMD’s Italy’s launch event, I immediately felt overwhelmed with the amount of testing that had to get done for the launch review. I wasn’t able to get testing done before the launch because other performance content dominated my time leading up to it. Thus, for the entire week leading up to the 2nd-gen Threadripper launch, I was benchmarking two PCs at the same time. When I was finally done, I kept twitching to look towards one of the test PCs, because I had been in the habit all week of moving back and forth.

It’s the kind of thing that makes me feel a little nuttier just thinking about it. After that week, I swore that I’d buckle down and figure out the automation thing, and fortunately, things are well underway at this point. Currently, our workstation GPU suite is effectively entirely automated, and best of all, the results generated from this automation aligns with the results from manual testing (if that wasn’t the case, my walls would probably fear me).

Little did I realize before diving into automation, it’s not that difficult if you already have basic scripting knowledge. Admittedly, I have what I’d consider intermediate scripting skills, as I’ve been dabbing with it since monochrome monitors were standard fare, but this level of automation seemed impossible to me. In reality, it’s more time-consuming than it is difficult, unless you’d tie “trial and error” into the difficult side.

Scripts ahoy!

I’m no programmer, but I’ve sure felt like a developer the past couple of weeks, taking advantage of batch scripting to make sure that not only the tests run without issue, but that their results get printed to a file in a sane format. I’d wager I’ve spent about one hundred hours on this project so far, and one friend has called me batchit crazy. That’s fair.

I should stress that this suite, which I call TGBench, will focus mostly on CPU and workstation GPU, as gaming is will require a vastly different set of considerations. Ultimately, it wouldn’t be safe to rid manual game testing entirely, since performance numbers only tell part of the story there.

What all of this ultimately means is that I’ll hopefully stop wasting so much of my life waiting for one test to finish so that I can run another. That in turn means that more content will get published, which is important, as we’re pretty much bleeding ideas we want to run with.

That all said, I am running behind on every review I’m tackling right now, but they will be handled pretty quickly once I have a substantial amount of results to publish. First up will likely be the RTX 2070 (which as I noted here, will be landing in the lab soon), followed by either Intel’s Core i9-9900K or AMD’s Radeon Pro WX 8200 – it entirely depends on which testing gets me to the goal first.

As always, stay tuned. We’ve been quiet, but there’s been a lot going on in the background, and soon, a lot will be happening in the foreground.