AMD Rejects BAPCo’s SYSmark 2012 – Should We?

The truth of the matter is, we haven’t used SYSmark 2007 Preview to a great deal in any of our testing for the simple fact that we’ve found its methodologies and metrics to be borderline useless. As far back as 2008, we’ve had discussions with both AMD and Intel regarding the issues we foresaw, and while AMD didn’t have much to comment on, Intel up to the current date has welcomed it to be used.

Similar to Futuremark’s PCMark suite of tests, SYSmark outputs a singular score where the higher the number, the better. The problem, though, is that in most cases, this number didn’t tell us a thing about the product being looked at. In fact, there were occasions where we’d compare a fast dual-core to a modest quad-core and both would appear equal – or the dual-core would win, due to its faster single-threaded performance. Of course, things aren’t quite so simple, and unless someone has super-specific needs, a quad-core is going to be much more preferable.

When Intel launched its first Nehalem-based processor, the Core i7-965 Extreme Edition, we expressed concerns to the technical folk there about how it compared in SYSmark to its previous high-end champ, the Core 2 QX9770. From a technical standpoint, the i7-965 was the far superior chip. It experienced a much improved architecture, the re-introduction of HyperThreading, a triple-channel memory controller, a faster bus, an L3 cache, and for an added boost, Turbo.

Clock for clock, Nehalem was expected to perform about 10~25% faster than its predecessor in single-threaded tasks, and up to 20~100% faster if multi-threading was involved. But according to SM2007, Intel’s latest and greatest (at the time) was in fact slower. Note these results where the Turbo mode was disabled on the i7-965 to make the clock-for-clock comparison more relevant:

Overall, the faster i7-965 proved one point less useful in SYSmark’s tests, and if not for the major boost in 3D rendering performance Intel packed into the Nehalem microarchitecture, the overall score would have been even less. How did this compare to real-world performance at the time? Please note that Turbo was enabled in this case, adding at most 8% to the overall clock in stressful workloads.

While the SYSmark results above shows these two processors as being equals, our real-world testing stated something quite different. The Core i7-965 didn’t just inch past the QX9770, in most cases it obliterated it. Please note that none of these benchmarks were “optimized” for Nehalem in particular. They were the same benchmarks used when the QX9770 first launched.

Is a benchmark, real-world or not, useful if it doesn’t manage to give a realistic perspective of a product’s performance, or a proper comparison of multiple products? We’re inclined to say no. While advanced, SYSmark simplifies things to such a large degree that its results just aren’t that useful or too telling.

Final Thoughts

After reading through Nigel’s blog, I’m having a hard time finding something specific to disagree on. As I’ve not been gung-ho about SYSmark and its methodologies for a while, not to mention the fact that we strive for real-world benchmarks here at Techgage, it’s nice to know someone else in the industry agrees.

It must be pointed out, however, that it’s a bit strange that AMD didn’t voice these opinions with the 2007 Preview launch nearly four years ago, since the exact same arguments could have been used there. From our recollection, AMD has on multiple occasions left and rejoined its post at BAPCo, for what reasons, I’m unsure.

While Intel doesn’t comment on issues regarding its competitors, it told us that it values companies such as BAPCo and continues to participate with many of them, as it’s important to have an industry-wide collaboration to ensure that meaningful performance benchmarks are being developed. With SM2012, Intel and AMD clearly don’t see eye-to-eye. At the same time, BAPCo is comprised of many other leading companies, such as Dell and HP… are we going to see others come out to agree with AMD, or is AMD going to be the awkward man out?

As always, we welcome your input regarding both AMD’s thoughts on the matters discussed here and also your own thoughts on SYSmark in general. Whereas PCMark 7 gives us similar simple overall scores for tests, its suites are a little more realistic for the average user. But does that make it a better benchmark? As evidenced in our content, our opinion is that nothing can beat real-world benchmarks, but we’re not opposed to using additional synthetics for the sake of completeness and interest from our readers – if we find value is indeed added.

June 21 (5:44PM EST) Addendum:

BAPCo has sent out an official statement:

Business Applications Performance Corporation (BAPCo) is a non-profit consortium made up of many of the leaders in the high tech field, including Dell, Hewlett-Packard, Hitachi, Intel, Lenovo, Microsoft, Samsung, Seagate, Sony, Toshiba and ARCintuition. For nearly 20 years BAPCo has provided real world application based benchmarks which are used by organizations worldwide. SYSmark 2012 is the latest release of the premiere application based performance benchmark. Applications used in SYSmark 2012 were selected based on market research and include Microsoft Office, Adobe Creative Suite, Adobe Acrobat, WinZip, Autodesk AutoCAD and 3ds Max, and others.

BAPCo is disappointed that a former member of the consortium has chosen once more to violate the confidentiality agreement they signed, in an attempt to dissuade customers from using SYSmark to assess the performance of their systems. BAPCo believes the performance measured in each of the six scenarios in SYSmark 2012, which is based on the research of its membership, fairly reflects the performance that users will see when fully utilizing the included applications.

Discuss this article in our forums!

Have a comment you wish to make on this article? Recommendations? Criticism? Feel free to head over to our related thread and put your words to our virtual paper! There is no requirement to register in order to respond to these threads, but it sure doesn’t hurt!