Chaos V-Ray Bench Update Yields Very Different Results For AMD Radeon GPUs

When AMD launched its Radeon RX Vega series, I decided to go about the launch a different way: by focusing on compute more than gaming. Ultimately, that turned out to be a great idea, as per my memory, that review turned out to be the most popular review we’ve ever had of an AMD card. That said, the review has now been a little marred due to the fact that the results from one benchmark were way off the mark.

Being that Chaos’ V-Ray renderer is so popular across the creative industry, I decided to add its benchmark to a preexisting test suite, when normally I’d wait until the next overhaul. Funny enough, it was an NVIDIA rep who tipped me off to the benchmark, after I inquired about whether or not they knew about any new benchmarks that I may have missed. Following that, I sure didn’t expect to see AMD’s cards give NVIDIA’s such a hard time.

As it happens, there’s a reason for what we saw: a bug existed in the 1.0.5 version of V-Ray Bench which tested AMD’s cards – and only AMD’s cards – incorrectly, involving the CPU when it shouldn’t have. I wouldn’t have even caught this if not for one of our eagle-eyed readers. Sadly, even when told about the issue, I underestimated how it’d affect the results.

Let’s do a little recap of the results seen in the RX Vega 64 review:

Here, RX Vega 64 ranks at the top, alongside NVIDIA’s biggest models, and meanwhile, the Radeon Pro WX series performs admirably against NVIDIA’s competition, as well. All of that changes when benching with the 1.0.6 version:

NVIDIA’s results hold still (minus a single second difference) between versions, but AMD’s results plummet. The Vega 64 went from 70 seconds with 1.0.5 to 150 seconds with the bug-fixed 1.0.6. That’s hardly a small difference, and in fact, it makes me sick to imagine anyone purchasing the card for use in V-Ray based on my results. In this renewed match-up, the RX Vega 64 beats NVIDIA’s Quadro P2000, a less capable card enjoying the fruits of NVIDIA’s driver R&D.

To call the bug in V-Ray Bench 1.0.5 minor would be an understatement, because it dramatically changed what we glean from the results. Overall, this single benchmark doesn’t change much about RX Vega’s compute capabilities. But if you’re a V-Ray user, NVIDIA clearly dominates your particular market at this point in time.

The only way we could have caught this issue ourselves is if we used monitoring tools to look at how the system was used, rather than put complete faith in the benchmark. I can’t remember a single occasion across the 12 years of Techgage‘s existence of having this sort of thing happen. Having to redo all of the testing wasn’t enjoyable, and having to write this post wasn’t, either. I regret that this has happened, and will work harder to make sure it doesn’t happen again.