The Quest for Useful Real-World SSD Benchmarks

As I have mentioned in our news section in the past, we’ve been working on getting some storage-related content up on the site for a while, but as things go, issues keep arising, and it never seems to happen. As it stands, however, we’re closer than ever to getting such content up, and I’m confident that you’ll be reading some articles in the next few weeks to come, tackling both HDDs and SSDs.

If there’s one thing we’ve learned over the past few months, though, it’s that storage benchmarking is hard. Alright, let me elaborate. It’s not that benchmarking is hard in general, but it’s hard as heck to find appropriate benchmarks that people care about. Throughout all of our content, we strive to deliver as many real-world results as possible. I think it’s fair that if we are to deliver information, it should be as realistic, and it should be information people can relate to.

But with storage, things are tough. Synthetic benchmarks will show benefits of faster drives fine, but when it comes to real-world tests, the task of showing the benefits of either HDDs or SSDs becomes very complicated. Believe it or not, the vast majority of scenarios we’ve tested simply don’t show real differences in performance from drive to drive, as long as we’re talking about drives with like speeds (7200RPM drives would of course be faster than 5400RPM, but even then, it’s primarily the synthetics that would prove it).

For the past month or two, Robert and I (yes, there’s another Robert in case you haven’t noticed) have been working together on conjuring up the best possible SSD test suite. The problem, of course, is despite having a nice collection of synthetic benchmarks, we have almost no real-world tests, with the main exception being SYSmark 2007 Preview (ironically, still not a true real-world benchmark). In our CPU reviews, I focus a lot on applications such as image manipulators, video encoders, 3D rendering tools and more, but in all of our testing, performance differences between an HDD and SSD in either of these is non-existent.

As a specific example, our Adobe Lightroom test sees 100 RAW files converted to 100 JPG. You’d imagine that a test like this could push a storage device nicely, but that’s not the case. In the end, the time-to-completion was identical on both an HDD and SSD. The same applies to video encoding. The problem is that neither of these scenarios push the kind of bandwidth we need to see faster drives, like SSDs, excel. But on the other hand, our scenarios are realistic.

What’s frustrating about all this is that SSDs are fast, and offer real benefits to consumers. But for the most part, what we’re seeing is that the benefits have exclusively to do with application loading, Windows loading, game level loading, and other like processes. Random writes are of course much faster on an SSD as well, but outside of a synthetic benchmark, it’s very difficult to find a real test that can help prove that.

So my question to you all is this… what would YOU like to see us tackle in our upcoming SSD content? Is there a real-world test that you’re confident would highlight the benefits of an SSD, or benefits from one SSD to another? Please let us have it, because we could sure use some ideas. If not, we’ll have to resort to including some more mundane tests, such as SSD noise levels! *

* Yes – that was sarcasm.