Does MobileMark Need to Change Its Focus?

At Techgage, we take our benchmarking methodologies seriously, as evidenced by my post from earlier today. In order us to feel completely confident in our results, we need to make sure that we exhaust our methods from all angles, making sure that the room for error is as small as possible. Though earlier I was talking about CPU, GPUs and motherboard testing… there’s another product that we often test that can be just as tedious: notebooks.

Where those are concerned, we again try to handle our testing a little differently than others, but one area where that’s not the case is with the inclusion of MobileMark 2007 testing. The primary goal of this test is to exhaust the battery while running a realistic scenario, such as while getting work done or while reading an e-Book. The problem, is that it’s not entirely realistic of a typical consumer.

The ever-outspoken Pat Moorhead, VP of Advanced Marketing at AMD, has made a new blog post talking about this exact issue, and I have to say, he raises some interesting points. I won’t repeat all of what’s said there, but the idea is simple… how useful is MM07 when it uses applications and scenarios completely atypical to the regular consumer?

The new question I raise is this… what’s the alternative? It’s difficult to compare one notebook to another where battery-life is concerned with real-world testing, because there IS going to be some variance. MobileMark tests things the exact same way with each run, so the results can at least be compared. It’s a tough one. Either MobileMark is going to have to re-think how they do things, or sites like ours will be forced to write our own scripts (the former sounds great to me right about now!).

Inside MM07, The “measured application task times” for Adobe Flash Creation is 33.6%, Adobe Photoshop CS2 is 21.8%, Adobe Illustrator CS2 is 16.7%, Microsoft Excel is 16.6%, WinZip 10.0 is 7.1%, Microsoft Word is 1.1%, Microsoft PowerPoint is 1.0%, Microsoft Project is 1.2% and Microsoft Outlook is 1.0%. I ask, when you use your notebook, do you spend 72% of your time recoding Flash videos, manipulating and compressing pictures in Photoshop and Illustrator? I’m going to go out on a limb here and guess that that is not an accurate reflection of most of our usage profiles!