• Vista Wallpapers
• Windows Wallpapers

The test

The tests will be carried out on the AMD Spider platform that I have set up in the lab (Phenom 9700, Radeon 3850 graphics card, 2GB of RAM …). I’ve used this system as the platform for a number of benchmarks I’ve run over the past week (for a full spec, see this post).

As for the tests, we took two identical images of Vista - one RTM, one SP1. We then loaded Microsoft Office 2007 Professional onto the system and applied all the patches (including Office 2007 SP1). The system was then defragged and rebooted several times.

Afterwards we downloaded and installed DMS Clarity Studio software which includes the OfficeBench software. This application comes with a comprehensive set of Microsoft Office test scripts. These test scripts, while not perfect (no benchmark solution is) are pretty good and simulate a number of real-world tasks that users might carry out in Microsoft Office. The metric that we’re interested in getting from this benchmark is how long OfficeBench takes to execute the test scripts.

All tests were run five times under no load and five times under load (here OfficeBench runs a small Windows Media Video in the background), and we rebooted the system between each run. The tests were duplicated on Vista RTM and Vista SP1.

Because the results were so well grouped for this test, no results were discarded and times taken to execute the scripts were averaged. The lower the time, the faster the scripts completed and the better the result.

Thoughts on benchmarking

I’ve been benchmarking systems for long enough to know that no matter how many questions I think that my results answer, what I’m really doing is creating about three new questions for each question I solve. This is what happened with my earlier run of Vista benchmarks - I’d run some tests, you’d then come back and offer different scenarios that you’d like to see done and different platforms for those scenarios.

Benchmarking is an artificial activity. The goal is to eliminate as many variables as possible and achieve some consistent metric. Problem is, by removing the variables you’re actually shifting the process out from reality and into a make believe land which only exists on the PCs being benchmarked. Before you carry out your daily PC tasks I bet that you don’t take elaborate steps to ensure that you get consistency. Hence my enthusiastic use of the phrase “your mileage WILL vary.”

Another fundamental problem with benchmarking is that neither the tests nor the results are exactly what people want. Ultimately, what everyone wants to see is a benchmark of their daily tasks carried out on their PC. That, I’m afraid, is something I cannot provide.