A new benchmark is born

Controversial new 3DMark 2003 ushers in advanced video testing

Jason Cross


This article originally appeared in Computer Games Magazine #150

If you’re a gamer, chances are you’re already familiar with 3DMark. I’ve used the 2001 version of this popular video benchmarking tool for over a year now. There’s been some controversy over the Futuremark (previously MadOnion.com) product, so maybe it’s time to review benchmarking in general.
At its heart, 3DMark is a synthetic benchmark. This means that it’s not an application that you would do anything else with but test the performance of something, in this case the kind of graphics load that high-end games will demand over the next few years. It is not an actual game and does not use a current game engine, so it won’t necessarily tell you how any individual game would perform. It’s important to remember that all games are different, and that no benchmark can tell you how well your hardware will run every title on the market.
The 2003 version runs four game tests meant to simulate different types of games and then spits out an overall score in “3DMarks,” which is derived from weighted sums of the frame rates. (Why they don’t actually use a weighted average, and thus produce a score that resembles something a gamer will understand, I’ll never know.) The first test is meant to be pretty easy on modern hardware—it’s a high-altitude WWII dogfight with some cool smoke and fire effects—and contributes only a small percentage to the overall 3DMark score. The next two, depicting a space-corridor shooter and what could be a scene from a 3D role-playing game, essentially run the same graphics algorithms. We’re talking some real eye candy here, since they’re made for DirectX 8 pixel and vertex shaders and feature artwork so demanding that only the very best cards can get a decent frame rate. The fourth scene is the coup de grace, a virtual remix of the previous Nature scene, this time utilizing DirectX 9’s new pixel and vertex shaders to produce truly amazing water and independently moving leaves and blades of grass. It won’t run at all without hardware support for DX9 shaders, and even the $400 cards of today don’t run it particularly well.
There are plenty of other tests, but none contribute to the overall 3DMark score. Instead, they’re used to test the individual aspects of video cards like fill rate, vertex processing speed, and so on. The sound performance tests are very interesting indeed. It runs a WWII flight combat scene first with no sound, then with 24 active ones, then with 60, recording the frame rate for each. It’s a good indication of the load that 3D sound acceleration places on overall game performance.
About a day after this new benchmark hit the web, the flames started flying. NVIDIA doesn’t like it. It contends that the first benchmark is not very forward looking, and that the second and third tests use an algorithm that does not efficiently cache the characters after they’ve been “skinned,” greatly increasing the amount of vertex processing that must be done. Only two of the nine shaders in the final game test are DX9, the other seven are DX8 shaders. Ultimately, NVIDIA feels that paying attention to 3DMark 2003 and optimizing drivers for it does not benefit real game performance.
Futuremark and ATI disagree. They feel that the benchmark is a valid synthetic test that fairly compares hardware, and is targeted at the latest and upcoming DirectX 9 cards. There are specific rebuttals to each of NVIDIA’s complaints, but this isn’t a point-counterpoint. I feel NVIDIA’s complaints might be true, but the benchmark is still valid. A benchmark, synthetic or not, only tells you how well your rig will run that benchmark. Nobody should ever get all excited over a single benchmark score, nor base their buying decisions on it. The charts in our video card reviews are provided to supplement the text, not the other way around.
At least for the time being, I will use 3DMark 2003. It’s a useful additional synthetic benchmark, and like all such tests should be considered secondary to testing performance with actual games. Buying decisions should be based on the text of the review, which is supported in part by all the benchmarks as a whole (as well as by value and stability). Only time will tell, but we believe that ultimately the cards that do best in the latest 3DMark over the next year will also do best in the latest games.

This article originally appeared in Computer Games Magazine #150