nVidia and FutureMark's 3DMark03

[Illuminati]

In light of the recent uproar in relation to 3DMark03 and some hardware review sites using unapproved nVidia drivers, we posted an editorial which includes results of our research in the matter. We used sources from FutureMark, nVidia, and the hardware community. We feel our editorial does a good job of tying up the loose ends of the matter and sharing our conclusion. We hope this editorial will educate the rest of the community and share with everyone this issues recent happenings. Here is a blurb from the editorial:

"On January 19th, Martin Luther King Jr. Day, I received an email from FutureMark asking that I kindly remove my 3DMark03 scores because I used a driver that was not listed on their "Approved" Drivers List. More information to follow on the "Approved" Drivers List. Nate and I did some quick scrambling to find out why FutureMark would ask such a question and what grounds they had to do so."

http://www.legitreviews.com/Reviews/editorial12604_1.shtml

[pastorjay]

BRAVO! Justin...

Futuremark better wake up or this will just be the start of bad things for them. Great post to help clear things up!

PJ

[Apoptosis]

You put together a bunch of info and tied up the majority of the loose ends! Sad it came to this though.

[Illuminati]

Thanks for the comments guys.

As far as our readers go... I'm assuming I cleared things up so well that no one has any questions, comments, or criticisms!... now seriously... feel free to join the forum and share your thoughts!

Hope to hear from new voices soon!

[tillyoubreakit]

Well first I must say that I just came across this website and found your article to be very interesting. Good writing too.

After having many systems (Watercooling, Vapochill, and experimental). I was always sceptical of the scores that 3D Mark gave me. I think it is great that you guys dropped 3D Mark because there is no real way of telling the real performance anymore. I personally stopped using Benchmarks. I sold my 2.6 @ 3.85 P4 for a new Dual XEON Platform and could not be happier with it.

All this Benchmarking and numbers made many people forget what we really need. I see many kids playing with 99FPS and V-Sync off @ 60Hz. The screen shakes and the eyes bleed. Id rather turn on v-sync at a 1600x1200 on my (old) "22" iiyama and play with 85 FPS (85Hz Refresh Rate). YOu will never notice the difference between the 99 and the 85. I bought a 18.1" Iiyama TFT the other day. And playing @ 1280x1024 at 75FPS is just the same too.

I am getting a little of point. But what I meant to say is that we need to value things like quality much more than the stupid numbers 3D Mark shows. I had 24K on my other system. And you know what ?? I never noticed a dang difference between the 24K and the 18K I get now.

-Florian

[jb]

Well first I must say that I just came across this website and found your article to be very interesting. Good writing too.

Yeap Me 2!

I was always sceptical of the scores that 3D Mark gave me. I think it is great that you guys dropped 3D Mark because there is no real way of telling the real performance anymore.

Well well. When did 3dmark EVER tell you the real performance? It never was able to and never will. 3dmark was only useful for comparing two cards to see where their strength and weakness lie. And that's about all its really good for. People have the misconception that 3dmarks score = some real world measure of performance that will equate to some FPS number in game XYZ. I think in reviews synthetic should be used only in a minor capacity. They help to show things like the weakness of NV PS2.0 engine compared to ATI's PS2.0 in the R3xx. That type of info is useful.


The issues that NV had with 3dmark2k3 are basically the same issues that existed from the very first version of 3dmarks. Its only when their cards were no longer in the top that they were kind enough to share with us (you all do remember the good 'ol nVidia optimizer that would read 3dmark scores and report the best NV card to increase your game performance). Granted there were a few specific issues with 3dmark2k3. But goodness. Pull out of the benchmark's beta program, threaten legal action when the word "cheat" was used, then go back into the same program you pulled out of? Then the amount of cheats in 3dmark is hard to keep track of, static clip planes, zbuffer tricks, using lower IQ shaders, ect Its amazing this crap that has gone on.

We always want to give significant and accurate results to our readers. We have been continuing to include 3DMark03 scores because we thought it was a benchmark that many still thought was significant despite nVidia's optimizations. We now realize that due to nVidia's actions, 3DMark03 scores are insignificant and inaccurate. We can not change this. Only FutureMark and/or nVidia can bring this benchmark back to life.

Actually you can do something about it. How do you think NV would respond if sites stopped review their cards until they cleaned up their cheats in the drivers? Yes a smaller sight would not do much. But you get one major site to do this, then I am sure someone over their would get this message. Maybe you don't have to do something so drastic, but a blurb on every review about cheats would also help to get the point across. Look what happened 2 years ago with the ATI Q3 Quake/Qauke thingy. Bug/Cheat what ever it was ATI got slammed over the coals about this crap. And they deserved it. However that "tough" love has forced ATI to get its act together. Now they have turned around their drivers and have learned their lesson. Maybe its time for the same? NV showed us their optimization guide which they break with their drivers (optimization must optimize more than a benchmark). So that tells me they are talking the talk. Its up to you to make them walk the walk.


But to set there and pull a benchmark when it provides some useful info (when used in context and with out cheats) is sending the message to nV hay its ok to cheat. NV wants to discredit 3dmark. Your helping to do that by NOT using it anymore. Are you also going to pull any benchmarks that one IHV cheats on as well (like your doing with 3dmark)? If so your going to to have to pull all D3D benches(lack of tri-linear on the FX cards) as some may find lack of tri-linear something which is been a standard on video cards for 3+ years now? Then Firing Squad showed that if you run the stock time demos and compare them to custom time demos, the lead that one IHV has suddenly is gone.

[Illuminati]

But to set there and pull a benchmark when it provides some useful info (when used in context and with out cheats) is sending the message to nV hay its ok to cheat. NV wants to discredit 3dmark. Your helping to do that by NOT using it anymore.

I'm not sure I follow your logic. nVidia is part of the development program at FutureMark... why whould nVidia want sites to stop using a benchmark that they are spending a lot of money to look good in?

Please feel free to explain further if my comment does not make sence. Thanks!

[Apoptosis]

JB,

Some very strong points in your comments there! I think that NV and FM should have worked this out behind closed doors. Since FM and NV can't work this out Future Mark is now "teaching" review sites how to benchmark with 3dmark2003. This is great and helpful to us, but why don't they work out the "cheating" issue with nvidia dirctly? Is it because they are on nvidia's payroll and don't want to risk loosing funding? Is it because they can't? Or have they simply given up?

Lastly, about the stock time demos vs. custom time demos.. Interesting stuff huh? On my upcoming prescott review I used custom demos/utilities for UT2003 and Call of Duty just to avoid possible issues. Sad how far people will go to make something look better than it is.

Thanks for the comments and a thought to leave you with...

It's a canned benchmark that is worthless in the real world. (or am I missing a use for it?)

[Scortch]

I don't understand why you even bother using 3DMark anyway. All the other review sites have proven it's no longer a valid and accurate benchmark and have moved on to more accurate ways of testing cards.

Futuremark opened this can of worms when they allowed the video card manufacturers access to their source code for the benchmark. FUTUREMARK rendered the benchmark invalid and inaccurate when they did this.

3DMark has already been dead for over a year. Some sites just refuse to believe this and continue using a useless benchmarking program to compare video cards.

The only thing 3DMark is good for is testing to see what kind of impact a change has made on your system.

It's time to move onto a more accurate and reliable benchmark program like most of the other test sites have done and forget about 3DMark. It lost it's validity long ago. Move on like everyone else has.

[jb]

I'm not sure I follow your logic. NVIDIA is part of the development program at FutureMark... why would NVIDIA want sites to stop using a benchmark that they are spending a lot of money to look good in?

Sorry about parts of my post as I had some typos and poor grammar. I wrote most of it during my lunch hour so I was trying to multi-task with a ham sandwich and chips at the same time


What I meant was that NV has done a 360 with 3dmark2k3. Remember during the launch of the GF3 back that NV also had 3dmark launch 3dmark2001 on the same day. In fact 3dmark2k1 was used during the launch event to show us the power of the GF3 PS engine with the Nature test. Then for the next 2 years it was all 3dmark2k1. Every web site had it. NV was fully behind support of it as its showed NV in the top spot un-touchable by anyone else. NV even markets their PA that uses 3dmarks scores. So every where you looked 3dmark was da bomb. Did you see one paper from NV saying back then that 3dmark was not representative of (actual) games, nor is it a good benchmark? Nope. Everyone loved 3dmark2k1.

Then you had a few events that challenged this belief. The launch of the R300 that shocked us all. The utter failure that was the NV30 in Nov of 2002. Beyond3d in Feb of 2k3 shows us that the "8 pipe" NV30 is really a 4 pipe design (8 with stencil ops). And reviews of the NV30 show it to be poor in most aspect to the R9700Pro. Then on Feb11 we have 3dmark2k3 officially launched. And once we see the DX9 performance between the R9700pro and the NV30..well yet it was that bad. Note other DX9 tests written by other users showed this same piss poor DX9 performance (Shadermark, RightMark, ect).

The same day of 3dmark2k3 launch we have this from NV:
http://www.extremetech.com/article2/0,3 ... 239,00.asp
Seems like good o'l NV does not like 3dmark anymore and has pulled out of the beta program since dec of 2002. (which is about when the were testing the first pass of their FX lines). Their take:

"The reason that we're not all gung ho about it is that (3DMark'03) is not representative of (actual) games, nor is it a good benchmark," said Tony Tamasi, senior director of desktop product management at Nvidia. "That means Nvidia has to expend effort to make sure it runs well on our hardware. All that energy that we spend doesn't benefit the user. None. Zero. All that effort doesn't go to benefit any game, either. That's kind of depressing."

We have B3D/ExtreemTech sites that have shown how NV has cheated at that time with clips plans and all shorts of BS. On May, FM responds with their take on this mess using the word Cheat: http://www.futuremark.com/news/?newsart ... 2003052308 along with a new patch 330 to get around those cheats.

NV claims:

Since NVIDIA is not part in the FutureMark beta program (a program which costs of hundreds of thousands of dollars to participate in) we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer. We don't know what they did but it looks like they have intentionally tried to create a scenario that makes our products look bad. This is obvious since our relative performance on games like Unreal Tournament 2003 and Doom3 shows that The GeForce FX 5900 is by far the fastest graphics on the market today.

Then just 10 days later, NV and FM get all nice again and announce a make up: http://www.futuremark.com/news/?newsart ... 2003060305 And only then does NV rejoin the beta program of 3dmark.

Yea optimizations? BS. There is no way in hell you can have static clip plane inserted into 3dmark be anything but a blatant attempt to inflate 3dmark score. I don't care if you don't like 3dmark at this point, but we can all see that is just flat out wrong. Way wrong. Weather we hate 3dmark2k3 or not, we still have to realize that lots of OEMs and other people use it to making buying choices off. And to mislead those people with cheats again way wrong.

The issues that NV brought up on the day of 3dmark2k3 launch are valid but they have ALWAYS been valid with EVERY version of 3dmark. Was 3dmark2k1 any better? Heck no. For example we have the K2 video card that got killed by 1000+ 3dmarks when compared to a GF2mx card. Yet in DX7/DX8 games (max payne, UT2k3) the K2 was usually about x2 faster than that GF2 MX card. That's just one example of how "good" 3dmark2k1 was. The fact that it was based off a real engine is point less. Look at q3 scores. Its a real engine right? Yet we know that every game based off the q3 engine runs differently. Thus Benchmarks from Q3 != Benches from RtCW != SOFII != COD != JK2 != JK:A != Star Trek != Alice ect. You see all of the games are based off Q3 engine (some are modified Q3 engines) but if you have such wilding varying FPS scores on the SAME engine..then does it even count? No. Every game will have its own bottle necks, slowdowns and other issues. Thus all of NV "issues" with 3dmark2k3 are moot because they have always been there. And it was not until NV said something about 3dmark2k3 that other web sites started to follow. I mean why do we all hate 3dmark2k3 so much when we loved 3kmark2k1 as they have the same issues?

Well there you have it.. Some where I was trying to make a point but I am not sure where now

This is great and helpful to us, but why don't they work out the "cheating" issue with nvidia dirctly? Is it because they are on nvidia's payroll and don't want to risk loosing funding? Is it because they can't? Or have they simply given up?

Mind you I was not part of the talks but do have some inside contacts. $$$$$$ is the reason why. NV has how much cash on hand? FM has about 1/1000 of that; for one reason. The other is legal action. Do not think for a minute the turn about was not forced by legal action. Claiming cheat is a serious thing here in the states. And NV has lawyers. FM does not per say. One of the production leaders of 3Dmark mark quit over this issue. They have tried to do but NV refuses to remove the cheats. What would you do in that case? You have seen NV optimization guide lines haven't you?

An optimization must produce the correct image.

An optimization must accelerate more than just a benchmark.

An optimization must not contain a pre-computed state.

And yet you have shown that with the last drivers they go against their OWN guide lines? If NV is not gonna to stick to THEIR own rules, how are they gonna to follow others rules then? I don't have an answer for that one.

Lastly, about the stock time demos vs. custom time demos.. Interesting stuff huh? On my upcoming prescott review I used custom demos/utilities for UT2003 and Call of Duty just to avoid possible issues. Sad how far people will go to make something look better than it is.

Yeap its sad. But until more sites stand up and take measures again INVs that allow this practice then we all will have to live with it. Pitty eh? And the whole lack of Tri-linear is an issue in my book is a pitty. I mean good grief, Tri-linear is so 1999......

It's a canned benchmark that is worthless in the real world. (or am I missing a use for it?)

Its up to your style and what info do you want to give. I still feel that all synthetics have their place. The goal of 3dmark was never to tell you what FPS you will get in the next gen games. But give you a way to compare two different cards to see which one has the potential of being "better" in those games. We all know that the FX line has an lower performing PS2.0 engine vrs the R3xx cards. How did we know this? Synthetics showed us this case 9 months ago. Recently we have developers that have also said this, Gabe at Valve and JohnC.

Hi John,

No doubt you heard about GeForce FX fiasco in Half-Life 2. In your opinion, are these results representative for future DX9 games (including Doom III) or is it just a special case of HL2 code preferring ATI features, as NVIDIA suggests?

"Unfortunately, it will probably be representative of most DX9 games. Doom has a custom back end that uses the lower precisions on the GF-FX, but when you run it with standard fragment programs just like ATI, it is a lot slower. The precision doesn't really matter to Doom, but that won't be a reasonable option in future games designed around DX9 level hardware as a minimum spec.

John Carmack"

And today's DX9 games show they tend to run just a bit faster on the ATI cards which now makes it complete. Synthetics told us one was faster a long time ago, developers also said the same thing and now DX9 games also show the same thing. Games like the FarCry demo for example run PS2.0 on the R3xx cards but if you try this on an FX card, it drops back to PS1.1

What about your readers? If you have JoeSixPack that can only upgrade once every other year. Then he has $200 to spend on new card which what should he get? If he was in the market a few months ago, then alls he had was DX7/8 games to look at. And we know both the new cards run DX7/8 games just fine. However since JoeSixPack's card will have to survive some DX9 games as he will have that card for 2 years. Then don't you think its your job as a reviewer to let him now that one card has a potential weakness in DX9? That that card might have to drop back to partial precision hints or drop back to DX8 (PS1.1) shaders vrs running as a true DX9 card as its advertised on the box? I think JoeSixPack as a right to know this. Don't you? Until we get more DX9 games out you can not used today's DX9 games as the sole indicator of DX9 performance. Nor can you use DX7/8 games to show DX9 performance. What else can you use but synthetics to give you a general idea of where the products are in terms of DX9 power?

I am not saying use the 3dmark score (that you might as well file to /dev/null as its useless). However some of the other DX9 test that 3dmark has or the other synthetics will help to give you a clue to potential issues. The key is to know how and when to use them (as well as which is worth reporting). And last time I knew giving more info was a good thing

[Apoptosis]

Very good points you brought up and I must agree with the majority of them. I am currently speaking with Justin West (Our Video Card Reviewer) to see what we can do to better our DX9 benchmarks and what can be used to better our results for our readers. If you have any benchmarks or tests you feel are better or are a subsitute feel free to let us know.

[Illuminati]

I also agree with the jist of your post... and learned a few things, too. I agree completely that we need synthetic benchmarks to make those future predictions. When I announced that I will stop using 3DMark03, that did not generalize all synthetic benchmarks.

I am currently analyzing various synthetic benchmarks and taking note of which ones show inconsistent results and which ones fit a purpose in our benchmark suite. And like Nate mentioned... Feel free to give us suggestions for which synthetic benchmarks you believe to give fair and consistent results.

Thanks for the time you gave us to write that informative post!

[jb]

I also agree with the jist of your post... and learned a few things, too. I agree completely that we need synthetic benchmarks to make those future predictions. When I announced that I will stop using 3DMark03, that did not generalize all synthetic benchmarks.

Well good then. I think that when used correctly they offer some (abiet small) usefull info.

I am currently analyzing various synthetic benchmarks and taking note of which ones show inconsistent results and which ones fit a purpose in our benchmark suite. And like Nate mentioned... Feel free to give us suggestions for which synthetic benchmarks you believe to give fair and consistent results.

I am by no means an expert. Currently I do like ShaderMark as thats done by an individual so he can react much quicker to changes than any company can, he follows HLSL and not some hand coded shaders like in 3d/aqua mark: http://www.shadermark.de/start.html

Shader mark can also make use of the PP hints to allow partial precision which developers are using (for example a little know factoid, that Halo for the PC used PP in its DX9 PS2.0 functions on the FX cards were ATI cards run at full 24bit mode). The author of shadermark is a B3D regular so gotta give him props

Maybe you can contact Dave B over at Beyond3d.com for his input as he has a wealth of information and first hand experince.

Thanks for the time you gave us to write that informative post!

Finally I am not an Anti-nVidiot by anymeans. I have really enjoyed their hardware. Its their PR/Marketing department that I have grown to deeply regreat. Most of NV issues can be tied back to their PR campain (except for the lower performing PS scores). And in fact its their PR that has dug them a deep. Hole. How deep does the whole go? Click here to find out (but warring its not a very nice trip)

http://www.beyond3d.com/forum/viewtopic.php?t=8282

Again I am not trying to hurt NV. I am glad that some of the truths that have been uncovered will come to light so more people will be come aware of the issues. And then sooner or later we can get these issue resolved. NV has made some kick butt hardware that can stand on its own. I would love to see them do it again. But for now, its clear that ATI is the top dog. In the end seeing ATI/NV duke it out means better products for us. Just that we need some policing to ensure its a fair fight

Glad I could be of some help

[cjb110]

A quick idea about the problem of driver optimization.

Can't futuremark add a software mode, ok it will be slow but surely it will produce the exact image that the graphics card should also produce?

And surely as long as the card is producing the exact image does it matter if there has been some optimizations?

[jb]

A quick idea about the problem of driver optimization.

Can't futuremark add a software mode, ok it will be slow but surely it will produce the exact image that the graphics card should also produce?

And surely as long as the card is producing the exact image does it matter if there has been some optimizations?

Well if 3dmark was anything but a benchmark then sure. On any other application who cares so long as the image is the same. The problem is its suppose to be a benchmark. Meaning it wants to set up to ensure that every card does the same ammount of work, following the same rules and produces the same results. If it allows one card to optimize then that card is doing less work and thus you no longer have an "apples" to "apples" comparison.

Personally I view any optimization for just a synthetic benchmark as a "cheat". But meh...thats me

[Illuminati]

Personally I view any optimization for just a synthetic benchmark as a "cheat". But meh...thats me

That's my opinion as well.

[cjb110]

suppose, though surely if you cannot see the optimization then its being clever and not cheating? :D

but again optimizing for a benchmark is a waste of effort if it only applies to that one situation...ah well just a thought.

[Weeidgit]

I am currently analyzing various synthetic benchmarks and taking note of which ones show inconsistent results and which ones fit a purpose in our benchmark suite. And like Nate mentioned... Feel free to give us suggestions for which synthetic benchmarks you believe to give fair and consistent results.

Thanks for the time you gave us to write that informative post!

IMO you have just got rid of one of the consistent benchmarks. Does this mean you will re-include 3DMark03? I still fail to see the reason for removing that as a benchmark, all it takes is a 5 minute driver change

[Illuminati]

I am currently analyzing various synthetic benchmarks and taking note of which ones show inconsistent results and which ones fit a purpose in our benchmark suite. And like Nate mentioned... Feel free to give us suggestions for which synthetic benchmarks you believe to give fair and consistent results.

Thanks for the time you gave us to write that informative post!

IMO you have just got rid of one of the consistent benchmarks. Does this mean you will re-include 3DMark03? I still fail to see the reason for removing that as a benchmark, all it takes is a 5 minute driver change

We do not agree with changing the driver just for one benchmark. Not because it takes too much work or because we are lazy... but because we believe it is important to run all tests using the same drivers to maintain a Standard Operating Procedure. That is just one of the reasons for dropping 3DMark03 from our test suite.

Welcome to our forum BTW!

[just me]

Hi,

Just read the article, very well written.

Seems odd to me that you'd drop 3DM03 thinking it is the only benchmark nV is optimizing for. Isn't that a little naive, on your part? :wink:

nV stated from the very beginning that 3DM03 is not representitive of gaming & they dropped out of the FM beta program before 03 was released because they knew their decision to go FP32 was a bad one & their cards would perform poorly in the benchmark.

Yes they are back in the FM beta program, but that doesn't mean they didn't do it to 'bring the benchmark down' from w/in. Lot easier than trying to do it from the outside. :wink:

nV doesn't like FRAPS, 3DM03, canned benches, Unwinders anti-detect script, etc. & says to use games. The problem with that is > you have to use games 'the way they were meant to be played' optimized or nV bitches about that too.

You guys really need to research what has happened (including nV public statements thru Derek Perez that nV will continue to 'optimize') w/ALL benchmarks (are you aware of Unwinder anti-detect script?) before you drop 3DM03, IMO.

3DM03 is the only bench that is actively monitored for cheats > the others aren't.

Keep using '03, contact Dave or Rev @ Beyond 3D for an insight into benchmarking & acquaint yourselves w/Unwinders anti-detect script.



.02,