GeForce GTX 280 Stopped Folding - Serious Error

[Apoptosis]

Think I might have wounded a GeForce GTX 280 graphics card by folding for nearly 72 hours non-stop... The card with default clock settings returns this:

serious_error.jpg (11.13 KiB) Viewed 6714 times

When I look into the log file I get this... What the heck could be going on?

[14:45:27] Folding@Home GPU Core - Beta
[14:45:27] Version 1.06 (Mon Jun 23 10:53:13 PDT 2008)
[14:45:27]
[14:45:27] Compiler :
[14:45:27] Build host: amoeba
[14:45:27] Preparing to commence simulation
[14:45:27] - Looking at optimizations...
[14:45:27] - Created dyn
[14:45:27] - Files status OK
[14:45:27] - Expanded 43895 -> 244433 (decompressed 556.8 percent)
[14:45:27] Called DecompressByteArray: compressed_data_size=43895 data_size=244433, decompressed_data_size=244433 diff=0
[14:45:27] - Digital signature verified
[14:45:27]
[14:45:27] Project: 5007 (Run 3, Clone 10, Gen
[14:45:27]
[14:45:27] Assembly optimizations on if available.
[14:45:27] Entering M.D.
[14:45:30] Opening C:\Users\Nate\AppData\Roaming\Folding@home-gpu\MyFolding.html...
[14:45:37] Working on 576 p5005_supervillin_e1
[14:45:38] mdrun_gpu returned -1
[14:45:38] Going to send back what have done.
[14:45:39] logfile size: 9604 info=9604 bed=25 hdr=1
[14:45:39] - Writing 10142 bytes of core data to disk...
[14:45:39] Done: 9630 -> 3457 (compressed to 35.8 percent)
[14:45:39] ... Done.
[14:45:39]
[14:45:39] Folding@home Core Shutdown: UNSTABLE_MACHINE
[14:45:43] CoreStatus = 7A (122)
[14:45:43] Client-core communications error: ERROR 0x7a
[14:45:43] This is a sign of more serious problems, shutting down.

[Apoptosis]

well I restarted the machine... same error...

un-installed the F@H GPU2 software and re-installed it and now it is up and running again... Let's see how it does... That error scared the crap out of me...

[DMB2000uk]

Well i had heard that the GPU2 client errors every 4 days and needs to be restarted, but i don't think they quite meant having to re-install everything.

What are the temps like on the GTX?

Dan

[Apoptosis]

high 60's/low 70's, but I manually set the fan to run at 70%

[vbironchef]

Sounds like a blowdryer set on high heat, doesn.t it.

[DMB2000uk]

high 60's/low 70's...

That's not even that high

I presume further back on the log it was folding fine (i.e. at the normal time per step) just before it started throwing the "mdrun_gpu returned -1" errors?

Dan

[vbironchef]

The heat coming out the back of my rig could make toast.

[KnightRid]

Well i had heard that the GPU2 client errors every 4 days and needs to be restarted, but i don't think they quite meant having to re-install everything.

What are the temps like on the GTX?

Dan

I havent had to restart mine at all, I just have problems when they try to send updates.

overclocked to 650/1625/971 I keep mine at around 60C with rivatuner usually having the fan 60% below 60C and 80% at 60-like68 or something, then 100

I DID have trouble with the regular nvidia drivers, but once I installed the CUDA drivers and stuff, it has worked fine. Are you using the CUDA or the regular drivers?
Maybe if it happens again, try the other driver set and see how it works.

Mike

[Apoptosis]

the official cuda drivers 177.35 and GPU2 v6 beta 8

[vbironchef]

I am having problems with the gtx 280 as well. Yesterday it was very fast. Just over 1 min per%, now it's taking over 11min per%. I unistaled the program and reinstaled the program and it's still very slow. It's like the gtx 280 is not working at 100%. What should I do? I have the lastest drivers and folding program. Did folding@home throtled back the speed?

[martini161]

is it the same WUs?

[DaIceMan]

I am having problems with the gtx 280 as well. Yesterday it was very fast. Just over 1 min per%, now it's taking over 11min per%. I unistaled the program and reinstaled the program and it's still very slow. It's like the gtx 280 is not working at 100%. What should I do? I have the lastest drivers and folding program. Did folding@home throtled back the speed?

I had a similar problem with Firefox and it's memory leak. I was monitoring the logs on one of my rigs and it went from 1m22s a step to over 4m a step. Checked that no virus or spyware scans were going on, then checked Task Manager. Task Manager showed firefox.exe using 160 or 180mb of ram. I killed firefox, and restarted the folding client and everything was back to normal.

Check to see if any other programs have started, virus/spyware scans, etc and start from there.

[vbironchef]

Project: 5008 (Run 3, Clone 67, Gen 6), I do not have firefox. It may just be this project. I am just going to let it run and see what happens. It's just very slow. Do you think my graphics card got damaged? The fan doesn't seem to be blowing as hard as yesterday.

[Apoptosis]

post up the start of the work long from what wu is running till about 20%... I'll compare it to my GTX280 log.

[vbironchef]

Here is what I did: I instaled older version driver. I am now running 1SMP client and the GPU2 client. I gave the GPU2 client it's own core. I set the SMP client on the other 3 cores. I don't know what else to do. Running 2 SMP clients were causing a conflict with the GPU2 client, causing the GPU2 client to crash. I believe that the GPU2 client needs its own core to be happy. As for me I spent the whole day on this. Please tell me if I should do something different. Thank you.

[dicecca112]

Here is what I did: I instaled older version driver. I am now running 1SMP client and the GPU2 client. I gave the GPU2 client it's own core. I set the SMP client on the other 3 cores. I don't know what else to do. Running 2 SMP clients were causing a conflict with the GPU2 client, causing the GPU2 client to crash. I believe that the GPU2 client needs its own core to be happy. As for me I spent the whole day on this. Please tell me if I should do something different. Thank you.

It does need its own core, that's probably why it was erroring out

[vbironchef]

I now just have 2 cores folding SMP client. 1 core GPU2 client. I don't want anymore crashes. I lost 10 hours of folding today.

[vbironchef]

Just updated to latest drivers and GMP2 still not happy with 2 SMP clients running. I'm back to 1 SMP and 1 GMP.

[KnightRid]

the official cuda drivers 177.35 and GPU2 v6 beta 8

Same here, just different card

maybe it is the viewer problems?

I KNOW some of the work units are a LOT slower than others! maybe that is what everyone is seeing that has slowdowns?

Errors suck, but maybe thats why they keep it in beta so long..?? so many differnet cards and setups, they are bound to run into something that causes problems.

Mike

mine runs about 1% every 3 minutes 10 seconds it seems on projects 5007 and 5008, your big boys should be running faster than that on those projects.