I have a similar configuration and am looking for detailed tweaks on the extreme tweaker screen. I am also looking for definitions for what the particular settings control. Any help would be greatly appreciated.
Here is my machine config:
MoBo: ASUS CrossHair
CHIP: NVIDIA nForce® 590 SLI™ MCP
CPU: AMD Athlon 64 X2 6000+ 3.01Ghz
RAM: 4GB Mushkin 996561 (2x2GB) PC6400
VID: BFG 8800GTS 640MB OC2
PSU: 750w Thermaltake ToughPower
OS: Vista Ultimate 64bit
SATA: Maxtor7L259S0 250GB 16MB Buffer
IDE: Optical Drive HP DVD 940
FLOPPY: generic 1.44
I don't know the specific settings in the BIOS for that board, but you should leave the memory settings on AUTO to start with.
As BiGg RiE mentioned, test the RAM first. There should be a CPU multiplier option in the BIOS, lower this to ~7x and then start to raise the fsb. When things get unstable, try adding some memory voltage and or chip set voltage. I notice you have 2GB sticks, these don't normally respond well to much voltage and you'll have to increase the timings to ~5-5-5-18
Once you find the limit of the RAM, set it to run @ ~533MHz, and increase the CPU multiplier back to stock along with the FSB. Then increase the FSB again to find the max the CPU can do.
After you find out the max for your parts, you can then find a happy medium.