1. Can the old FX-5200 be overclocked ? - I have no clue as I never tried to overclock that card. Just try it and find out. I'm thinking that it can but the results will be severely limited and mostly unnoticeable.
2. How can i know that I am overclocking my card in a right way ? How can i avoid mistakes? - Read the directions that come with Rivatuner or whichever overclocking software you choose. Most people will just increase the core clock and memory clock speeds by small increments until they notice artifacts...I just read the guide and experimented. You can also search the internet to see what overclocks others have gotten on the card.
I also use HW monitor to monitor my temps. Watch that the temps don't climb too high as that will make the card unstable. Sometimes you can increase the fan speed. In somes cards that makes a big difference, in others with small fans, it doesn't matter a whole lot. Some cards only have a heatsink (fx 5200) and no fan. I have seen people install fans to keep them cooler, but as I mentioned before, I doubt you will see a noticeable difference with that AGP card.
I suspect that the 8800GS card would overclock quite well.
Read this
Article