I use two methods for everything like this, though one of them is a little controversial.
First method, take a piece of rubber and gently rub away the dirt then get some paper, newspaper is good but printing paper will get it done - I also recently found that the card which Rizla boxes are made from is good - and rub that along the contacts. Rubbing card or paper on contacts is also a good idea if you make something from stripboard as it seems to make the copper cleaner and eases soldering.
Second method, brasso, gently applied to the contacts and left to sit until it turns white - DO NOT rub the contacts with the wadding! - just brush the solution onto the contacts. Problem with brasso is it tends to hang around, lighter fluid or alcohol will see to that (Swan's N²0-based one works best), dampen a tissue with it and rub the brasso away, you can then use the card method to remove anything that is left over.
Furthermore, brasso seems to work very well on heatsinks, even with a brand new copper heatsink (was a water block actually) I was able to lower temperatures by 3-7°C by polishing the surface or the heatsink and chip - you do apply force this time, leaving it to sit a while and then removing the leftovers with petrol. It seems to be good at removing some glues and most thermal compounds too.
Brasso also works with rust, but WD-40 gets it out of the way better, you can then clean that away, give brasso a go (because it removes the WD-40) and then clean that away as mentioned above.
Of course, if you're going to use brasso / alcohol / petrol you need a well ventilated area, many a time I've come away with my head spinning. Another thing to note is that it might react with the glue and thermal compounds (usually any left-over paste turns black) and I can't guarantee the safety of the fumes that may come from it...
Also keep in mind that this is coming from the man that attempted to use electrolysis to remove glue from a heatsink and almost passed out from the resulting gases... I did end up with a smooth, clean surface however.