Lifehacker has a couple of good articles on improving and cleaning your PC. The first deals with speeding up an old laptop. I would say (as they do in the article) that most people won’t need 4GB of RAM and the improvement from 2GB to 4GB will be minimal, so I’m not sure I’d bother. A Solid State Drive is not a bad idea, depending on how old your computer is, laptop or desktop, and they’re not difficult to install. They are price though.
Cleaning up your machine is a great, usually free, way to get some extra performance though. Which leads into their second article. Admittedly this covers more than just your computer and is basically just a link to some older articles, but it’s still valid.
Reinstalling your operating system, while a laborious process, does return the machine to original state and you’d be surprised how much faster it runs. Removing any old applications, performing disk de-fragmentation and clean-up also offers some boosts. If your hard drive is running out of space, move files onto an external (or separate disk) and you should get some improvements too.