Are APU's a good, cheap alternative to buying a new processor and graphics card?

I already have a PC with basic (but outdated and bad) components and want to cheap way to upgrade my PC on a budget. I just want to run low spec games and programs (Minecraft, Terraria, ect) and don't need a brand new powerhouse of a PC.

Not really. You can't just plug in a graphics card based around an APU and expect the overall performance of your PC to increase. Infact straight off, you'll see zero increase in performance! The only time you see any increase is when you are using software that is specifically written to take advantage of the APU. Frankly when using such software the improvement can be dramatic. But little software is written to take such advantage, because APUs vary so much!