It depends how crazy you want to go. I think there's a point where you just have to live with things.
Having looked on detailing world, it seems to me that the most sensible option is to use something like AG Super Resin Polish, which is a very gentle polish that also fills the light scratches/swirls. Then wax on top. Many people seem reccomend AG (AutoGlym) Extra Gloss Protection for after the SRP, rather than a pastey wax.
I've gone for Dodo Juice Lime Prime and then Purple Haze wax for after. Basicly they do the same job as the two AG products but they smell nice

I've yet to get a chance to use them though.
I want to protect my paint but I'll not be able to get all the swirls out without a machine and I drive my car so often that it would just be a waste of my time trying to keep it flawless. If it looks a bit swirly, just stand a foot or two further back