Like others said, wax and clay bar should work.
I'd shy away from using a buffer, especially if you don't use one often. I've seen a lot of fender edges rubbed down to the primer because they buffed an edge too much and didn't know it or ran the buffer the wrong way up to the edge.