This happens. I'm surprised your seeing it with 165 single springs though. The problem is leverage. When the rocker pushes against the valve stem, the force works against the rocker pedestal and pushes it away from the spring. The width of the pedestal where it sits on the head is a factor, a wider footprint would make it more stable. Some of the very early engines had a pedestal with a wider base.
This is the reason girdles exist. It ties all 6 together, and they work together to keep each other from moving.
Pedestal/valve cover machining won't stop the covers from moving, as they bolt directly to the top of the pedestal.