I think this really depends on the lens and camera body. But, in my experience I have noticed that some of my lenses won't allow me to manually focus unless I switch the lens's focus to manual (as in the focus ring won't move or it won't actually engage the focus). So, personally, I find it easier to just leave the camera body set to AF and toggle the focus on my lenses. I've been doing this for years with no ill effects on either the lens's or camera's auto focus systems.