To directly answer your question, it introduces more variables that can cause damage.
Keep in mind the mechanics of an interchangeable lens camera. You have mechanical and electrical links between the camera and the lens. Having the power on means that there could possibly be gears moving or a current moving between the two and you'd be interrupting this link without affording the camera an opportunity to finish what it's doing. While most cameras don't have a problem with this, it's not the intended method of breaking the link. Powering off the camera will ensure that there isn't a link between the two.
As Chris mentioned, you will attract dust and this will indirectly introduce more opportunities to damage your lens as well.
That said, I've done hundreds, possibly thousands of hot swaps on my 7D and have not had any issues. Most cameras are designed to be able to hot swap. Possible signs of damage would be a failing auto focus or trouble recognizing the lenses.