It really depends on the camera being used and how it is designed. There are certain things that the camera should do before a lens is removed such as ensuring the mirror is down to protect the sensor and potentially disengaging mechanical links to drive aperture and focus (though these are generally designed to be self-disengaging when you remove the lens.)
Many cameras, however, can tell when you push the lens removal button and can do this process by the time you actually unscrew and remove the lens. The best advice is follow the manual, if it cautions against doing it, then you are best off to listen to the manufacturer that designed your camera. If they don't object to it, you should be fine.
Personally, I use a Canon 5D Mark iii and have no problems with removing the lens on the fly and reattaching a new one. I've even done this in relatively dusty environments without a problem yet (over the course of a year). The only thing it specifies is that if you are going to use a teleconverter, you should attach it to the lens first so that the correct information is available when you attach the lens assembly to the camera.
As far as damage that could occur, the most likely issue would be that something could happen to the sensor or mirror as the mirror moves to try and cover the sensor as you are removing the lens or if the mirror doesn't move at all, leaving the sensor directly exposed to outside contaminants (such as dust). There could be other issues specific to your camera body though and depending on lens design, it could be possible (however unlikely), for example for the mirror to collide with part of the lens. This would do substantially more damage and possibly break the mirror, rendering the camera effectively useless until repairs can be made.
While I can't prove it, if the sensor having a charge actually could draw in dust, then why on earth wouldn't camera makers cut the power to it when the lens is removed. If they can do it when the camera is powered off, they can do it when the lens is removed too and it would be bad design not to (either that or it simply is an old wives tail that it's a problem).
One final thought, it may actually be worse to power off for a change. While there isn't any evidence of dust attracting being an issue, powering off and powering on does potentially result in extra self-cleaning cycles as well as other power off and power on activities. These activities have a much higher chance of causing wear and tear on the camera and resulting in eventual damage than the remote possibility that it might attract dust (which could simply be cleaned off). I doubt that either of these is even a measurable contributor, but if I had to gamble, I'd gamble that turning off for every lens change is actually more harmful (by an insignificantly small amount.)