It really depends on the camera being used and how it is designed. There are certain things that the camera should do before a lens is removed such as ensuring the mirror is down to protect the sensor and potentially disengaging mechanical links to drive aperture and focus (though these are generally designed to be self-disengaging when you remove the lens.)
Many cameras, however, can tell when you push the lens removal button and can do this process by the time you actually unscrew and remove the lens. The best advice is follow the manual, if it cautions against doing it, then you are best off to listen to the manufacturer that designed your camera. If they don't object to it, you should be fine.
Personally, I use a Canon 5D Mark iii and have no problems with removing the lens on the fly and reattaching a new one. The only thing it specifies is that if you are going to use a teleconverter, you should attach it to the lens first so that the correct information is available when you attach the lens assembly to the camera.
As far as damage that could occur, the most likely issue would be that something could happen to the sensor or mirror as the mirror moves to try and cover the sensor as you are removing the lens or if the mirror doesn't move at all, leaving the sensor directly exposed to outside contaminants (such as dust). There could be other issues specific to your camera body though and depending on lens design, it could be possible (however unlikely), for example for the mirror to collide with part of the lens. This would do substantially more damage and possibly break the mirror, rendering the camera effectively useless until repairs can be made.