The main problem with using EF-S mount lenses on a FF camera is as you said the risk of damaging the mirror. EF-S lenses can protrude further back into the camera body than EF lenses, which means the mirror might hit the rear of the lense. This is what can cause damage to the mirror itself or the mechanism for flipping the mirror.
For some EF-S mount lenses this might not be a problem, since the lens doesn't extend back far enough to cause the mirror to hit it. Depending on the lens design and if the lens is rear focusing or not. Rear focusing lenses will use the rear group or elements to focus and this can extend the lens backwards into the camera. These lenses can work on a full frame camera at some focus distances, but not others. A zoom lens can also use the same technique when zooming, and cause the same problem.
The answer is really that it depends on the lens if it's safe to use or not.
The image quality will not be as good either, since the EF-s mount lenses will be optimized for smaller image circles than EF mount lenses.