I once experimented with standing in a dark room and directing a laser pointer towards my digital camera with a long shutter speed while moving the two about. I made a few images that way; one was quite satisfying once the colours were altered in GIMP. But, after the first few, it occurred to me that doing this might risk damage to the sensor. Is there a real worry, here?
Oh man. Yes lasers can and will damage your camera's sensor! Your laser pointer is probably weak enough, but I still wouldn't risk it.
Light painting (as you were doing), is okay, so long as you're not pointing your laser directly into the lens, however, if you plan on doing more light painting, I would switch to an LED or incandescent source instead of a laser.
I would be concerned about the sensor's Bayer filter. If the red, green and blue dyes in the filter become discolored by your laser then your camera becomes color blind.
I would also be very concerned about your eyes. Never look through the viewfinder when playing with laser pointers. Use a camera with live view. If the laser destroys your camera you can buy a new one. If it destroys your eyes, the damage will be permanent.
Laser pointers for presentations are typically in the 5 mW range. Lasers used in concerts are substantially stronger (10-10000x). Those have been shown to permanently damage a digi cam sensor (usually straight burn lines across the entire sensor). I haven't done any tests, but can't imagine sensors being damaged by small laser pointers. Given @Menace's post that a sensor was damaged at 200 mW, I'd say anything below 5 mW might be quite safe.
I wouldn't worry too much... it's taken them 50 years to work out to generate enough power to shoot down an aeroplane, and they are using considerably more power than the AA battery in a hand held pointer has...
Having said that, small torches (maglights or similar) are fine for light painting. When I've done it I've never pointed the light source at the camera, but "painted" over the object I'm photographing: http://www.flickr.com/photos/mrbennstep/2242968500/