I was cleaning up the reflection mirror under the strong sunlight in an afternoon, after that I realized that could potentially damage my sensor, since I found the mirror is somewhat semi-translucent, am I overly worried?
As long as the shutter was closed the sensor itself would be protected by the shutter curtain from the light of the sun. I have heard of shutter curtains being damaged by sunlight, but that is usually in the context of a powerful telephoto lens being mounted to the camera and pointed towards the sun on an older camera with cloth curtains. On the 5DIII, the semi-translucent part of the mirror allows light to pass to the Auto Focus sensor array. There is a small chance that too much light could damage the focus array, but I doubt unfocused sunlight when there is no lens mounted could do so.
What I would be more concerned about is damaging the mirror. It is the most fragile part of your camera and should never be touched on the surface. Unlike most mirrors in other applications that have the reflective coating on the back of the glass, the mirrors in cameras have the coating on the front surface and it is easily damaged. At most, use an air blower to gently knock dust off of the mirror.