Exposure compensation (for that is its name) allows you to adjust the exposure of a shot in the semi-automatic modes in situations where you think the light meter might be reporting 'incorrectly'. It essentially alters the zero-point of the light meter.
Consider this situation. You are taking a shot of a man dressed in a light grey suit against a black background, in Aperture priority mode. In matrix/evaluative metering, the camera's light meter will look at the whole scene, and come to the conclusion that it's dark.
Since all light meters 'want' the camera to render any scene as 18% grey, it will adjust the shutter speed accordingly, and you will have a grey wall and a vastly overexposed subject (the man). Centre-weighted metering (assuming the man is in the centre of the shot) will likely return slightly better results, but it will still take the black wall into account and overexpose slightly.
With exposure compensation, you can still have the benefits of an automatically set shutter speed (or aperture in Shutter priority mode), but you can also override it in situations where you know the light meter will likely be fooled. In the situation above, you would set the exposure compensation down a stop or so, and the result would be a well exposed subject with a black background.
What I have described here is somewhat proactive. In reality you are more likely to take a shot, check it on the LCD, note that it is either under or overexposed, then set the exposure compensation accordingly and retake.
Exposure compensation systems are also often used for HDR photography, either manually by the photographer or automatically by the camera in the case of Auto Exposure Bracketing.