It's complicated but I'll have a go at explaining it. Light from a single point spreads out in all directions and hits the entire front surface of the lens, this light gets focused back down onto a tiny dot by the lens if the lens is focussed at the distance of the point. If not (i.e. the point is part of the background) the light lands on the sensor in a big circle and creates bokeh.
If you put a shape on the front of the lens you block out bits of the spread of light from the point and the result is that instead of a circle, your shape is projected onto the sensor.
Your shape has to be very close to the lens for this, otherwise the light from the point wont have spread far enough by the time you start occluding it.
I would just expect your image to be in that shape (like looking through a keyhole) with the outside of the shape being black.
The reason you don't simply see an image in the shape of your bokeh filter like you suggest is that the filter is very close to the front of the lens and so doesn't block the centre of the lens. Any light which hits some part of the lens front element has a chance to get into the picture, so there are no black edges etc.
If the shape is a little distance in front of the lens, then there will be parts of the scene which are blocked from hitting any of the lens surface, these areas will appear black in your image.