I know that for pinhole cameras and camera obscuras the holes are usually round (they are circular). But I don't see the link between the shape of the hole and the formation of the image.
From Wikipedia's Pinhole camera article,
The best pinhole is perfectly round (since irregularities cause higher-order diffraction effects), and in an extremely thin piece of material. Industrially produced pinholes benefit from laser etching, but a hobbyist can still produce pinholes of sufficiently high quality for photographic work.
There is a theoretical optimal size to a pinhole aperture. From a geometric standpoint, the smaller the pinhole aperture, the sharper the produced image will be — but this holds only to a point. Smaller than this size, the image sharpness becomes diffraction limited. The wave-like behavior of the light interacting with the aperture walls creates its own interference pattern, and at (or beyond) the diffraction-limited aperture size, this dominates the image clarity.
A circle is the only shape with constant curvature. Any other shape will have either tighter curves or closer edges than a circle. If you strictly maintain minimum curvature and edge distance, but neither are constant, then by definition you have part of the shape that is larger than it needs to be. Thus, a circle is the optimal shape to balance the desire for a smaller aperture (for image clarity purposes) with a lower limit to aperture size due to diffraction limitation.
The pin-hole casts (projects) tiny image circles on the back wall of the camera. This is also the place we position photo sensitive material so that we can replica this image on film or paper or perhaps even digitize it. For the image to appear sharp, the image circles need to be so tiny that they appear as points without perceivable shape. The criteria: ½ mm diameter viewed from normal reading distance 14 – 16 inches (355 - 400mm).
Now the image cast by the pin-hole is quire feeble. You would be tempted to enlarge the pin-hole in an attempt to gain image brightness. As you enlarge the pin-hole, the image circles enlarge and this degrades the sharpness of the image.
Likewise, you might be tempted to reduce the pin-hole diameter to gain additional acuity. As the pin-hole shrinks the image dims and acuity is gained, to a point. Soon the twin demons of diffraction and interference take their tolls. Diffraction is an edge effect, light rays that encounter a sharp edge do not stop abruptly, they bleed into unobstructed rays near the boundaries of the obstacle (pin-hole border). Interference also caused at the boundary edge. This is both a nullification and a reinforcement of the light rays. These two phenomena are jointly responsible for a loss of acuity as the pin-hole is compressed.
Faced with these twin demons, it is best if the shoreline of the pin-hole be as short as possible. This will be a geometric figure that is regular and as compacted as possible, In other words, a circular aperture.
Let me add that lenses replace the pin-hole to gain acuity. In most instances the aperture they sport is also a circle of the same reasons as above. However, in modern times, out-of-focus backgrounds are deemed attractive. They add a soft buttery look that is especially prized for some types of portraiture. This effect, in the jargon of photography is bokeh (a Japanese word for this effect). If the shape of the lens aperture is altered to a 5 or 6 sided figure, or given pointy appendages, the bokeh is heightened and some like this very much.
First of all, I am going to be practical. Here is an example of a sharpening tool.
A good way to have a sharp tip is by doing a circular rotation while sharpening a pencil. Not only when using the pencil sharpener, but you can also rotate the pencil while drawing on a paper to produce a very thin tip.
Let's explore other shapes.
I am exaggerating the shape of an ellipse, but it is obvious that an ellipse has more area than a circle in this case. The tool that made this shape could be sharpened further by doing a circular motion.
But now let us answer the second part of the question
I don't see the link between the shape of the hole and the formation of the image.
The image produced by a pinhole is not perfectly sharp. It has a small blur due several rays not hitting the exact same spot.
But if you use a non-circular shape, for example, an ellipse, you now introduce another type of blur, astigmatism, which is a blur on the same angle as the shape of the ellipse.
And finally, shapes that have acute corners at some point make those corners susceptible to produce interference patterns due to diffraction of light.