What is the cause of this non-uniform bokeh effect?

Question

A friend of mine is thinking of buying a used medium-format film TLR camera (a Mamiya C330), and he showed me some of the test shots he'd taken with it. I was struck by the curiously non-uniform bokeh in some of the photos, like this one:

^{_{(Subject's face blurred for the sake of privacy, since they're not my kids.)}}

If you look at the background, especially the trees at the top of the picture, you can clearly see that the bokeh is not circular but elliptic, and that the long axis of the ellipse seems to be orthogonal to the line from the center of the image. It almost looks like circular motion blur, as if the camera was rotated during the shot, but the lack of blur in the foreground makes it clear that it's not.

I rather like the effect, especially the way it draws the eye to the center of the picture. (It's not so effective in this particular shot, since there's no strong central subject for the eye to be drawn to, but in some of the other photos with a more central composition it worked really well.) What I'm wondering, however, is what's causing it, and is there a name for it?

I can sort of see how it might arise from the way the light travels diagonally through the iris near the edges of the image, emphasized by the relatively large film format (my own camera, which I've never really noticed such an effect with, is a Nikon DSLR with a comparatively tiny APS-C sensor), but is that really all there is to it, or is there something more complicated going on? And how could I deliberately achieve the same effect, short of switching to medium format myself?

Ps. Here's a close-up of the top of the photo above (click to enlarge):

The picture was taken with the Mamiya-Sekor 80mm f/2.8 lens. Unfortunately, I don't know the exact aperture and shutter speed settings used.

Answer 1

The shape of the bokeh is related to the apparent shape of the aperture of the lens.

Straight on, this will produce a bokeh that is approximately a circle. As the subject moves away from the center of the field, the bokeh starts to look like a sliver of the circle.

This can be reduced by stopping down the lens.

More on this can be read at Shape of the blur patch and Cat's eye effect.

To achieve this affect, you need a wider field of view in the lens (a long lens will never see the aperture from a steep angle) and shoot wide open.

---

The optics of this can be understood by looking at the light rays through the lens system:

However, its not quite this simple as the lens construction itself also plays into the shape of the bokeh.

The only difference between the four images is the lens. Each lens has a different front and rear pupil size and lens blade count and shape. If this effect wasn't caused by the lens, then it would indicate some other phenomena. But instead, we do see a difference between different lenses on the same camera indicating that the lens construction is the cause for the shape and the place one should look to understand the nature of this bokeh.

Ultimately, the cat's eye bokeh is a form of mechanical vignetting similar to when the lens hood is too long for the lens and blocks some of the scene.

This type of bokeh can also be seen in the apparent changing of the shape in DIY bokeh shape filters:

Note the shape of the heart and mickey mouse at the edge of the frame. If one was to go and look at the lens from those point light sources, one would see that the shape of the heart or mickey mouse on the camera is the shape of the bokeh (with some additional adjustments for the apparent shape of the pupil).

Answer 2

This is field curvature. Simple lenses naturally project a curved field, not a flat one to match film or digital sensors. Modern lenses attempt to correct for this, but many older designs do not. In fact, it's sometimes called the "Petzval effect" after a classic design famous for this look.

Interestingly, just this week Sony showed prototypes of a curved digital sensor designed specifically for this reason — by curving the sensor, the lens can be much simpler yet still yield across-the-frame sharpness.

There's more information on (and examples of) field curvature in my answer to What image-quality characteristics make a lens good or bad?, and more technical details at What does "Flat Field Focus" mean? and Do lenses have focal planes or focal spheres/ellipsoids?

---

There is a prevalent Internet theory which holds that this is primarily due to optical vignetting — as the currently-accepted answer here says, "Ultimately, the cat's eye bokeh is a form of mechanical vignetting similar to when the lens hood is too long for the lens and blocks some of the scene." This sounds reasonable, but contradicts what I've seen from non-Internet sources. Fortunately, though, the "lens hood too long" situation is easily replicated, so I decided to do a simple test.

I have an 23mm f/1.4 lens, and it happens that the hood from a 56mm lens also fits, physically. So, I did a series of images, with: no lens hood, the matching 23mm hood, the 56mm hood, and the 56mm hood extended even further via a piece of black construction paper and some ever-handy gaffers' tape. In each case, set the lens to f/1.4 and focused on my friendly little photography assistant, Beefy, for some obvious background highlight artifacts. The results:

We can see that there is an impact on the bokeh circles, and in the last two images, you can see a degree of "swirl", particularly in the paper-tube case. But, the effect isn't very strong compared to the sample image here (or to the one in this other question), even though the corners are visibly blocked by the 56mm hood — or completely blocked by the tube. In fact, if I weren't biased to look for "swirl", here, I might instead describe it as "compressed around the edge of the circle". The physical vignetting theory isn't adequate; something else is going on in lenses which produce this look without losing the corners entirely.

Additionally, check out this crop from the top right edge of the visible circle from the paper-shade test. The highlight shapes transition rapidly from near-circular to half-moons. It's the outer lines of those half-moons which cause the swirl effect in this image. It's hard to tell in this sample, but in most images with the Petzval effect, the transition is more gradual. And, despite the line-diagram illustrating otherwise, these shapes are clearly not "cat's eye". Again, something else is causing this — and, it's the field curvature.

So, to really replicate this effect, you could try extending your lens hood, but for better results, you can either find an actual classic lens with the Petzval design, or some more recent but still "vintage" Soviet-era Helios lenses, or pick up Lomography's recently-launched Petzval art lens for Canon EF or Nikon F mounts.

Answer 3

To build on the other answers, this is indeed caused by a lens with undercorrected petzval. It is also vignetted, but more in one plane than the other. Here you can see a ray intercept plot for a large format tessar lens, these are also known as RIM plots or H'-TanU' Curves. On a RIM plot of the line is clipped, it indicates vignetting. You will notice that this example is vignetted in the tangential plane, but not the sagittal plane. This causes the spot and when defocused the bokeh to be larger in the sagittal plane than it is in the tangential plane.

This is the same condition that your example has, so I believe the particular lens has undercorrected petzval as well as "asymmetrical" vignetting, as I might call it.