I can't answer the part on optical quality. In my opinion it absolutely depends on the given combination. I wouldn't expect a general answer like "vintage TC is better" or "modern TC is better" to be valid.
The distances argument used in some other answers isn't convincing to me (see below).
Camera / Lens Communication
One aspect not yet mentioned is about the camera/lens communication:
As a decades-long owner of various Canon EOS equipment I know that this communication can be tricky. At least with analog bodies, there was a mechanical switch in the lens mount to be engaged when mounting an EF lens, which dumb-lens adapters did not activate, thus telling the SLR body not to expect a lens obeying the Canon protocol. When using dumb lenses, I was used to shooting in M or Av mode with aperture set to 1.0.
I once had my surprise when I tried to combine a dumb f/8 500mm with an EF teleconverter (for a lunar eclipse event) on a 620 body: I got an error from the camera. That was because the EF converter was designed to support EF lenses, therefore activating the lens-mount switch, making the camera trying to communicate with its lens, and of course failing. My "solution" back then was to not fully engage the teleconverter onto the body, but stop a few degrees before the locking position.
If something similar applies to your camera system as well, your only choice might be to have the adapter mounted on the body, and then use the vintage teleconverter and the lens.
So I'd recommend to test the combination with the modern converter before buying: it might fail to function.
Distances
Regarding the distances where the various components get located, both configurations are fine.
For my argumentation, we consider three values (giving example values to make it easier to follow):
- Flange focal distance ("FFD") of the vintage system (let's say it's 55mm)
- Flange focal distance of the modern system (let's say it's 45mm)
- Mechanical length of the TC (let's say it's 50mm, but actually, it's irrelevant)
We can safely assume that the vintage FFD is greater than the modern FFD, otherwise adapting would fail, by not allowing you to focus up to infinity.
The adapter just allows you to mount the vintage lens in a position according to its expected FFD, so its thickness will be equal to the FFD difference, being 10mm in our example.
Vintage TC Configuration
- vintage lens
- vintage TC
- adapter
- body
Regarding the projection distances, this configuration is fine.
A vintage TC will expect a projected picture 55mm behind its front flange, enlarge it, and project it to a plane 55mm behind its rear flange. This new projection plane is behind the original plane by just the adapter's thickness, 50mm.
The lens projects its picture onto a plane 55mm behind its flange, and that's where the TC expects it: fine.
The vintage TC produces its enlarged image 55mm behind its flange, and due to the adapter with its 10mm plus the body's FFD of 45mm, the sensor is just where the picture gets projected: fine.
Modern TC Configuration
- vintage lens
- adapter
- modern TC
- body
Regarding the projection distances, this configuration is fine as well.
The modern TC will expect a projected picture 45mm behind its front flange, enlarge it, and project it to a plane 45mm behind its rear flange.
The lens projects its picture onto a plane 55mm behind its flange, and due to the 10mm adapter, that is 45mm behind the front flange of the TC: fine.
The modern TC produces its enlarged image 45mm behind its flange, and that's just the FFD of the body, so exactly where the sensor is located: fine.
