f/N is not the diameter of the physical aperture, it's the diameter of the entrance pupil, i.e. the image of the aperture as seen from the front of the lens. And yes, it changes as you zoom: look at the front of your lens while you zoom and you will see.
Edit 1: A related question is “How come a 5 mm aperture is ‘faster’
than a 35 mm aperture?” I don't know whether this was somehow implied
by the original question, but anyway, here it goes:
At the tele end, the lens only collects those photons that come from a
very narrow field of view, while at the wide end it collects light from
a much wider FoV. Then, although the tele setting gets 48 times more
light from any given element in the scene, the wide setting collects
light from a solid angle 123
times larger. It thus ends up having 2.5 times more light gathering
Edit 2: Here is where the numbers come from: I compared the two ends
of a 18-200 mm f/3.5-5.6.
The amount of light collected from a given small element of the scene
(small enough to always fit within the FoV) scales like the surface area
of the entrance pupil, i.e. like D², where D is the diameter. If we
compare the 200/5.6 setting to the 18/3.5 setting, the ratio is
(35.7 mm / 5.14 mm)² = 48 (the tele end collects more light)
The solid angle FoV scales like 1/f², then, for this same zoom, the
(200 mm / 18 mm)² = 123 (the wide end has more FoV)
The amount of light collected from an extended scene (as oposed to a
small scene element) of known average
scales like (D / f)² = 1/N², where N is the aperture number.
For this zoom you get
(5.6 / 3.5)² = 2.5 (the wide end collects more light)