Ok, the only possible explanation is that similar to Gauss' law: the sub-section, sphere surface area cutout, smaller field of view actually produces less light than the large field of view did.
The field of view is proportional to the focal length AND the working aperture, since theres a cone from the center of working aperture, out the barrel, to the fringes of the field of view, and scaling horizontally also scales the cone vertically.
So -- even though there is more light gathered, the actual scene being focused is producing less light, and they happen to balance out because focal length is correlated identically with working aperture and the field of view.
In summary: only the F-number depicts the amount of light gathered per unit area of the scene, regardless of zoom/distance and framing