I know how aperture works and why is it not restricting field of view, but I somehow cannot get around the following:
Aperture doesn't restrict field of view for the camera, but when I take the lens off camera and I look through it with my own eyes, it definitely restricts the area that I can see through the lens.
So why is that?
This is because of differences in how your eyes and your camera work.
In your camera, the rear element of the lens is the final optical element before the image is formed. The aperture iris is placed at or near the point of "maximum out-of-focusness" (the optical center of the lens) so that the shadow of the iris is spread evenly over the entire image.
By looking through the lens with your eyes, you place an additional active element in the system: the lens of your eye. This changes things dramatically, and the aperture iris is now nowhere near the optimal point, thus the restriction in field of view. You can see a similar effect sometimes in very old photographs, where aperture control was done using a device in front of the lens rather than integrated into it, causing the corners of the image to be darkened.
It appears you are using the term "aperture" specifically to refer to the physical opening in the lens, and as you say it does not limit the field of view when mounted on your camera. But the "aperture" of an optical system is the opening that limits the amount of light collected. When you take the lens off of the camera and look through it, the optical system is now lens + eye and the physical opening in the lens is not the aperture for that system.
