The difference is that almost everything you need to know as a photographer depends on the effective aperture (also known as the entrance pupil diameter) whereas the actual size of the diaphragm is only relevant in the design of the iris / lens barrel.
Diffraction is the ultimate limit to how much detail can be extracted from a photograph, and there is no free lunch here. If it depended on the physical aperture only then clever lens designs could bypass the diffraction limit and we wouldn't need electron scanning microscopes etc.
The reason the entrance pupil size is different to the physical size of the opening is the presence of lens elements in front and behind that change the magnification. Trying to cheat diffraction by using a physically larger opening, would require reducing elements in front of the aperture to make the image of the opening smaller, but also enlarging elements behind the aperture.
The smaller spread of light due to the larger physical opening would then be magnified into a larger spread equating to a blur of the same radius regardless of the physical aperture size.