I have a Nikkor 18-55 kit lens that has a 52mm filter, and recently I was looking that the 17-55 lens has a 77mm filter.
Is it because the latter is a better quality lens? I assume a bigger circle lets more light in.
Or because it has a greater maximum aperture (F/2.8 for the 17-55 vs F/3.5 for the kit lens)?
Or something else?
As Nick mentioned, there are two reasons why a lens might need a large filter diameter:
In the particular case of the 17-55, I think it's more of the latter than the former -- the aperture on a 55mm f/2.8 is 19.6mm; far smaller than the 77mm filter size of the lens. Even the old Nikkor 55mm f/1.2 AI had a 52mm filter ring.
For visual proof, here's the lens at 55mm, f/2.8:
As we can see, the apparent aperture is much smaller than the front element, even at f/2.8.
If we look at an angle at 55mm, f/2.8:
we see the edge of the image circle before the edge of the front element.
Considering 17mm,
once again, the aperture is much smaller than the front element.
However, this time, if we tilt the lens,
we can still see through the lens at an extreme enough angle that our aperture appears adjacent to the edge of the front element. I'm fairly certain that the wide angle, in combination with the lens's long physical length, is the reason this lens needs such a large filter size.
There are a combination of factors affecting the filter thread size. Aperture is one, certainly, and is the most likely factor in the two lenses you refer to. However the focal length can also affect the filter size; a wider angle lens would need a larger front element than, say, a 50mm lens with the same maximum aperture simply because of the wider field of view. Aperture becomes a factor again with telephoto lenses as the physical size of the aperture, which is (focal length/f number) increases in direct proportion to the focal length.
The filter thread size is generally mandated by the size of the front lens element. In the case of 50mm lenses, there are usually f/2.8 or f/2.5, f/1.8 or f/1.7, f/1.4, and f/1.2 aperture lenses. For a given focal length, to get a greater amount of maximum light, the lens body needs to be made wider to accommodate a larger maximum aperture, and the front lens element needs to be made larger to gather more light. It may not seem like much, but the difference in light between f/1.4 and f/1.2 is quite a bit, and the logistics of getting that much light down a 50mm lens without losing quality are rather complex.
In the case of the two lenses you listed, one has a constant aperture, while the other has a variable aperture. The difference between f/2.8 and f/3.5 is not that significant. The 18-55 has a maximum aperture of f/5.6 at 55mm, while the 17-55 has a maximum aperture of f/2.8 at 55mm. That IS a significant difference, as 55/5.6 = 9.8mm physical aperture while 55/2.8 = 19.6mm physical aperture, about double the size. That would be why the f/2.8 lens requires a larger front lens element, which thereby dictates a larger filter thread.
One thing affecting front element size, and thus filter size, is also sensor size. On so called "crop" sensors, sensor itself is narrower, which results in lenses specifically engineered for those lenses to be smaller.
For example on Canon camp, EF lenses are for full frame sensors and EF-S for crop sensors. EF lenses can be mounted on EF-S cameras and work perfectly, but because they support also the full frame sensors, their lens elements need to be larger, too.
