According to a FOV calculator, 17mm gives a 76.2 degree diagonal FOV for Canon APS-C sensors. That's an angle of 38.1 degrees either side of straight ahead.
Sin(38.1) = 0.617+ so for each mm forward, a light ray at the edge of the FOV goes sideways by a bit over 0.617mm.
You probably need to add a few mm for a step up ring to attach the bigger filters to the camera, so lets say 3mm for that, giving you a range of 13 to 16mm.
13mm gives 8.02mm extra needed on each side = 93.04mm, and 16mm gives 9.87mm extra needed on each side = 96.75mm
Since standard sizes are 82, 86, 95, and 112mm, you'd probably want to go for 95mm (which might give you slight vignetting on the corners with thicker filters - it depends whether the original 77mm is actually needed, or whether that's just the closest standard size over the actual requirement), or 112mm.
[Edit - just realised I got the trig wrong - sideways distance should be tan (= opposite / adjacent) rather than sin (= opposite / hypotenuse).
tan(38.1) = 0.7841 so for each mm forward, a light ray at the edge of the FOV goes sideways by a bit over 0.7841 mm.
This gives 10.193 mm per side for 13mm thickness , = 97.387mm, and 12.546 mm per side for 16mm thickness = 102.09mm.
However, the original 77mm is presumably chosen to work with a single filter
so if you reduce the effective thickness to compensate for that, you're somewhat better off - if you knock 3mm off (which is probably conservative) you're back to 92.682 mm for 10mm thickess so 95mm is probably OK ]
