I've got an EF 24-105mm f/4 L IS that is 'well broken in' with a very loose zoom ring. When the camera is pointed straight down, as it does when hanging from a sling or neck strap, the front of the lens creeps all of the way out to the fully extended 105mm position.
I use a plastic wristband bracelet stretched around the lens barrel with part of the width of the bracelet on the zoom ring and part of the width on the stationary part of the lens barrel. The band provides enough resistance to prevent the zoom from creeping due to gravity. How much of the band is on the zoom ring and how much is on the barrel determines just how much resistance is added.
The same type of band can be used to give a focus ring more resistance. There's even a company that markets such bands specifically for camera lenses, but I've never tried them because they seem to be the same as the rubber wristband bracelets I already have that work for me.
The TS-E 24mm f/3.5 L is slightly smaller in diameter (approximately 78mm) at the focus ring than the barrel of my EF 24-105mm f/4 L IS (approximately 83.5mm). The TS-E 24mm f/3.5 L II is slightly larger in diameter (approximately 88.5mm) than my EF 24-105mm f/4L IS. I would expect you could fairly easily find a generic plastic/rubber wristband of the type used to promote charities and businesses that would fit either one.
From a comment by the OP to another answer:
I bought this lens a few months ago (brand new). I think that's the way it is, but I find it weird it is so loose by design.
The focus ring of all Canon TS-E lenses I've handled are fairly well dampened without being stiff. If the lens is under warranty, I would not hesitate to contact Canon Service. If you send it to them and they say there's nothing wrong with it, at worst you're out the price of shipping.