A factory refurbished model can be one of two things. It can be a used that was damaged and has been repaired by the factory to spec. Or it can be a merchant-returned copy that cannot be sold as new for some reason (a sales return, an open/damaged box, a sales demonstration model, etc.) but is to all intents and purposes new and unused. So, when you purchase refurbished, the risk is that you get the former and not the latter.
Some people might actually prefer to get a refurbished camera, because if they do snag a mostly-unused model, it will have actually gone through a factory test and adjustment, which most of the new cameras coming off the line do not. And such purchasers know that any potential issues have been caught. And in the USA, you can purchase a refurbished model directly from Canon on their website. Such models come with a warranty (and everything in the box) identical to those of new cameras.
Cameras refurbished by third parties, however, may not be put through as rigorous a testing or repair process or have had access to the same parts. So you have to know the source of the refurbishment.
Ultimately, whether or not it's worth it is up to you, your budget, the warranty that comes with the "white box" unit, and your own personal comfort level with entering a copy lottery.
Personal anecdotal data: I purchased both a 50D and 5DMkII refurbished from Canon via their "loyalty program". Both cameras are still in (very light as I mostly shoot MFT these days) use by me, about eight years later. Both had shutter actuation counts below 20 when they arrived.