Like most mirrorless systems, the RF mount has relatively short flange focal distance, of only 20mm, as opposed to 44m of EF (and EF-S). This has a number of optical advantages, better allows for adapted lenses, and is just plain smaller.
Additionally, the new mount has a slightly wider throat diameter, about which Canon says:
With a large 54mm lens mount, large diameter elements to match and placement closer to the image plane, RF lenses provide brilliant image quality edge-to-edge with minimal light fallout, enabling new possibilities for enhanced optics and compact designs.
And, also, while they were at it, it seems they took the time to redesign the electronic communication mechanism (which seems like a good idea, because once you've got the basics of that in place, you have to keep compatibility basically forever or else people will complain on the internet that you made the change just to force people to buy new lenses rather than for any actual good reason).
It may be worth considering that when Pentax dipped their toes in the mirrorless waters with the K-01, they decided to use the same classic K mount (well, K-AF3, for those keeping careful track). The camera has some questionable design choices, but that's not really why it flopped. It flopped because without a lens lineup designed to take advantage of the format, there wasn't much reason for it.