They vary by designed sensor size, flange distance, and the number/arrangement of the pins. But what else differentiates them?
EOS M cameras have sensors the same size as Canon's APS-C DSLR bodies that can use EF or EF-S lenses. EOS R cameras (so far) have sensors the same size as Canon's full frame DSLR bodies that use EF lenses.
As for the number of pins, not all EF lenses have the same number of pins, either. Those EF lenses compatible with EF1.4X and EF2X extenders (i.e. teleconverters) have three extra pins that other EF/EF-S lenses do not have.
No EF-M lens can be mounted on an RF or EF/EF-S mount camera and be able to focus to infinity. This is because the EF-M mount is has the shortest registration distance (a/k/a flange focal distance) of the three mounts. The EF-M mount also has a smaller throat diameter, which is the diameter of the hole in the camera's flange ring. Further, the shape of the lugs is different for each of the three mounts. So an adapter ring would be required to attach an EF-M lens to an RF or EF/EF-S camera. This would push the lens that much further from the camera's focal plane, shortening the maximum focus distance to even less than if the lens could attach directly to one of the other cameras.
On the other hand, EF/EF-S lenses, with a registration distance of 44mm, are easy to place the correct distance in front of either an RF camera, with a 20mm registration distance, or an EOS M camera, with an 18mm registration distance. All that is needed is a spacer with a thickness of either 24mm or 26mm, respectively. The EOS R and EOS M protocols are extensions and enhancements of the original EOS protocol, so it's easy to pass through signals from EOS M and EOS R cameras to EF and EF-S lenses. The cameras are programed to be fully compatible with all EF/EF-S lenses, as well.
RF lenses can't be adapted to EOS M cameras. This is because even though the RF mount is 2mm longer than the EF-M mount, the lugs on the back of an RF lens extend more than 2mm behind the flange. The throat diameter of the RF mount is also 7mm larger than the throat diameter of the EF-M mount. It's physically impossible for the lugs of an RF lens and the flange ring of an EOS M camera to occupy the same space at the same time.
Lack of a way to use RF lenses on EOS M cameras - and vice versa - doesn't seem to be considered much of an issue by Canon. The EOS M system is marketed to an entirely different sector of the camera buying public than the EOS R system.
EOS M is Canon's product line aimed at potential buyers who want a small, lightweight, and affordable camera with a small, lightweight, and affordable lens or two (all EF-M lenses in the EOS M system have the same outside diameter: roughly 61mm, give or take a fraction of a millimeter) that they can use as a dedicated camera for several years in situations when their smart phone camera isn't quite enough.
The EOS R/RF system is aimed at professionals and enthusiasts who want to build an entire system of camera bodies and multiples lenses. Though many pros and enthusiasts might find an EOS M camera attractive as a "pocket camera" for those times when they don't feel like lugging their more advanced, larger, and heavier cameras around they do not seem to be the primary market that Canon has created the EOS M system to serve.
