There is a bit (pun intended) of truth to it. When some but not all images on a card are deleted the 'holes' between the images that remain are not all a uniform size because image file sizes are dependent upon the contents of the image. This is true of both raw and jpeg files. Depending on how well wear leveling is implemented in the card's memory controller, attempting to write a larger file over these spaces will fragment it, thus slowing down the write speed and very slightly increasing the chance of file corruption.
Formatting the card regularly using the camera can actually be beneficial.
It insures that the file system on the card is in the form needed by the camera. When a card is formatted in another device, it might lead to issues if the camera can't find files it expects to find on the card. Different cameras have different versions of storing image files to the memory card. Most, if not all, are DCIM compliant, but there is enough variation between them that using a card formatted in another camera or on your computer may lead to corrupted images or file directories.
It is possible to create a 'disk image' on another device that replicates the file system created by a camera when a card is formatted in the camera. Formatting in the other device and then copying the disk image should work just as well as in-camera formatting. For more about that, please see this answer to Canon 6D resetting folders on card format - a way to stop this? But the easiest way is to just do it in your camera.
You're also much more likely to extend the life of the card, as compared to never formatting and only erasing images in the card, by formatting regularly.
The way flash memory cards (and USB flash drives) work is that the memory controller on the card assigns different areas of the card for specific directory locations each time the card is formatted. They do this for what is known as load balancing/wear leveling.
Flash memory has a limited number of write cycles it can tolerate before it fails. The number of write cycles each bit on a flash memory card can handle before it fails is very large but it will eventually wear out. So the controller tries to ensure that each storage location of the entire card is written to roughly the same number of times over the life of the card. What this means is that even after you format the card the controller will continue using parts of the card that have not yet been written to until each storage location on the entire card has been written to before it will go back and begin using the locations that have already been used.
That's good for recovery because it means not much is overwritten until the entire card has been written to once. But then it's going to go back and use the very beginning again if that space is showing as empty. When you delete a file normally the space that file used is marked as empty but the state of each bit in that space is not altered. When you format a card pretty much all of the regular storage space on the card is marked as empty and available for use.
Formatting regularly helps the memory controller on the card do a better job of wear leveling. It also allows for better card performance by increasing the likelihood of sequential write operations. If you leave the same files on half the card and repeatedly write files to the other half, erase them, and rewrite other files then all of the wear is going to go to only half the card and the life expectancy of the card before half of it fails will be shorter!
Additionally, if a memory card's controller has detected bad blocks on the card it will remove those blocks from the listing of available blocks for the card. It does this each time the card is formatted.
Most memory cards have a bit more memory than their listed capacity. When a bad block is identified, the memory controller will no longer use the bad block and replace it with some of the 'spare' memory on the card. One of the common differences between top name brand memory cards and generic/no name memory cards is the number of bad blocks mapped out of the card's total memory before it leaves the factory. The greater the amount of memory that must be mapped out, the less reserve memory is available on the card for the controller to use as other blocks fail later on. All of the major brands (Lexar, SanDisk, Transcend, Kingston, etc.) get their components from the same handful of suppliers that actually manufacture the memory chips and controller chips. So do the off-brand names, but they usually buy the leftover components that may or may not have been good enough to pass the QC of the major brands' buyers. One of the thing the buyers look at is how much reserve memory remains on the chips after the bad blocks have been mapped out at the factory.