I get that it has to do with the lens, but what exactly is causing this?
Several focal-length-related factors can influence sharpness:
It is usually harder to manufacture long focal lens, so at constant cost, expect a 50 mm to be sharp, a 200 mm to be not-so-sharp, and >300 mm to be really low quality.
On a related note, it's hard to make a lens with both a long focal lens and a wide aperture. So, long focal lens often have a narrow aperture, hence more diffraction.
Long focal lens implies shallow depth of field, so any focus imperfection (front or back focus) is much more noticeable.
Long focal lens implies more motion blur except if you use a tripod or have a very fast shutter speed.
Long focal length are often used to shoot distant subjects, in which case the atmospheric effects can be important. Depending on the distance and weather, the atmospheric effects can be the limiting factor, and then whatever the lens, you'll always get bad pictures (thanks @mattdm for the hint, I forgot this one).
It's probably because the lens you're thinking about is a low-cost telephoto or supertelephoto 70-300 or 18-300 type lens.
A longer telephoto focal length is not inherently soft. My EF 400mm f/5.6L USM prime lens is very very sharp and fast to autofocus and is my go-to birding lens. But it also cost more than US$1000. Canon's so-called 'great white' telephoto and supertelephoto lenses all have great reputations, but they do have pricetags to match.
But with a budget entry-level telephoto or supertelephoto zoom, the reasons the lens is generally perceived as soft are manifold:
The design of a zoom lens is typically compromised on image quality somewhere in the zoom range to cover a variety of focal lengths. Most typically, these weaknesses will be at the ends of the zoom range (usually distortion and sharpness). The bigger the zoom range, the more compromise you tend to run into. A smaller zoom range (or a prime) and more corrective elements can improve performance, but this comes at a cost both in versatility and price tag.
The variance in image quality also happens over the aperture range of the lens as well. Very few lenses perform well wide open. But because of the 1/focal_length rule, telephoto lenses are often used by newbs wide open at the extreme telephoto end of the range. That's compounding two weaknesses together.
1/focal_length rules of thumb also mean that shutter speeds and iso settings are higher than most newbs are aware of. It's not uncommon to use apertures in the f/8-f/16 range, with iso settings in the 400-1600 range in sunny-16 conditions, just to get the shutter speed up fast enough to eliminate blur from handholding. Using a monopod or tripod can also help with this. But newbs tend to default to handholding.
Autofocus accuracy is also more of an issue with a telephoto or supertelephoto lens, because the depth of field gets shallower, even at smaller apertures. Lower-cost zoom lenses tend not to have features to increase AF accuracy/speed, such as ultrasonic motors or focus limit switches.
The nature of telephoto subjects. The most common subjects that someone buys a telephoto zoom for are sports and wildlife. Both of which move quickly and require good technique and anticipation/skillz of the shooter to get a "clean" shot. These are demanding disciplines that take a lot of practice to master. And in the case of wildlife, often your lens just isn't long enough to frame the way you want, so you end up cropping, which reduces your resolution and thereby your sharpness.
In essence, it's a combination of lack of technique and the performance of a low-cost telephoto zoom that's creating issues of perceived softness at higher focal lengths. But with good technique or a high-end supertelephoto lens, it may not be an issue at all.
"Some images" will look softer due to chromatic aberration which splits white light into it's spectral components; the thicker glass elements of a telephoto heighten this effect in high-contrast, backlit images. You will always see this this, for example, in tree branches photographed with a bright sky, or snow patches on a dark mountain. Left edges will be highlighted in cyan and right edges highlighted in pink. This can be corrected in post processing or with in camera lens profiles, but it does compromise these kinds of edges.
"Some images", particularly landscapes taken at the magic hour, are prone to the shimmering effect of high temperature gradients in the atmosphere. What should be a static subject now moves very quickly, though by small amounts, but enough to smear everything except sky. Unless these pictures are shot at a high ISO, they will be blurry. Even tripods help little in these cases. So, even though edges may be sharp if shot correctly, the overall image suffers due to high ISO.
Shooting high-quality images with large telephotos is hard!