So, I understand that a tilt-shift lens solves certain problems with shooting buildings and getting straight lines and such, but what I've not understood is why. I'm looking for the technical reasons (maybe diagrams?) as to why tilting and/or shifting has this effect or other desirable (or undesirable) effects.
You don't use the tilt element of tilt-shift lenses to fix perspective, only the shift element.
Tilting tilts (or swings - that's the term for a horizontal tilt) the plane of focus. It's mostly used for increasing apparent depth of field. Imagine you're taking a picture of a football field. You want the entire field to be in focus but you're using a large format camera and you can't stop down too far. Thanks to movements (tilt/shift) you can tilt down your plane of focus, so instead of vertical, it's tilted forward covering the field. You lose focus on the sky, but that doesn't really need to be in focus anyway, right? What's funny is that although you tilt part of the lens, the scene you're framing never actually changes.
All this is covered in this fine wikipedia article on the Scheimpflug principle.
Shifting is pretty simple. Inside your camera, your lens works a bit like a projection lens works on a movie screen. This is why longer focal lengths give you more "zoom" - the same thing happens when you move your projector away from the screen. Your lens projects and image onto your sensor or film - just like you have it projected on a screen in a movie theater. With 99% of lenses, the position of the lens is fixed and the image shown on your sensor is just big enough to cover it... or sometimes, not, that's when vignetting happens. With shift lenses, your circle is much bigger, so when you shift, you really just move the lens around in front of your sensor, as if you were moving a projector.
Shooting large format really helps you understand that a camera isn't a fixed object, it's really two planes that can interact with each other in a huge number of ways.
This is my intuitive understanding of the shift lens, based on the Wikipedia article, and especially the first diagram from that article:
Let's say we want to shoot a tall building, where we stand at the base level. Our field of view doesn't allow the whole building to be captured in the frame. In fact, when the camera is level, half of the frame is "wasted" on image of the ground.
It is obvious why, if we tilt the camera so we do capture the whole building, we get the converging vertical lines due to the perspective effect.
If we had a bigger film (sensor), then the FoV was bigger and we could capture the whole building. So, shifting the lens up relative to the image center effectively "extends" the frame in the opposite direction, so now we have a bigger "half frame" and we capture the whole building while at the same time we reduce the other "half frame" and eliminate the ground!
As for the tilt part - consider that in optical geometry is more or less symmetrical around the lens center. Now the plane of sharp focus is parallel to the lens. So, when you shoot, say, the ground, which is not parallel to the lens, you see the effect of DoF. Now, if you tilt the lens, then the plane of sharp focus is tilted as well. This way, you get on the film a much deeper DoF from the non-parallel plane that you are shooting.
If your film plane is parallel to the subject then no lines on the subject are distorted. Conversely if you angle the film plane so its no longer parallel to the subject, lines converge towards the further away edge of the film plane and diverge from the closer edge of the film plane.
Using a tilt shift lens you can keep your film plane parallel to the subject and just move the lens about to change what is in the field of view.
Let me first explain the tilt portion. An image is projected straight from the lens to the camera. If the lens is tilted relative to the sensor, then different parts of the image will be in focus at different lengths. The location in focus depends on 3 things. The first is the distance of the lens to the sensor. The second is the distance from the lens plane to the object of interest. The third is the focal length. What's happening in the tilt portion is that there is different lengths from the sensor to the lens plane. Thus since the focal length remains constant, the location of the in focus part must change.
The shift part is a bit more difficult, but let me give it a shot. What happens is you are moving the lens up and down. This causes a different area to be in focus, but the lines still remain as you have it originally.
The other answers are great but I wanted to add one more thing.
You can do the same correction in Photoshop or with various other tools (some panoramic tools can be used this way, such as Autopano Pro).
So why bother with tilt-shift lenses anymore? Because when you correct this in software, the results may leave some areas far less sharp than others - because when you correct distortion you are literally "stretching" some pixels greatly, and those areas will be much less sharp (while in other areas you are compressing pixels and throwing out detail).
If you look around you can find third-parry tilt/shift lenses for most mounts, if you really want to explore using one. Or you might be able to rent one for your camera.