The AF-ON button can be used as the sole means of activating focus on cameras with configurable buttons. There are many occasions where activating AF at the time the shutter button is activated is NOT desired, and in fact can result in less than optimal camera performance. Separating AF and shutter is of particular importance and usefulness when using AI-Servo or Continuous focus mode, where you are focusing constantly on moving subjects.
Personally, I use what is often termed "rear-button focus". I use the Canon EOS 7D, which allows me to customize the functions of a lot of my camera buttons. I reconfigured the shutter button to only activate metering and the shutter, and configured the * button to activate AF, and configured the AF-ON button to stand in for the original function of the * button (lock auto exposure). I use this particular button config simply because the * button is far more convenient for activation by my thumb than the slightly inset and farther away AF-ON button (as I use AE Lock much less frequently than AF ON.)
I am now free to focus to my hearts content without actually taking any photos, which can be useful when you just need to observe the behavior of a subject (in my case, usually birds and wildlife). Even though space is cheap, it is wise to take a shot only when your likelihood of capturing the right kind of behavior, the right kind of action, is high. A shutter-happy photographer can easily burn through 64 gigs of CF cards in a 6-8 hour period of time (trust me, I know first hand, as I was rather trigger happy when I first started doing bird photography.) These days, I usually get away with one, maybe two 16Gb CF cards over a similar time period.
There are other benefits to separating shutter and AF functions. The shutter button also activates metering and image stabilization if half-pressed. You have far more fine grained control over the behavior of your camera when you separate out the critical AF function from the most used button on the camera. You can activate IS/VR if you need it, or not, as necessary. You can activate metering with a shutter half-press, lock it in, then recompose, AF, and take the shot. This is almost an essential behavior with the kind of photography I do, where I usually meter off the sky first, lock in that exposure, then manually compensate according to the subject after re-framing.
Similarly, separating AF out to its own button makes "focus & recompose" a synch, where as with shutter+AF combined, you usually have to hold the shutter button half down to maintain your focus during recomposition (and then, only in non-servo modes.) Another side effect of using a dedicated AF button separate from the shutter button is that you can leave your camera in AI Servo all the time. If you need "single shot" behavior, simply hold the AF button until your subject is focused, release it, and take your shot or shots. When you need servo functionality, just hold the AF button down until you no longer need focus, and take as many shots throughout that period of time as you need to, even with continuous shooting modes.
Separating AF and shutter can also help reduce missfocus. It is often the case that you focus on a subject, capture some frames when it does something interesting, wait for the subject to do something else interesting only to have the camera suddenly start hunting for focus the moment you start shooting. Separating AF out to its own dedicated button allows you to lock focus, then keep the camera focused on that point until YOU, rather than the camera, decides to change it. This greatly reduces the chance of your camera behaving poorly and refocusing a scene that was already focused. If, for whatever reason, your subject moves, you can instantly refocus just by pressing your dedicated AF-ON button.
Fundamentally, AF-ON, in cameras that have configurable buttons, gives you a dedicated AF button that can be configured as the sole means, rather than an alternate means, of activating and deactivating AF. This, in turn, gives you more explicit control over your camera'e behavior.