Currently I trust the catalog file to hold all of develop settings for my files. I don't use xmp/sidecar files.
If you use them, why?
I'm searching for a reason why I should use them.
I always use sidecar files. I find it easier to backup them along with the photo data than to backup a separate catalog.
Also, I can open the files on separate computers without having to synchronise the catalog.
(I should mention that I actually use Photoshop with Camera RAW, not Lightroom, but the principle is the same.)
Note that there are two ways of using xmp sidecar files:
1) activate automatic writing of xmp files in lightrooms settings.
This way Lightroom will write and refresh the xmp files for your images as a side task if it got no other important things to do. Meaning: you'll never know if the sidecar files are really up to date or not (but usually they are).
2) order Lightroom to save the xmp files for the currently selected images (alt/option+S if I'm not mistaken).
Lightroom will start a real user task for this, showing you the progress while writing the xmp files.
The develop settings will be stored in the xmp files, so it makes an easy file wise backup - but it would be way better to use Lightrooms catalogue backup functionality instead.
You can use xmp files to move image development settings from one computer to another (again you might use a catalogue export/import instead, but in this case I find xmp files easier to use).
And there might be some 3rd party apps which can read xmp files and do some interesting stuff with them (though I know none).
My advise would be: don't use the automated "write xmp sidecar file" setting. If you want to move images with dev settings to another comp, explizitely save the xmp files using the "save" command instead, then copy the raw and xmp files and import on the other computer.
I use it exactly because I don't trust the big catalog file.
What if some bug in a new version somehow causes your DB to go corrupt (remember, no app is bug free), and your recent backup was 200 photo's back, you'll have to redo the work for those photo's. I like my develop settings close to the file, allows easy backup of photo + meta data + develop settings and better compatibility with other programs.
To now, I've used Bridge and Camera Raw, with changes being saved to the sidecar XMP files. This is fine by me as the individual files are easy to back up and synchronize across computers. I've been trialling Lightroom, and plan to buy it soon. The process has been helped by the fact that it works with my existing sidecar files, so those files are thus more portable than the LR catalog.
When I started using Lightroom I had it write the XMP files automatically. But I found that having the drive that holds my raw images filled with thousands and thousands of little XMP files rather bothersome when it comes to synchronizing that drive with its local and off-site backups.
So now I just back up the Lightroom catalog. It's a big file but it compresses really well with 7-zip. I make regular backups and new backups do not overwrite the old backups. So if I discover six months from now that I messed up the metadata of last year's photos, I'll be able to retrieve that from an older backup of my catalog. If I relied on XMP files, the messed up metadata would have been written to the XMP files too.
To answer your question: XMP files are useful for transferring metadata from Lightroom to other raw processing software.