Modern digital cameras offer a wide range of output formats, giving the photographer all sorts of options for file sizes and quality of saved images. There are already some great questions dealing with various aspects of this problem once you start to break it down a bit (links included where I thought they'd be helpful), but I'll take a shot at a high-level roadmap here.
First, your output is going to either be some form of RAW file or a processed file (most likely JPG). Most cameras also give you the option of saving as a TIFF file, but I'll skip that for simplicity (personally, I don't think there's anything there you'd miss).
RAW formats give you the maximum information available from the sensor, delivered straight to you for post-processing using your preferred tools & techniques. In short, if you're doing your own post-processing (beyond stuff like tagging), there's a very good chance you want to be working with RAW files. Some Canon cameras offer multiple size options for RAW files, but I don't believe this is true for the D800 -- if you shoot RAW, you get the full file, no matter what.
If you output to JPG, you're getting an image that's been processed by your camera's onboard processor for all sorts of factors that aren't applied to RAW files, including, but not limited to:
- White balance
- Sharpness
- Noise reduction
- Color saturation
- Compression (Fine / Normal / Basic, plus "size priority" and "optimal quality" settings on the D800)
In addition to compression, the image sizes (L/M/S) also apply to JPGs, giving you all sorts of options for reducing the size of the output files. So, which of these settings is best? Unfortunately, it really depends on your needs and the nature of the subject you're shooting. As I mentioned, since the D800 doesn't give you a small-RAW option, the downsampling you'll see in JPG processing is based on algorithms in the Camera's EXSPEED processor.
The algorithms used to create in-camera JPGs are generally very, very good, but the one think you'll want to keep in mind is that processing done by the camera is a one-way trip. If there's anything about those settings that's not quite optimal for any given shot, your post processing is going to be working to repair the product of that processing rather than working directly from the information coming from the sensor. That's why I believe in most cases, if you're planning on post-processing images, you'll want to work from a RAW file.