I would like to fire three off-camera flashes (SB-800, SB-600, SB-600) in manual mode with the on-camera flash in commander mode (--) on a Nikon D600. It's a portrait with two lights in front at different outputs levels and one behind the subject over exposing a white background by one stop. It requires the three flashes to have different output levels. Can I achieve that with only two groups (A and B)? I could put a cover directly over one of the flashes to lower its output I guess, but can I dial it in somehow on the camera commander screen or the flashes?
one behind the subject over exposing a white background by one stop
It sounds like you don't necessarily have to adjust this one as frequently as the other two. If you set this background flash to manual dumb optical slave mode (that is: let it fire in the manually adjusted settings when it sees another flash) you can distribute your other two flashes over the now free two separate groups/channels.
You can figure out the setting for the background flash beforehand. Make sure that you have a test subject in front of the background to control the spill from the background. If you have a relatively large background-to-subject distance, this should not matter too much.
Also consider the overall light level to ensure reasonable recycle times. A full power flash will certainly blow out the white background, but will take very long to recycle. And you don't have to blow out the entire background either. If you get "pure white" around your subject, that's enough. Fixing a not entirely blown out corner of the image is such a simple post processing task that you could even do it in paint. Being able to work quick and do successive images is important when it comes to portraiture and thus recycle times should be short. There's often this face/expression after an image that's actually worth capturing, possibly even more so than the original staged expression. Being able to click again (and flash again) quickly can produce some winners.
Get more groups, or at least different triggers.
Just me, but consider getting some additional gear. Picking up a cheap 3rd party CLS-capable flash, such as a Yongnuo YN-586EX (make sure you get the Nikon version), would let you use your SB-800 as your commander, and give you four groups (three off-camera). Or using TTL radio triggers (Phottix Odin, Godox X1, or Yongnuo YN-622, etc.) that allow for three off-camera groups.
You could also use cheap manual radio triggers, have your flashes in M mode, and simply give up remote control from the camera, and get used to walking up to your flashes and dialing in the power on the back of the speedlights. That is the Strobist way, after all. :)
Portrait photographers use multiple lights. Lighting effects are achieved via adjusting position and intensity of each. We are talking lighting ratio. Typically a main lamp is set high and off to the side to simulate afternoon sunlight. Shadows cast by the main are softened by a subordinate lamp called a fill. Best if this lamp plays on the subject, filling shadows from the camera’s viewpoint. To accomplish, best to set the fill near an imaginary line drawn camera to subject and at lens height.
Portrait photographs deal in lighting ratios: 2:1 is flat lighting. 3:1 delivers normal contrast and is often called “bread & butter” lighting because it delivers a pleasing contrast that sells. 5:1 is contrasty, often termed “masculine”. 9:1 is super contrasty, often termed “theatrical”.
2:1 is achieved by setting the main and fill equal as to intensity at the subject. Say the main delivers 100 units of light on the frontal areas of the face. Say the fill delivers 100 from the straight on. The forehead and cheeks receive both -- thus 100 units from main + 100 units from fill = 200 units on frontal area. The shadows do not receive light from the main; they receive only 100 units from the fill. The ratio is 200:100 which mathematically reduces to 2:1 (Flat Lighting).
3:1 set the fill subordinate to 1/2 power (compared to the main). The frontal areas receive 100 units from the main and 50 units from the fill. Thus the frontal areas receive 150 units, the shadows 50 units. The ratio is 150:50 = 3:1. This is the “bread & butter” ratio”.
5:1 set the fill subordinate to 1/4 power, the ratio is 100 units main + 25 units fill, shadows get 25 units. Ratio is 125:25 = 5:1 (“Somewhat contrasty”).
9:1 set the fill subordinate to 1/8 power, the ratio is 100 units main + 12 1/2 fill = 112 1/2 frontal and 12 1/2 shadows = 9:1 (“theatrical”).
If all your flash units are equal in power, you can use distance as the controlling factor.
2:1 main and fill equal distant from subject.
3:1 measure distance main to subject, multiply by 1.4, this calculates fill distance from subject.
5:1 fill is set back 2 X main distances.
9:2 fill is set back 2.8 X main distances.
Distance is based on the “law of the inverse square” as flash units approximately follow this law.
To place the background lamp 1 stop brighter than main, multiply main distant by 0.7
To place the background lamp 2 stops brighter than the main, multiply main distance by 0.35
This method was taught at the Professional Photographers of America School for continuing education.