It's easiest to explain what custom white balance does. Custom white balance asks for a particular image that is primarily white or neutral grey. It takes the sample in the image and calculates from it adjustment factors to apply to the response of the red, green and blue sensors such that they are balanced with each other so that equal amounts of each produces a neutral grey. It is a fairly simple calculation and produces high quality results, but the accuracy is dependent on the sample image being neutral grey or non-overexposed white.
Auto white balance on the other hand is when the camera is left to try and guess. It has the advantage of not having to specifically setup a grey card or find something neutral grey or white to take a photo of, however it gives up accuracy since it is only making a guess. The algorithms have certainly improved a lot over the years, but it still can't do things perfectly. Techniques can very from manufacturer to manufacturer and can include such things as looking as the brightest area of the image that isn't overexposed or looking for what appears to be flesh tone. If it can find something it thinks should be a certain color, it is then possible to calculate the correction factors for red, green and blue based on what was sampled vs what was expected.
It is also possible for the camera to look at known profiles for particular types of lighting and see if it matches up with one of these. If it does, then it can have a pretty good idea that it is close to correct. This is why AWB tends to work better in normally lit rooms as opposed to areas where the lighting is distinctly colored (since the custom lighting won't match an established preset.) AWB also tends to suffer in either very low light or very over-exposed images where the contrast is either too low or the image is too bright to be able to effectively sample a range of color and contrast.
As an extreme example, if you took a photo of a blue and yellow card, the camera would have know way to tell if it was bluish light or yellowish light. The card could be white and yellow, blue and yellow or blue and white. Without something else to reference, AWB would fail miserably with no way to guess. On the other hand, custom white balance would have no problem since it was preset from a reference image.