What is the physics that explains the phenomenon observed when out of focus areas increase in apparent brightness?
The answer to this is simple: the non-bright areas are also diffused into large circles, but since photography is capturing light, that's what you see. A large circle of no-additional-brightness doesn't change the result.
Your question is a little unclear as to what you are referencing. If you are referring to bokeh then @mattdm's answer suffices. But there are other possibilities that your question, as written, may be considering.
All photographic systems have a maximum brightness they are capable of reproducing.
With film it is the point at which enough light strikes a part of the negative to make it completely opaque. When making a print from such a negative the brightness of the photo-sensitive paper is the limit of the brightest possible value.
With digital, maximum saturation is reached when a pixel well reaches full well capacity. Any additional light that falls on that pixel well will not increase the brightness recorded by that pixel well. The maximum numerical value for that pixel has already been reached. With older digital sensor technology often the excess electrical charge from additional photons striking a full pixel well would bleed over to adjoining pixel wells. With newer designs and sensor processing algorithms, though, blooming is much better controlled and it takes considerably more excess charge to induce it.
Under such a situation, if the light that is spread out is still more intense than needed to fully saturate each of the higher number of pixel wells it is striking then more pixel wells are recording the same amount of light per pixel well.
Let's say that in one photo the light from a strong source is concentrated on an area of 50 pixels. The maximum amount of light recorded will be the full saturation value of those 50 pixel wells. On our next shot we defocus that light by changing the point of focus or by opening up the aperture if the light source is already out of focus. Now imagine that same light is spread out over 100 pixels and the field density is still intense enough to fill each pixel well to capacity. 100 pixels at full well capacity have recorded twice as much total light as 50 pixel wells could.
The concept is similar with analog film. The wider an area over which a bright light source is spread, the more grains of light sensitive materials in the emulsion can absorb that light.
In either case, the brightest points of the image are the same brightness. But the larger area of the photo at that maximum brightness makes it look, to our eyes and our brain that interprets the light our eyes capture, like the image is brighter.