Depth of field refers to the distance between the nearest and farthest objects that appear "acceptably in focus"; it does not convey information about absolute sharpness in the resulting image. All lenses suffer aberrations of various kinds, and these aberrations tend to be the strongest at the perimeter of the lens, where they are harder to correct for. Stopping a lens down reduces the contribution to the image from these peripheral parts of the lens and thus tends to increase sharpness in the plane of focus. At very small apertures diffraction comes into play and reduces sharpness again.
For a typical small-format lens, the sweet spot that minimizes information loss occurring from both aberration and diffraction is likely to lie close to f/8, hence the "universally agreed" rule of thumb you mention. To illustrate, here is a series of 100% crops of a book cover photographed from a distance of about 0.3 meters using a 50mm lens on an APS-C sensor (the detail is about 1cm x 1cm):
f/3.5 – largest aperture of this lens. The subject is in focus but not necessarily very sharp.

f/8 – this is better

f/22 – smallest aperture of this lens. Again, the subject is in focus but now appears blurry because of diffraction.
