Since the film days we have used 0.03 CoC for determining depth of field and similar calculations. In digital terms, does this best match with 6 or 12 or 24 or 36Mpixel full frame sensors? If this matches with 12Mpix (a guess), then could you suggest a useful value of CoC for 36Mpix please?
The camera lens changes the direction of travel of incoming light rays. After their transient through the lens, the rays trace out a cone shaped path. Hard sharp focus is achieved when the distance, lens to film or digital sensor is such that the apex of this cone just kisses the prepared surface. In perfect world, a cross-section of the point of contact would be a point with no discernable dimension. In actuality, due to residual uncorrected aberrations, we get a diffused circle with scalped margins. We are taking about the circle of confusion. This is the smallest fraction of an image that that conveys intelligence. The entire image area consist of countless such circles, some tiny some huge. Size matters. For us to perceive a image as tack sharp, the circles must be so small that they are observed to be points not circles. The permissible size to have this happen is based on viewing distance, image contrast, lighting level, and the acuity of the observer’s eye.
The basis used to determine the permissible size of the circle of confusion is the resolving power of the human eye. Under bright light, an observer with 20/20 vision will perceive a coin at a distance of 3000 diameters, to be a point and not a disk. Stated another way, a 1 meter wagon wheel viewed from 3 kilometers (1 yard viewed from 9,000 feet or 1.7 miles. Such is too stringent for photographic work. This is because of the contrast of our media and the ambient light that we typically work in. More realistic for photo purposes are a disk viewed from 1000 times its diameter. This works out to 3.4 minutes of arc. Some examples: 1/100 of an inch viewed from 10 inches corresponds to 0.254mm viewed from 245mm. Another way to state this is 1/50 of an inch (0.5mm) viewed from 20 inches (500mm).
When we talk about the size of the circle of confusion at the focal plane of a camera, the size of the camera’s format must be taken into account. A 35mm film camera yields a tiny image that must be enlarged about 8X to yield an 8x10 inch image. Therefor the size of the circle of confusion must be made tiny to allow this 8X enlargement. So we end up with a hodgepodge of circle sizes that may or may not fit the viewing requirement. The industry has generally settled on using a circle size of 1/1000 of the focal length. Such a scheme roughly takes into account that the image will be enlarged to make a satisfactory display. For critical work Kodak uses 1/1750 of the focal length and Leica 1/1500.
Using the 1/1000 rule of thumb with a 50mm lens, the permissible circle size is 0.050mm. Such a circle size will permit 10X enlargement 0.05 X 10 = 0.5mm viewed from 20 inches.
That’s how this stuff goes, lots of gobbledygook ?
Reframing your question: How large can the circle of confusion be for pixel peeping?
This basically amounts to calculating the size of a pixel and multiplying by n, where n is chosen based on what you consider acceptably sharp?
If the pixels are all square, it doesn't matter whether you perform the calculation along one of the edges or along the diagonal. If the pixels are not square, the results will differ slightly depending on whether you use an edge or diagonal. If you remember the formula and can calculate square-roots, go ahead and use the diagonal. If you just want a rough idea, calculate for each edge and average the results.
The CoC shrinks with smaller sensor sizes or increased image resolution. The CoC grows with larger sensor sizes or decreased image resolution.
With an APS-C sensor that is 24mm x 16mm and an image resolution of 6000x4000 (24mp), the size of a pixel is 0.004mm. If you consider a 5-pixel "blur" to be acceptably sharp, the CoC would be 0.02mm.
With the same APC-C sensor, but an image resolution of 4000x2664 (10.6mp), the pixel size is 0.006mm. For a 5-pixel diameter, the CoC would be 0.03mm.
With a full-frame sensor that is 36mm x 24mm and an image resolution of 6000x4000 (24mp), the pixel size is 0.006mm. For a 5-pixel diameter, the CoC would be 0.03mm.
DOF CoC is not about megapixels, it is about sensor diagonal size, and the degree of viewing enlargement expected. When we used popular 35 mm film, the common estimate was CoC 0.03 mm. Because, the 35 mm film diagonal is 43.3 mm, and the estimate for viewing the standard definition of an 8x10 inch enlargement (viewed from a distance of 10 inches) was diagonal / 1442, which gives 0.03 mm. This being the dot size a human eye might be able to see (to perceive an enlarged dot size seen at 10 inches), and DOF computes when CoC and enlargement becomes that size. The 0.03 mm only applied to 35 mm film size (the sensor diagonal). Also full frame size DSLR today (also 36x24 mm), and for the popular APS-size sensors (24x16 mm), it becomes 0.02 mm, and still assumes viewing enlargement to an 8x10 inch print.
In past history, there were other estimates for CoC, diagonal / 1732 in the beginning, and later on more recently, diagonal / 1500. However, the 0.03 and 0.02 mm CoC numbers became popular with Japanese camera manufacturers, and that is an implied divisor of 1442 (used by virtually all on-line DOF calculators).
The smaller the sensor (needing greater enlargement), or for greater enlargement than 8x10 inch, this enlargement allows visually seeing the blur better, so the CoC limit has to become smaller (to still recognize the blur the same equal way). Or instead, if you view a smaller image, or have a larger sensor, CoC can become larger.
Depth of Field calculators must ask which sensor size to compute usable CoC, but the DOF standard still assumes viewing 8x10 inch size at 10 inches (rarely mentioned today). I have one at https://www.scantips.com/lights/dof.html that also asks print size (enlargement) and also provides for different divisors.