If mirrorless cameras focus through the lens and directly focus using the image on the sensor, why do some lenses still backfocus and need calibration? What is the mechanism that allows this issue?
Using only contrast detection to autofocus and confirm the subject is in focus requires multiple move and measure cycles. This is slower than using a single phase detection measurement to calculate the amount and direction of lens movement needed to bring a subject into focus and then making a single lens movement by the prescribed amount. This was particularly the case when mirrorless interchangeable lens cameras first began to be popular. To increase the speed of autofocus in such cameras, hybrid systems that also used various forms of phase detection AF were developed. With such main imaging sensor based phase detection methods the difference between the camera's calculation of the amount of movement needed and the actual amount of movement needed as well as the difference between how far the camera instructs the lens to move and how far the lens actually moves must both be taken into account.
These differences are what AF calibration methods for mirrorless cameras attempt to correct.
To understand why it is useful to do focus calibration with lenses used on mirrorless cameras with hybrid PD+CD AF or even PD only methods, it helps if we first realize that there are multiple reasons why doing autofocus calibration is useful for reflex cameras. The difference in optical distance between the lens to the PDAF array and the lens to the main imaging sensor is but one among several.
When autofocus systems were first designed during the film SLR era, they needed to be both at least as fast and at least as accurate as a typical user could focus manually with a non-AF lens or camera. But merely matching manual focus systems in both speed and accuracy didn't give the user a reason to spend the extra money for the AF cameras and lenses. They needed to be noticeably better in at least one of these two categories. Camera manufacturers chose to concentrate on making early AF systems as fast as possible while more or less matching the typical accuracy that could be achieved by moderately skilled users of existing non-AF SLR camera systems.¹
Part of this had to do with the nature of roll film. It doesn't lay perfectly flat against the back plate behind the shutter curtains, especially after having been coiled up in a film cartridge for weeks, months, or even years. The dye layers in color film emulsions are layered and slightly different distances from the lens. Beyond that, most consumer grade lenses for 135 format SLRs weren't designed to be significantly sharper than what the limitations of roll film could exploit. As long as early AF systems were "good enough" to match the expectations of users transitioning from manual focus SLRs, a noticeable speed advantage was enough to make the new cameras and lenses with AF desirable to many buyers.
The main compromise designers of AF systems for SLRs made was to design an "open loop" system that measured how far and in which direction the lens needed to be moved to bring the subject into focus. Once that measurement was accomplished, if the shutter button was fully pressed the mirror started to move up out of the way at the same time the instructions sent to the lens were being carried out. Without the mirror being fully down, the PDAF array was totally blind to the image projected by the lens. The camera didn't wait around to confirm that best focus had been achieved because that would have required leaving the mirror down for much longer. It moved the lens however far it had calculated and took the picture without confirming how accurate the calculation was or how accurately the lens had actually moved by the instructed amount. Eventually, increasingly accurate focus position sensors were placed in lenses to more precisely measure how far the lens had actually moved. If needed, additional movement instructions were sent to the lens to place it at the more precise focus position desired. This was all done when the mirror was already moving or in the up position and the PDAF sensor array was blind.
Early non-SLR digital cameras, on the other hand, did not use phase detection technology at all to autofocus the lens. Instead, the camera measured total contrast on a specific area of the main imaging sensor and moved the lens back and forth until it found the focus position where contrast was maximized. This proved to be much more accurate than the capability of PDAF systems at the time. Since contrast focus requires several measure and move cycles, it also proved to be painfully slow enough as to make it virtually unusable for any moving subject to be brought into accurate focus, both in terms of timing the frame capture for compositional intent (i.e. catching the subject at the exact position one desired as it moved within the frame) and in terms of being able to actually focus on the subject at all before the subject distance had changed.
Increases in processing capacity within the size, cost, and power consumption constraints camera makers faced helped to speed up contrast detection AF over the years, but PDAF remained significantly faster. Improvements were also being made to PDAF speed, partially due to the same increases in processing capacity.
Eventually the resolution limits of digital sensors began to exceed the resolution limits of roll film. These advancements in camera sensors also gave rise to new generations of lenses needed to complement these higher resolution sensors. Resolution limits of sensors and lenses reached the point where typical manufacturing tolerances between a specific copy of a camera model and a specific copy of a lens model could noticeably affect how well they worked together. Only then did concerns about improving the accuracy of AF systems without sacrificing speed move to the front burner. Allowing the end user to measure and adjust for the very slight differences in manufacturing tolerances without having to send both the specific camera body and the specific lens to a service center for adjustment is the purpose of autofocus calibration.
Various camera makers tried several different methods of hybrid autofocus that incorporated both phase detection and contrast detection using the main imaging sensor in mirrorless cameras, as well as with SLRs being used in "Live View". These worked with varying degrees of success as they attempted to balance the need for speed with the need for increasing accuracy as resolution limits continued to improve. The closer phase detection could get the lens to the position of sharpest focus, the faster the total hybrid system could work. Eventually main image sensor based PD reached a point where it was accurate enough for most purposes and at least one camera maker stopped using contrast detection to fine tune the final focus position before capturing the frame in fast action shooting scenarios.²
For more on the differences between how contrast detection and phase detection autofocus work, please see this answer to a tangentially related question here at Photo SE.
¹ When Canon released the first USM lens, which significantly increased the speed of AF lenses, and then the EOS-1 camera body capable of exploiting USM lenses speed, it was game-changing for professional sports shooters and photojournalists who need to catch peak action of moving subjects. Until USM and EOS-1 series bodies came along, most pro shooters still preferred to manually focus. The earliest AF cameras and lenses were sold mostly to amateurs and enthusiasts, not to full-time pros. In the late 1980s Nikon had held a greater than 75% market share of SLRs used by professional photographers in the 135 ("35mm") format for about two decades. After introducing faster focusing USM lenses, by the mid-1990s Canon was being used by a majority of those same professionals and this continued for the next two-plus decades.
² Ironically, it was Canon, the company who developed main imaging sensor based "Dual Pixel AF" to increase performance of AF when shooting video by leveraging phase detection AF, who released an SLR in 2020 - The EOS-1D X Mark III - which replaced the traditional PDAF line array with what amounts to a miniature secondary imaging sensor placed in the same position as the PDAF line array had been placed at the bottom of the light box. Things had come full circle as processing power finally increased to the point it made image sensor based AF as fast as PDAF while keeping the other advantages of image sensor based AF.
Phase detection AF is not open loop (one measurement, one movement)... it will not confirm AF and take a picture unless it sees what it believes to be an in-focus image. Sep 28, 2022 at 13:15
Early PDAF systems were most certainly open loop, particularly when operating in Servo mode. Yes, they did sometimes do additional optical measurements in static One Shot mode when the shutter button was half-pressed until a focus confirmation light comes on. But Servo never locks focus, thus never gives a confirmation light. Latter systems may or may not use optical confirmation in some AF modes, but many early SLRs did not. Just ask Uncle Roger Sep 29, 2022 at 3:50
@StevenKersting The biggest flaw with the "squirrel test" in the DP Review article is that it assumes measurement and calculation are instantaneous upon initiating AF with either a half-press of the shutter button or a press of an AF-On button. That's ridiculous. So is basing tests of 2011 models to make determinations about cameras introduced in 1985-87. Sep 29, 2022 at 4:03
Further, since any SLR using viewfinder based AF systems do not use the actual imaging sensor to confirm AF before opening the shutter to expose the sensor, none of the are true "closed loop" systems. Whatever is confirmed prior to the shutter opening to begin exposure is based on the discrete PDAF array, not on the actual sensor used to take the photo. Sep 29, 2022 at 4:27
1@StevenKersting A careful reading of the above answer would reveal that it makes no such claim that modern systems use "partial open loop" methods in all AF modes. It points out that in the development history of AF, which began being available to consumers in the 135 format in 1985, such was the case. It also points out that in certain AF modes the camera does not always pause to get the absolute best possible AF accuracy, but instead exchanges time for slightly less accurate focus that is still good enough and more likely to catch some moving subjects in focus. Oct 2, 2022 at 15:11
Some cameras have phase detectors built in the sensor (Canon "dual pixel" for instance), so they work roughly the same as DSLRs in that regard. Phase detection is faster (and likely less energy-intensive) than contrast detection.
"In focus" is not a finite value, it varies.
For instance, say you wanted to take a picture of a blurry painting, or a silhouette of a tree/animal in fog... the camera will still take those pictures and not require the same degree of detail/contrast/focus it could achieve in another situation.
And there has to be some limit at which the camera accepts best focus is achieved whether it actually is or not; otherwise the camera would continually hunt for focus or focus extremely slowly.
It's also possible for a mirrorless camera to experience focus shift when using lenses of very large max aperture (a shift of best focus position at different aperture settings). And adapted lenses/older lenses may have less accuracy in their focus drive mechanisms. Additionally, many lenses which are made for SLRs/DSLRs may have a focus offset setting programmed into them... I would expect a mirrorless camera to ignore that offset value; but I don't know if that's actually possible.
But if you are experiencing consistent focus errors with a mirrorless camera and native lens it is almost certainly due to user error or a malfunctioning component.