In addition to the other great answers already shared here, there is the following psychological aspect to consider...
When you are face-to-face with someone, you are focused on the present situation, centered around your communication with them as another individual
You are aware that they are also observing you, as part of an exchange happening in real-time. So as far as your cognitive bandwidth is concerned, this interaction takes precedence over picking apart their facial imperfections.
More on what's going on in our brains when we talk to people:
On the other hand, if there's something exceptionally unusual about their appearance, this feature could actually distract you from the conversation. ("Oh... should I pretend not to notice his eyepatch? He definitely notices that I notice. Should I just ask about it to get past the awkward phase? No, it's gotta be a touchy subject. Note to self: do not make any pirate references... Maybe he will bring it up first, so I don't have to. No luck so far... stuck in awkward limbo.") It can be anything, but you get the idea.
When you focus your attention somewhere for an abnormal amount of time, the other person will likely take notice. One example is the classic “my eyes are up here” situation. Another is the guy who uncomfortably notices people frequently glancing at his receding hairline, possibly without the conversation partner even being aware they are staring rudely.
When you're looking at a person face-to-face, there's an elaborate communication situation taking place, full of social cues, and you're observing the other person's face within the context of the situation.
But what about when we're not looking at someone face-to-face, but perhaps glancing at them as they walk past?
If you're looking at someone from an arbitrary angle where their line of sight is not focused on you, you likewise won't be seeing their face straight-on - and this means you are once again unlikely to notice any finely detailed asymmetry in their facial features.
When it's a photograph
When you're just looking at a photo of somebody, you know that photo can't stare back at you. Because of this, you just scan over it like any other image. Well, maybe a face is a little different from just any other image, because our brains have a dedicated region for recognizing faces called the fusiform face area (FFA).
Paradoxically, our brains do expect some degree of asymmetry in faces. This is what makes photrealistic human faces one of the most challenging subjects to create if you are a 3D artist. (Yes, I know this statement is becoming obsolete as I type these words because AI even generates 3D models now, but it's still not perfect, and by "3D artist" I mean in the traditional sense, not relying heavily on AI.)
In short, knowing we're looking at a face will make our brains take notice, but being immersed in the interaction will also necessitate that we overlook some small facial details, to instead focus on having effective in-person social interactions.