I don't see that there's anything "wrong" with the subject not looking at the camera, but there is an unavoidable and striking difference between subjects eyeing and not eyeing the camera. There is a primal instinct to react to a subject that is making eye contact. Humans are social animals and so eye contact can trigger a bunch of core survival circuits: Is this person trying to engage me as an ally? Do they need help? Are they a threat? Are they eyeing me as prey? As a mate? If the subject is not looking at the camera the observer is free to just observe without any of those instinctive reactions being engaged.
For example, here's a busy subject, but contrast your reaction to the first version where her gaze is only slightly to the side of the camera to the one where she is looking directly at it. To me these are completely different effects: In the first case my eyes feel free to wander and take in all the points of interest at their leisure. In the second case it feels almost awkward to stare at anything other than her face – my eyes are drawn back there because she is looking at me and it feels like I'm ignoring her to not answer her gaze.
Note that the same holds true for animal subjects, and for mostly the same reasons: If the animal is looking at me I have to decide if it's a threat, or if it's a meal that is about to run away. If it's neither then I have to decide whether to interact with it. The animal looking at the observer is making the same calculations. We know that and sense that. We're unavoidably engaged.