I don't see that there's anything "wrong" with the subject not looking at the camera, but there is an unavoidable and striking difference between subjects eyeing and not eyeing the camera. There are primal instincts to study a subject that is making eye contact: Is this person trying to engage me as an ally? Do they need help? Are they a threat? Are they eyeing me as prey? As a mate? Furthermore, depending on the circumstance, and sometimes informed by culture, it is fundamentally rude to return (or not return) eye contact. But if the subject is not looking at the camera the observer is free to just observe without any of those instinctive circuits being engaged.
For example, here's a busy subject, but contrast your reaction to the first version where her gaze is only slightly to the side of the camera with the second where she is looking directly at it. To me these are completely different effects: In the first case my eyes are free to wander the image. In the second case it is almost awkward to stare at anything other than her face – my eyes are drawn back there because she is looking at me and it feels like I'm ignoring her to not answer her gaze.
(Click for uncropped image.)
Note that the same holds true for animal subjects, and for mostly the same reasons: If the animal is looking at me I have to decide if it's a threat (or perhaps whether it's a meal that is about to run away). If it's neither then I have to decide whether to interact with it. The animal looking at the observer is making the same calculations. We know that and sense that. We are unavoidably engaged.