The short and simple answer is that a photograph is a form of expression. I'm free to express myself as long as I don't infringe on somebody else's rights in doing so.
In the case of something like street photography, that usually works out fairly simply: if I take a picture in a place where somebody has a reasonable expectation of privacy, then I'm infringing on their rights, and I can't do it (without their permission). If they're in a place where they don't have a right to privacy, then I'm not infringing on their rights.
Looking at it from the other direction, going into a public place gives implicit consent to be watched, talked about, written about, videotaped, photographed, etc. If somebody prefers not to give the consent implicit with going into public, that's fine -- they're welcome to stay in a private place as long as they wish. When/if they choose to go into public, they have given up any right to privacy, and others' right to freedom of expression becomes the controlling factor, and a photograph is no different from any other form of expression.
Giving the subject of the speech, writing, photograph, videotape, etc., control over the results would infringe on the freedom of expression of the speaker/writer/photographer. The mere fact that they are the subject of the expression in question does not give them a right to limit or infringe on others' freedom of expression. Attempting to grant them such a right would utterly destroy freedom of speech/expression.
From a philosophical viewpoint, I see little difference between publishing a photograph of something "scandalous", or writing about the same subject. If we allow the subject to control that, it's a short, slippery slope to "you can't gossip about me", and even "you can't think nasty thoughts about me."
Edit: As @John Cavan pointed out, there are limits on the loss of privacy implicit in going out in public. In particular, what I make public becomes public, and what I keep private remains private. There mere act of walking out my door does not, of course, give anybody the right to look in my bank account, remove my clothes so they can take pictures of me naked (not that anybody would want to in my case!), break into my house to see what videos I like to watch, etc.