Summary: The short answer is that a photographer seems to have very wide rights in Australia - more so than in many other countries.
When in a public place you can take photos of people also in public, and of people who you can see from the public place, with some limitations re looking into buildings etc. There are some limitations on photos of armed forces property - which are probably of no great surprise. When on private property you may take photos but must stop doing so if requested to do so by the owner or their agent. Photos that you have taken up to that point may be retained and used.
You are not restricted when in public from taking photos of children or 'famous' people and they may block your line of vision but may not actively interfere with you - not legally anyway :-).
There is an very large amount or relevant information in the following references. All the Australian ones are included due to high relevance. Non Australian are of varying direct relevance to Australia but informative and probably interesting.
Australian. Excellent. Briefish.
The white hat guide to taking photos in Australia
One of the few pages which leans more to the subjects point of view rather than the photographers. The answers are the same but you are looking into the lens .
NSW photo rights and legal issues
Another: Australian Privacy Foundation
Australia, NSW: Here is a highly relevant an interesting Australia-specific discussion, written by an Australian solicitor and photographer. At a quick skim through it looks utterly superb, addressing all the usual questions about location, subject material, private & public property and more. Updated within last year or less.
NSW Photo Rights - Australian Street Photography Legal Issues
A few of his key comments
In Australia the taking and publication of a person's photograph, without their consent or knowledge but within the limitations outlined below, is not an invasion of privacy, nor is it in contravention of case or statute law.
Privacy advocates may disapprove, but in this country people-photography has always been, and for the moment remains, a perfectly legal thing to do.
In Australia most forms of "unauthorised" photography have in fact been authorised since the 1937 High Court decision in Victoria Park Racing v. Taylor (1937) 58 CLR 479 (at p.496). This was reaffirmed recently in ABC v Lenah (2001) HCA 63, where the Court ruled that despite the passage of decades since Victoria Park, any concept of a Tort of invasion of privacy still does not exist in Australia.
As Justice Dowd put it with ruthless clarity in R v Sotheren (2001) NSWSC 204:
A person, in our society, does not have a right not to be photographed.
The following by is an analysis of legal issues which apply to street photography in NSW Australia.
Created in response to objections to my Sydney Unposed project, it is written from a photographer's perspective, with a focus on what rights shooters have (and don't have) when it comes to candid photographs of people. Please note: it is not an encyclopaedia on every possible aspect of photographic law, so it does not attempt to address issues like anti-terrorist legislation, council photography permits or National Park commercial photo restrictions. Instead the sole purpose of the following is to discuss legal issues which apply to people photography only.
In case you are wondering, I am a photographer and qualified solicitor (UNSW 1991) who worked for a short while at a large Sydney law firm, before leaving the profession in 1992 to find a more honest way to make a living. So the following is based on an (ex) practitioner's understanding of Intellectual Property and Privacy Law, and not just the usual Internet Hearsay. =)
Australia - excellent. 2007.
The following site is a little scrambled with some material appearing in different forms on differnet pages (eg 1 & 3) are versions of the front page etc.
But it seems to have much to say on many issues.
Not all pages have content - the links below are to pages that did have material. There are probably others.
Even just reading the subjects will give food for thought.
Agh!!! - now I see there is an index in the left margin that largely makes the following redundant. Not wholly so as some pages are not referenced from all others.
Unauthorised photos - This page considers Australian debate about unauthorised making and publishing of photographs
Page 1 - http://www.caslon.com.au/photonote1.htm
...and similar http://www.caslon.com.au/photonote3.htm
Anxieties and issues - http://www.caslon.com.au/photonote2.htm
Index page - http://www.caslon.com.au/photonote4.htm
Paparazzi - http://www.caslon.com.au/photonote8.htm
Venues - http://www.caslon.com.au/photonote9.htm
Defence - http://www.caslon.com.au/photonote10.htm
Justice - http://www.caslon.com.au/photonote11.htm
Skies (from above) - http://www.caslon.com.au/photonote12.htm
Streets - http://www.caslon.com.au/photonote13.htm
Incidents - http://www.caslon.com.au/photonote14.htm
Your image - http://www.caslon.com.au/photonote16.htm
Australia Consent for photography not required - much as elsewhere.
Photography rights in Australia and other countries.
Links to NZ - unlawful photography in public places
** Other: **
Related - not actually photo taking: In Singapore I was asked to take a tripod off a DSLR that I was carrying as I boarded a train. At the time there were few people around (unusual in Singapore :-))and the tripod would not have caused inconvenience to others but I can see that in busy periods it would be a less user friendly piece of baggage and I appreciate that rules need to be simple to follow.
Non Australian but good. Photography and the law - know your rights
Book. Amazon. $21.47 Legal Handbook for photographers
Andrew Kantor comments with links
Reporters committee for freedom of the press - Photographers guide to privacy or PDF
2006 - USA - Photographers rights
USA - video may be useful - did not view
USA - Flickr - Photography is legal
US discussion - useful
Lisa Law Photography Australia - what happens when you use a search engine :-)