Your two main choices are Gallery 2 and Zenphoto. Both offer plugins that offer shopping cart functionality. Gallery has more sophisticated choices, but unfortunately, the Gallery developers have a habit of throwing everything out and starting again — Gallery 2 is a total rewrite of Gallery 1, and Gallery 3 is another total rewrite.
And, the latest version is stripped down to the point where it doesn't have very much of a plugin ecosystem. If you just want to theme it, that's pretty easy to do (especially if you've got some basic php skills), but anything sophisticated is likely to be more work than it's worth. Luckily, there still are a lot of people that
are actively working on Gallery 2 plugins and so on, so it's not necessarily a dead-end choice.
Both Zenphoto and Gallery (in either version) should be easy to integrate nicely into your main site.