'How do I know when I'm done' is almost as important of a question as 'what do I take pictures of,' and it's a question that most fine art photographers I've ever met or worked with struggle over. There's a certain 'luxury' sometimes to having a client attached to a project because it forces completion (you're on their deadline and you don't get paid if you don't finish!), but when it's a project with no 'end-user' it can breed things like endless fiddling while trying to get every image 'just right,' seeking out 'just one more' to add to the collection,' and hating the work because it never quite matches up with what the artist 'imagined' it would all look like. In fact it is a common enough dilemma that many of my professors at art school talked about strategies for overcoming the somewhat natural tendency for artists never to finish what they start.
The three techniques that I come back to time and time again in my own work are:
On The Clock - Setting a hard deadlines is the primary thing I do to keep myself on task... and I set lots of deadlines. My total project deadline may be 3 months, my session deadline may be 2 hours, and my deadline for finishing the post production on the photo I'm currently working on may be 20 minutes. The key is to set the deadlines, and honor them... Yes, they're artificial, but I treat them as if they're not.
Your own best advocate - Another thing I frequently do is to enlist someone to be the 'gatekeeper of done' for my work. This is someone I trust completely who I can show my work in progress to and discuss it openly. This person (actually I have a few) is able to tell me when they think I've moved from productive work on a project to simply 'spinning my wheels.' In essence I grant them the power to tell me it's time to stop.
Tell the world - There's nothing like a show or an 'unveiling' of some sort to light a fire and switch me into 'just gotta get it done' mode. Because I'm not famous enough to be able to call up fancy schmancy galleries in order to have a real show, I often set up shows at my house and send out invitations to folks over to look at my work, have cocktails, etc. Turn it into a fun evening. I've even seen this work successfully as a 'virtual show' where someone tells their group of friends/family that they'll be releasing a website of a project on a certain day... Although for me personally there wouldn't be the level of motivation necessary to get it done in a 'virtual' release like that.
Now obviously the one big thing an artist needs in employing any of these techniques (or others you may come up with) is the integrity to stick with the commitments you make to yourself, even in the face of the variety of emotions that artists go through regarding their work: not feeling like the work is done, any good, worth showing, etc. For me part of the 'exquisite pain' of being artistic is realizing and accepting that I'm probably rarely (if ever) going to be completely satisfied with anything I ever do... That dissatisfaction is usually at the heart of me not getting things finished and once I acknowledge that fact it is often easier for me to simply let things go.