Convenience is a matter of your own preferences. Some people find eating out more convenient than cooking at home, while other people find it more of a hassle. Some people think biking is inconvenient, while others can't stand sitting in traffic. This is really one of those same things. If there were a universal answer, one of the options wouldn't exist on the market. Lucky for us, there's enough people who have all sorts of different preferences, so the camera gear market covers us all. You just have to figure out where you fit.
I use only prime lenses, and I don't find it inconvenient. I generally know which lens I want for a given situation, and then stick with it for a while, planning out in my head when I'll swap.
Having done the lens swaps quite often, I can do it in matter of seconds, and am not worried about dropping my lenses or getting them dirty. With my first DSLR, sensor dust was a problem, but since then then built-in dust-removal has gotten so much better that it's a non-issue.
It's undeniable that this is the minority and that the majority of photographers use zooms today, but I don't think it necessarily follows that you should. You should do what you like and what feels best to you.
There are plenty of advantages in prime lenses. Small size for great optics makes them more convenient in many ways. I carry around three or four different lenses every day in a very small camera backpack, and I don't think twice about the weight. (In fact, all of my regular-use lenses together are half the weight of a 24-70mm f/2.8 zoom.) And since you mention the importance of low light, prime lenses are often a stop or two faster.
For some of your specific concerns:
- As noted, dust isn't a big problem in most cases.
- I actually do use my 40mm prime lens most 80%, although I second what Matt Grum suggests about "30%-each for a set of three primes" as a common situation.
- You're right that extreme crop isn't a replacement for higher focal lengths, but a) you can use primes with higher focal lengths just fine and b) don't discount the image quality you can get that way. Take a look at the butterfly crop from my review of the Panasonic Leica DG Summilux 25mm / F1.4 ASPH. I'll put crops from that lens and camera combo up against a point and shoot zoom any day.
And, there are other advantages in primes too. Take a look at the related question Would a fixed or zoom telephoto lens be better for learning?, and at the Mike Johnston article The Case Against Zooms linked from my answer there.