I the single most universal and important thing will be to make sure the machine has an abundance of RAM. Memory needs for photo (and, if you follow the market into this space, video) are significant, and the one thing you want to make sure you're not doing a lot of is paging, as doing a lot of that will wreak serious havoc on performance.
Now, what is "an abundance"? Well, that's a moving target. These days, I think it's safe to say you'd want an absolute minimum of 4GB, but I'd recommend 8, or even more. And thanks to Moore's law, and the side effects of that law on software, those numbers will keep getting bigger and bigger as time goes on.
There are sure other things you'll want to consider, and I hope other answers will go into them. I think this is the single most likely to make a huge difference, though.