The difference, considering you can keep all else equal (DOF, framing, metering) is the "compression" of the Z axis (depth). Your background gets enlarged, and with the same DOF that makes it appear more blurry, because the larger we view a blurry part of an image, the more blurry we perceive it.
The worst part is that the face you are portraiting gets affects, too. Because each depth gets enlarged differently and the face spans 20-30 cm of depth, you get a certain distortion. Anything below 50mm will be perceived as distortion, and the upper limit is around 135mm.
I did a test with my wife with 28,35,50,85, and 135mm and she liked 50mm so-so and loved 85-135mm. This was on a aps-c.
Here I show 50mm vs 135mm on a dog:

Notice how the nose of the dog seems short in the 135mm lens. The face gets flatter with higher.
Here's a person with 20-200mm:
http://www.petapixel.com/2012/08/12/how-focal-length-affects-your-subjects-weight-in-portraits/
So in conclusion, if you can get away with it, go for 85mm for the best balance, 135mm if you need to flatten the face a bit, and 50mm if you don't have room, or you need to enhance some features.
For a portrait on a aps-c sensor you need to be 1.6m away on 50mm, 2.72m on 85mm, and 4.32m on 135mm. With F/1.8 you get 7cm DOF, which is good to keep facial features and the nose in focus, while the back of the head gets blurry. Mind you, a 50mm 1.8 costs $100, 85mm F1.8 $300, and 135mm F1.8 costs $1700!
http://www.dofmaster.com/dofjs.html