I find the EF 24-105mm f/4 very useful and have probably taken more shots below 105mm with it than any other lens. Not only is it versatile and covers many of the most often used focal lengths one might need in a 'walk around' lens, but it is built like a tank and solid as a rock. It can 'take a licking and keep on ticking' better than a Timex watch can!
But no lens is perfect for every role.
With your EF 24-105mm f/4 L lens you've discovered a common attribute of most zoom lenses: that they have varying optical performance at different focal lengths. For wide angle, wide angle-to-normal, and wide angle-to-short telephoto¹ zooms, the shorter end of the focal length range is usually the sharper end from one side of the frame to the other and the longer end usually demonstrates less geometric distortion and vignetting.
¹ The 24-105mm falls into this category
That's one reason why many portraitists prefer to use prime lenses (a lens that doesn't zoom) - because it can be optimized for the single focal length it provides. You also can generally get better optical image quality at a lower price with a prime lens than with a zoom lens. Of course, what you give up is flexibility. By the time you buy a set of 24mm, 35mm, 50mm, 85mm, and 100mm prime lenses you've likely spent more than you would have for that 24-105mm zoom.
In terms of quality of picture and overall detail of subject (e.g. the face of a child), would it be better to stand closer to shoot at 50mm or stand back at 85mm (assuming the quality of the lens was equal, and disregarding the difference in depth of field, and difference in perspective).
Assuming the quality of the two lenses were equal and the shooting technique does not affect the longer focal length more than the shorter focal length, the overall detail would be the same.
But the quality of the two lenses is never equal. So in the end it comes down to which lens gives you the better quality at a price you can live with. Sometimes that might be the 50mm lens, sometimes that might be the 85mm lens.