The size and weight of a 70-200mm f/2.8 zoom is based on the laws of physics. Since the f-number is a function of focal length and entrance pupil size (apparent or effective aperture), the objective of any 200mm f/2.8 lens must be a minimum of 72mm in diameter. In the case of a complex zoom lens that includes corrective elements made of dense optical materials and often Image Stabilization, the weight adds up rather quickly. Housing such expensive optics requires a robust shell as well if it is to be durable. But you likely already knew all of that.
The key question each photographer considering such a lens must answer is: Is the extra cost, size, and weight worth the advantages the lens provides over cheaper, smaller, and lighter alternatives?
For generations of photojournalists, the answer has been a resounding, "YES!" The focal length and speed are both frequently needed in that field and the nature of covering live news as it happens doesn't always allow time to swap out prime lenses. Many of those guys are carrying two or three bodies with a wide angle, medium, and telephoto lens so they can swap focal lengths quickly.
Many portrait photographers, especially those working primarily in field locations rather than a studio, prefer to have this type of lens in their arsenal. In the case of the EF 70-200mm f/2.8L IS II USM, image quality approaches and even sometimes matches that of prime lenses over the entire focal length and aperture range of the lens. Changing lenses in the studio is one thing, changing them in the field and then later discovering dust spots on all the images in post production is another. Redoing the shoot could cost thousands of dollars.
Almost every sports specialist will also own one of these types of lenses. Most will have longer "super telephoto" glass as well, but they consider this their mid-range zoom and will usually have one mounted on a second body when shooting an outdoor sporting event. Indoors it often becomes their primary lens.
There are many other times when having a fast, high quality 70-200mm zoom is essential to obtaining the results you need. If another cheaper, smaller , and lighter lens isn't up to the task then such a lens is your best option.
You may notice that the list above is made up of those for whom the quality of their images has a direct effect on their ability to earn a living and maintain a professional reputation. While the most important piece of any photographer's gear is the 8-12 inches behind the viewfinder, there are some shots that can only be obtained with higher quality gear. As the technology improves, so do the expectations of editors and clients.
The favorite piece of gear I own is my EF 70-200mm f/2.8L IS II USM. While other zooms I've owned in the past could handle many of the shots I use it for, that last 15-20% are often the photos that make me say, "YES!" when I view them on my monitor for the first time. I'm often carrying 15-20lbs of gear when shooting, so the weight of one lens isn't as large a factor as having the tools I need to get the results I require.