I'm just trying to decide which new lens to get. I need to take surf and windsurf photos from onshore. Sometimes 300 mm is enough but there are many times they are too far away. I have a sigma 70-300 but I need something better, if the light is not perfect it doesn't give me the sharpness I hope for. I can not afford the 400L 2.8 of course, so I'm trying to choose between the Canon EF 100-400L, Canon EF 70-300L (which I heard is far better than the base one), or Sigma 50-500. My body is a canon 600D.
Based on several friends' experience with both the Sigma 50-500mm and 150-500mm I wouldn't recommend either. Once you move past 250mm or so they get softer and softer as the focal length increases. Most reviews and test charts I've seen for those two lens bear this out. There are several good alternatives that will give you better image quality than whichever 70-300 Sigma version you are currently using, but once you get beyond 200mm it is hard to find a zoom lens that gives excellent image quality at a price that isn't on par with something like the $10K+ EF 400 f/2.8 L. Primes in the 300mm+ range aren't cheap either, but you do get a lot more bang for your buck than with zooms.
Some very good photographers like (as opposed to love) the Canon EF 100-400mm f/4.5-5.6 L IS as a good compromise between absolute image quality and affordability, others don't care for it. It is an older design that is past due for an update. It was very good for the film era, but its shortcomings can be revealed by high resolution digital sensors. The optical performance of most of Canon's telephoto lenses released since 2008 or so are a step above their predecessors. Unfortunately, so are the suggested retail prices. The push/pull zoom of the 100-400mm takes getting used to, but also creates an additional issue when using it at the beach. Although the air that must move in and out of the barrel when the lens is zoomed is filtered somewhat, there's a limit to how much it can catch and allow the lens to remain useable. In a salt water environment with both heavy moisture in the air and salt spray you would be much better off using a lens that doesn't pump air in/out every time the focal length is changed.
Your question compares the EF 400mm f/2.8 L to the EF 100-400mm f/4.5-5.6 L IS and the EF 70-300mm f/4-5.65 L IS, but neither of the last two lenses are near as fast as the first. The EF 400mm f/5.6 L is in the same price range as the 100-400 and the 70-300 L, but gives better IQ than the 100-400 at 400mm and obviously has more reach than the 70-300. The EF 300mm f/4 L also delivers very good IQ for around $1,500. Both the 300mm f/4 and 400mm f/5.6 are older designs that could probably use an update, but the newer "II" super telephoto series Canon introduced in a flourish in 2011 only included the fastest version for each focal length: 300mm f/2.8, 400mm f/2.8, 500mm f/4, and 600mm f/4.
If you need the versatility of the zoom lens and can spend a little more than the $1,400-1,600 for the 100-400 or 70-300 L, you might consider the following combination: EF 70-200mm f/2.8 L IS II + EF 2x III. That combo will set you back around $2,600, but that is still a far cry from the $5-10K for the Super Telephoto "II" series lenses. For your money you get the best 70-200mm zoom in the world when used bare. When combined with the EF 2x III you still have a 140-400mm f/5.6 lens that can focus fast enough to shoot sports (most lenses coupled with a 2x extender can not) and IQ that holds its own with the 100-400 and even the 300mm f/4 and the 400mm f/5.6. The "II" version of the 70-200 is the only zoom lens I would consider using a 2x extender with for any kind of sports/action.
Anytime you are considering a lens in this price range, it is always a good idea to rent one for a few days to see if it matches the need you have for it. If there's not a good rental house in your area, borrowlenses.com and lensrentals.com both have excellent reputations.
It's difficult to say much without seeing examples, but it sounds like what you really need is a lens with a greater focal length. How much do you actually need a zoom lens? If not, you'll in general get better bang for the buck with a prime lens.
In particular, you haven't mentioned a reason why you've ruled out the EF 400 f/5.6 which is around the same price as the EF 100-400. It doesn't have IS, but that may or may not be an issue for you depending on whether you're using a tripod and what shutter speed you're shooting at.
If you do any work in closer as well, I highly recommend the 70-200 f/2.8L IS II with 2x teleconverter. The 70-200 f/2.8 is an amazing lens for in closer and is still pretty fast with the 2x Extender III. It is also weather sealed and fixed volume. The 2X teleconverter does make it much slower, but in daylight that shouldn't be a problem.
I've personally used that combination and was very, very pleased with the results. The corners lose a little sharpness, but the center is still very sharp. The color reproduced by the 70-200 with 2x was far nicer and far sharper than my 70-300 f/4-5.6 IS II (non-L).
I would avoid the Sigma 50-500 as any lens covering that range has to make major compromises that will have a negative impact on it's overall performance.
As far as between the 70-300 and 100-400mm Ls (assuming you can't swing the 70-200 with 2x) it really would depend on how much reach you need. If 300mm gets you there, then that would work, if not, then it doesn't matter.