Short:
If you had a 70-200mm f/2.8 non IS lens or a 70-200mm f/4 IS lens that were otherwise identical (see below) then, if you value the ability to shoot handheld in marginal lighting situations, based on my solid personal experience of the functional equivalent of these two choices, you would end up much much much happier overall with the IS lens. This seems to go against the general advice being given here. Given that my answer is based on specific practical direct comparison between IS and non IS systems I'm not sure why other say what they do.
IS gives you the ability to compensate for camera movement at massively lower shutter speeds. This then allows you to take SOME lucky/clever shots that work with a given target type. Without the IS these clever/lucky shots are not able to possibly succeed.
One stop of improvement is always nice to have but of itself is not vastly useful. If one stop gain makes the difference between sharp & sparkling success and blurry dismal failure and/or the viewfinder brightness difference is stunning you are taking photos in an alternate universe and the answers here do not apply.
The difference between f/2.8 and f/4 is minimal, all else being equal. The one stop difference (2 x light level change) is relatively unseeable in many cases - a say 40% change in ISO and a 40% decrease in shutter speed would restore the same lighting situation and then give you 3 or 4 stops of IS. If you can hand held at 1/200s in a given situation you can probably try that bit harder and manage 1/150 s. If ISO 800 is OK then odds are ISO 1200 is not too bad - those two give you the 1 stop change back - and now you have 3+ stops of IS available - see below.
The question What is more important, f-stop or IS. There is almost no price difference between the Canon 70-200 f2.8 without IS and the f4 with IS has [wrongly] been deemed to be identical to this one. As that question has been closed in favor of this one, I'm posting my answer here with minor mods to reflect the move.
Given the straight choice f/2.8 and no IS or f/4 and IS I would go against the majority opinion here and choose the IS lens. This is based on very convincing (to me) personal experience.
Until recently ALL my lenses were effectively IS as I have upgraded through a line of first Minolta and then Sony DSLR's - all with in-body anti-shake, which is essentially the same as having IS on any lens you use - no matter the price or age. While you can turn anti-shake off (mainly intended for when a tripod is used), in almost all cases anti-shake is enabled and no great attention paid to its effects. I have had the Minolta 7D and Alpha 5D (same internals), Sony A700 and Sony A77 SLT.
Then I bought a Nikon D700 in addition to the A77 and used them together. I expected the D700 to excel in low light situations, and (of course) it does. When a tripod or stable shooting position is used the D700 is vastly superior to the A77, as expected. But in real-world hand-held low light situations, when a non-IS lens is used, the A77 may be significantly better than the D700 in many situations, due to the A77's in-body antishake. In a given situation, if maximum aperture for both cameras is say f/3.5, the A77 may operate acceptably at ISO 800 and the D700 at ISO 3200. In that scenario the D700 will allow 4 x the shutter speed - say 1/120 th second for the D700 against 1/30th for the A77. So far so good. But the A77's antishake gives it 3 to 4 stops performance advantage. How much depends on perception and situation but the end result for me is that with the same maximum aperture lens on both cameras, and no in-lens IS in either case, the A77 with in-body antishake will give superior sharpness results in the same low light situation for similar noise content. This is NOT what the world tends to tell you.
Worse, if you are not comparing a low noise and high noise camera but only f/4 versus f/2.8 lenses on the same camera, IS will offer you very significantly lower shutter speed for the same ISO and light level. The ability to take acceptable photos of real-life still objects in low light OR the ability to use lower ISO and so lower noise will be vastly improved.
What about non stationary targets?
IS, people always point out, only helps with camera motion and not subject motion.
This is true at a basic level, but misses a very important point.
For a given light level, the say 1/30s usable with an IS lens or body may be too slow to accommodate a moving target in some cases. BUT, not in all cases. Maximum possible care, panning, waiting for a pause in motion or change of direction and similar will "sometimes" [tm] allow say 1/30th to work. If the speed does not cause problems from camera shake due to "IS" then you are able to try these careful/luck/tricky shots and some will work. Without IS the same situation will almost certainly deliver almost 100% mush.