I can speak from actual hands-on experience here, for a change :) I owned both simultaneously; I kept the 24-105 and sold the 24-70. For ME, the benefits of IS outweighed the benefits of f/2.8 and the much better lens-hood of the 24-70. Your mileage may vary though.
The 24-105 IS allows me to shoot at 1/10 second at 105mm and expect a sharp photo. Or I can go to 1/5 and take three shots of every motive, and expect at least one to be sharp. With the 24-70 I'd have to live by the 1/focal length rule, exceeding 1/50 at 70mm would be iffy. The slow shutter speeds with the 24-105 lets me stop down quite a bit for depth of field for similar exposure as the 24-70 would give me at f/2.8. Extra depth of field is a Good Thing most of the time - I use the full-frame 1DsII so I don't get any free DOF from a sensor crop.
This, of course, only applies to stationary subjects. Which, fortunately, is what I shoot with the lens. In my experience, for moving subjects, the one extra stop that f/2.8 gives over f/4 is neither here nor there; if f/4 isn't fast enough then f/2.8 probably won't really cut it either. For low-light photography of moving subjects I take out the 85mm f/1.2L which is a portable black hole which will suck up any light that is present in the room. Candlelight is more than enough.
That said... the 24-70 was a more fun lens to use than the 24-105. It had a certain something. The 24-105 is a staid, practical, useful lens without a poetic bone in its body. Ho hum. The plain sister with the good personality, if you will. But it's the lens I use the most.