Actually it's more than just the aperature.
The First lens on your list also has built in Image Stabilization, which will allow you to hand-hold your lens at near 2-3 stops 4 STOPS!! lower than what is possible without IS. One handy rule for shutter speed is that is should be 1/(Focal length), so at the maximum reach, 200mm, you would need a shutter speed faster than 1/200s. With Image Stabilization you can hand hold around 1/25s (3 stops) or even 1/10s ( 4 stops) if you have steady hands!
So that is one reason for a big bump in price, even when compared to the Canon EF 70-200 EF f2.8 Non IS.
Since the lens opening is larger, the optical elements have to be designed differently to account for the larger amount of light. It's not as simple as making a larger max opening. It's a whole change in optical characteristics, which includes higher precision glass, number of elements, which translates into higher production costs. And again, when you tack on built in Image Stabilization, you have a much more complicated system, which costs more to design, and to build properly, which is reflected in the high sticker price (for a fun time, I recommend looking around for the Canon EF 400 F2.8 IS...).
And just so we're clear, the difference between F2.8 and F4 is non-trivial. That is 1 full stop of light, which if everything else is equal, you can shoot your camera with 1/2 the shutter speed as you could with a maximum lens at F4. In doors this can be the difference between getting the shot, and not. Not to mention the depth of field and background blur that occurs at F2.8. Also many Canon DSLR's have high precision AF points when combined with F2.8 lenses, plus F2.8 produces a brighter view finder.
Finally, the Canon 70-200 F2.8 IS II is a new lens, which is hard to come by, which means you will pay list price for it. Just wait it out, and you'll probably be able to pick it up near what the 70-200 F2.8 IS Mark I went for, about $1600 street.