Its all personal decision, I personally don`t see the point in buying the 70-200L IS f4, the 70-300L covers that range and its f5 at at 200, one stop makes no difference practically at that point. Image quality wise its excellent, its fatter, shorter and heavier than the 70-200 f 4, its a good general purpose lens particular on a crop sensor.
If aperture is important and for low light action and portraits it is, you really need to look at 2.8 anyway.
It all depends on what you use it for, personally I like the flexibility of the 15-85 and the 70-300L, then I use a few prime lenses for anything else.
Canon Powershot S95, Canon 6D,7D, Canon 40 2.8 STM, Tamron 24-70 2.8 VC, Canon 17-40 L, Canon 15-85, Canon 85 1.8, Sigma 30 1.4, 50mm 1.8, Canon 100 2.8L Macro, Canon 70-300L +Kenko 1.4 Pro 300DGX, Canon 430EX II and RS 4 Classic