I don't know about other systems (I've heard 5e closes the gap a lot) but in 3.5e the gap between casters and non-casters is huge.
First of all, a lot of spells are "silver bullets", the kind of spells you have no reason not to take because -if succesful- they can shut down 90% of all combats. Even as early as level 1, a well-timed sleep or color spray can completely wreck the enemy to the point where they might as well just drop dead and save your party the trouble. At higher levels the options only become more ridiculous.
Secondly, utility. Besides casting spells, 3.5e casters can do so many things. A 3.5e cleric, without ridiculous investments in cheese, can outfight a fighter (divine power and many other buffs), outsneak a rogue (one of the plethora of spells that give you a skill bonus, with one of them even giving you a flat +20 on any skill check), heal, summon enemies, fly, attack from a distance with various spells, debuff the enemy, buff the party etc. A fighter on the other hand can hit his enemies with a sharp stick and then.... yeah, hit again. Maybe plink away a few arrows if he packed a bow.
The problem in 3.5e is that full casters are incredibly good at what they do, and "what they do" is almost anything you can imagine doing.
Admittedly at low levels this is less of a deal. If the aforementioned first level wizard casts sleep he can trivialize an encounter, but if he fails then he's completely useless save for a handful of crossbow bolts he might fire at the enemy. At higher levels he gets so many spell slots that he's unlikely to run out of them before the fighter types run out of hp.