It's more that while Servants do the flashy fighting, mages do behind the scenes stuff, often fighting enemy Masters, because killing a Master effectively disables a Servant by denying them a mana/prana supply. And in that sense, they may do as much fighting as their Servant.
An especially smart or magically powerful Master could be a greater threat than his own Servant.
Of course, you could also be some random jackass that stumbled across a book on summoning. Your call.