To make it a little clearer, there's not many bandits, at most a couple dozen. It's an ambush party and they're genuinely not prepared for the Iron Rose Knights to show up as demonstrated by Mister Half the Man he Used to Be.