There are eight of them there now, right? If five leave, plus the leader, all that would be left would be two. Though, I don't know that I would want to take the risk of trying to get the carriage through before they get back. I suppose all of us involved need to make a plan.
There are more in the cave, but they are ghouls who don't work well during the day. Think the pale guard, but much more feral. Not that Ssarak or crew know about them, but they're there. As for the away crew, the robed man is only going to take three with him, not including the big guy. So that'll leave four bandits in camp with four more heading south towards the bridge you all came from (That robed man at the bridge was named Zarak). There are three feral ghouls in the ravine with the other robed mage, and another three somewhere to the north who are out hunting. In total, there are potentially 15 opponents to fight, not that you have to all fight them at once, but that's the body count.