You might not want anything to do with us, but that doesn't mean Sarel needs to reflect that. Surely even he would have some tactical sense. The Bandit's single entrance makes it much easier for them to survive an attack for the same reason fortresses do in real life; they can stay put and life off their supplies for much longer than the attackers, who has to make some sort of lea way before they get tired. Sure it makes escaping harder, but unless someone really puts the efforts into blockading their little cove, they don't have to worry about a long drawn out siege. There's only one way to attack them, so the bandits can focus all their efforts into defending it. We just have to hope they're as incompetent as we want them to be.