The bandit camp isn't exactly an entire village, you all could have seen each other the entire time just by looking towards where each of you were XD

(Its probably about the size and width of a football field)