Delilah is the only one upstairs. Aaron, Destiny, and Harlan are all on the ground floor. I see Charles did climb up, but he's interacting with stuff/people that are still downstairs.
Charles had entered through the window Destiny broke (presumably), took ammo from the bag he threw in earlier, and acknowledged Aaron before Aaron spoke to him. Also, where does it reference that some of them went upstairs? Delilah is the only one who went upstairs; Aaron, Harlan, and Destiny are still downstairs.