But perhaps they left in the evening instead of middle of the day? 

After all, the train was noted as going straight to the castle instead of Hogsmeade Station. As well as that they took boats previously instead of only First Years taking the boats and the rest of the students taking the thestral drawn carriages.