he didn't actually get defeated
After everybody else left and all, Xanxus went to ask the Plegian King (his uncle in relativity) to grant the Ylissean Army permission to go into their land and search for the Shepherds (of course, not intending on mentioning this), but the King already knew about Ylisse purging the Shepherds and tried to stop Xanxus there. The King would've died until someone intervened, and I'm gonna post the continuation at some point. The short battle did end in a draw in Xanxus' favor, but could now spark a conflict between Ylisse and Plegia.