Ahh sorry, I wasn't thinking this through. 
For the sake of simplicity, think of each kingdom like a traditional JRPG kingdom. There's literally a castle, and the town.