That is a very good point, forgot about that. I think it started shortly before the events of Skyrim, which were a year before the current time. He would have needed to move down to Leyawiin in the year between then and now.