Because the Joining actually grows weaker the longer they're apart, although those memories may never come back.