Basically, the further away something is, the lesser the pull on it is. I assume we're standing in a sort of courtyard/clearing in the buildings. Basically, it made everything directly around him have a stronger pull towards himself, which did indeed make everything else have a slightly stronger pull, but by the time it reaches the buildings, the pull is still too weak. Also, he deactivated Gravity Center before using the downward force.