What exactly do you gain from going to one of the "greatest" colleges in the world? You're guaranteed to get a quality education, but you can also be guaranteed of that very easily at other places. As many people have already described, the interest will be what becomes the greatest issue. You should also consider that you are not likely to obtain a job that can bear that kind of weight right out of college. It's common now to walk out of college and have nothing to do and nowhere to go, unless you go for more schooling and therefore build more debt. I'm not sure how entry into the physics world works or even how it works at all (besides "publish or perish"), but I imagine it can't be unlike many industries with high paying jobs with high qualifiers; it may take you a while to get a job and to start raking in the kind of money that you anticipated.
I'm an advocate of the idea that I'm not someone to listen to ever, but in the modern era I'm fairly certain that the most ancient and prestigious schools don't hold the edge that they used to.That's not to say all universities are equal, but ones like Imperial carry only a name and a price tag, though that name will be a great benefit. I'd say learning theoretical physics at an amazing college of repute is good enough,
Unless you have scholarships. Scholarships are an enormous factor. I wouldn't do it with no scholarships.You should also look at financial aid and anything else that could lighten that load (you should do that with all colleges). If you can make that debt more reasonable, it's something to seriously consider. As it currently stands, however, at $83,000 that's not worth what you'll be getting, at least not to me.
Your life, your call. I'm a future English major (or something similar), so you have greater authority over numbers than I do.