It's only 50 or so pounds. Modern military forces train to swim with heavier weights on their back, instead of being distributed evenly. A man wearing armor can fight with dexterity equal to an unarmored foe.

For reference, this video has a clip showing a man sprinting in an accurate recreation of 15th century plate.
http://youtu.be/NqC_squo6X4?t=34m29s
Immediately afterwards, it shows a short video of two knights fighting on foot with polearms, further demonstrating the speed and mobility available.