Humans today are only slightly taller and heavier, and that's due to our diet. Humans a few centuries ago would be more trim, with hard, wiry muscle, but still have roughly the same physiology. Any difference is practically negligible; it's a matter of a few inches.
Second, Vietmyke is giving us firsthand knowledge of swimming with notably heavier weights. I think most soldiers carry somewhere over a hundred pounds of gear, and most of that is placed directly on the back. A set of plate weighs roughly fifty, and is evenly distributed around the body. Fifty pounds of metal is lighter than a hundred pounds of anything else. And I just gave you video evidence that a suit of armor maintains full range of motion, so your movements aren't hampered in the slightest.
Also, armor tended to be rounded off and have mostly smooth surfaces. It'd stay relatively hydrodynamic.