Men-at-arms who wore "light armor" usually did so because they could not afford a full set - it was costly and usually nobles could only afford them. In the latter middle ages, most infantry had at the least a helmet, some with a breastplate, some with a bit more. 

As Grave said, armor is a liability in hot climates - a factor which greatly favored the Muslim armies during the Middle-East Crusades.

There are plenty of sources out there on medieval warfare, and it doesn't take a lot to put together a coherent picture of what it was like. Course, there are differences depending on what era you're looking at. A knight in the 12th century is very different from a knight in the 14th century.