This is an idea that is proposed now and again in various mods so nothing quite new. In RC this is for example partially implemented for cavalry at least, with barding leading to a speed penalty. The point is simple: armor is heavy and so are big shields. Wearing such heavy objects you can not move as fast as otherwise. Therefore, all units should have their movement speed modifier calculated after how heavy their gear is, along with all other things affecting that value. It can be quite simple, like this example:
Unarmored: 0
Light armor, padded and leather: - 0,1
Light mail: - 0,2
Heavy mail, light plating: - 0,3
Partial plate: -0,4
Full plate: - 0,5
Light shield: -0,1
Medium shield: - 0,2
Large shield: - 0,3
Light weapon: 0
Medium weapon: - 0,1
Heavy or unwieldy weapon: - 0,2
The Gondorian infantry upgrade to what I would call at least partial plate and wear shields of medium size and a longsword that is of medium weight and size. They will incur a - 0,7 penalty to their movement and have a movement speed modifier of 0,93. If you prefer to have the animations look best for the heavies you could start counting down from 1.05 or so instead of 1.0.
The Muhad tribesman on the other hand wears only leather or hide armor, a light shield and carry some javelins, which are light but many and therefore might still count as a medium weapon. He only suffers a - 0,3 penalty and will outrun the clumsy Gondorian after peppering him with javelins. If one count the javelins as light, which may be sensible too considering that he can be expected to lighten the load as soon as he gets the chance, the tribesman only incur a - 0,2 penalty and the Gondorian is even more outmatched. That is, until the ammunition runs out and the half-naked barbarians must close with the by now very angry heavy metal fans.
Or send in the snaga! The poor snaga skirmisher is unarmored and carry nothing but javelins, and so only incur a - 0,1 penalty. Goblin have a racial bonus to their speed, like elves, and move at 1.1 as base. The snaga will thus move at 1.09, easily outrunning the stupid humans.
Equipment of elves, dwarves, Dunedain, Uruk-Hai and troll men of Harad would perhaps count as one category lower than what they wear since the elves are good at crafting particularly light yet strong equipment and the others are unusually strong and therefore not slowed as much by heavy burdens as others. Dunedain in this case refers to the rangers and battle-ready Dunedain in Eriador which are described as particularly hard and strong, enjoying the advantages of their Numenorian ancestry.
If this is implemented, light infantry will be able to be light infantry and not just bad infantry. I believe it would add a lot of depth to tactics and allow all sorts of troops to be useful in their own way.