That would be what i was getting at. It would be strange if another faction could get all the troops of their allies. At most i would suggest the militia level troops maybe the next level? So for taking over Gondorian settlements would mean you would get the militia archers, swords and maybe the heavier spearmen. Another nice addition could be the condition that you cannot be at war with Gondor in order to recruit the units Wouldn't be hard to do.
Another idea would be an increase in unrest if you do go to war with them