Idea: I could move levy archers and peasants to the farm building.
It'd make sense, and help spread the units out a little. And that way, even pure economy settlements would be able to get at least some defense.
I'd also just remove the mustering hall, since that's where you get all peasant infantry.