Very simple solution to the problem! 
Go into the 'export_descr_buildings' text file in your 'data' folder, scroll down to where it says:
This is data for the castle walls. Add in the line "population_growth_bonus bonus x" just below where it says "recruitment_slots", changing "x" to whatever you like. This will add a population growth bonus to the castle wall building.
If you do this for every level of a castle (in a decreasing increment I would suggest to balance out farming improvements), then you can make it so that all castles receive a population growth bonus.
Hopefully now you can upgrade a castle to its next stage before turn 150!