My takeaway from your testing is: what provide for "population growth from trade" are:
- sea trade even if no grain is involved (not in your testing, but my guess is: the better the buildings, the more growth).
- grain resource for the exporting settlement (but not the importing one) if there's any trade (again, from the Alavaria's observation on roads: the better the trade buildings, especially roads, the more growth).