Yeah, true, it just seems annoying because when it comes to field battles, in my experience, they mostly occur near cities, so in my game the vast majority of field battles had one side or the other reinforced by a garrison (plus any potential reinforcements already there!)
But you're very right, it's just a game, and since it makes expansion more difficult, stopping the arcadey-ness of the game, I really shouldn't complain!