That is the point of this idea. You don't have to attack the bandits, but if you let them remain than in the long term you will suffer. And so half stack battles are saved fo these bandits whilst the achievement battles are for direct. large scale battles against the opposing faction.