The small garrisons are surprisingly effective if used correctly. A minor settlement has a citadel in the centre with 3 entrances. Usually the AI attacks 2, so put your limitanei/comitatenses spears on one as they're a good defensive unit, and put your cohors/legio on the other. Hide your scout equites in woods outside the settlement, and put the exploratores by the other exit from the citadel ready to rush out (they are fast moving infantry and have a super fast movement ability).
When the enemy engage your troops at each entrance, use the exploratores to attack them in the rear on one, and the cavalry on the other. Even if you don't win (unlikely thanks to the morale shock) you will deal huge casualties since the infantry sandwich is a meat grinder and the cavalry charge is absolutely devastating, even from light scout equites. You need to make sure to get a good charge up with the cav, and if it devolves into a slow melee, pull them out and charge again.