There are many strategies included for defeating siege towers etc before they get to the walls in the
Medieval Total War 2 Settlements and Siege Guide You might like to take a look. Note that you can't use ladders against a Huge City and for settlements with stone walls, ballistas are only do significant damage against their doors. If you use just a few siege devices against a city/fortress equipped with ballista towers and a reasonable garrison you won't see any of the siege devices get to the walls. Remember also that the city towers won't shoot if they don't have a friendly unit on the wall next to, or behind them.