This is the situation that made me decide to start this thread. I had it paused while I typed the original post. 
It isn't the worst case of ghost-occupied towers that I've seen, but it highlights my point.
Notice the occupied tower on the left side of the screen. Then notice the closest unit to it on the right side of the screen. That tower is on the southwest corner of the west wall of a castle (Caernarvon), and the closest unit is at the centre of the south wall.
Earlier in the battle there was a tower in the centre of the east wall that was active, but the closest unit to it was on the south wall. I checked for stragglers in both cases, in case there was a single guy anywhere near the towers.