As far as i can see on the screenshots you were on a significantly higher position than your opponent. Ballistas (at least in the game engine) are not suited for high angles and it probably tried to get into a firing position, then noticed the angle was impossible, then moved to where it had a line of fire (back) but was out of range again. I have seen this stuff happen with organ guns as well. In Vanilla, that is. I had my men paintakingly drag that damned gun all the way up to a plateau, which had a bit of a declination and it refused to fire (even when i had a LOF, when i zoomed directly to the guns) and did a weird foxtrot dance back and forth.(the ballista was doing weird stuff... moved forward, turned back, turned forward all the time)
Iīm not sure about it, but i suppose the engine doesnīt calculate the declination of the ground an artillery piece stands on, when it checks, if the target is at an suitable angle for a shot.