Considering the Victory would sight her guns based off the enemy ships top mast, top deck, and hull (long range, medium range, point blank), its possible that damage could be caused (torn sail, random chap w/o a head...). The loss in velocity and resultant force wouldn't breech anyone's hull, but would carry a fear factor depending on the situation (poor merchant thought he could get away
). As to the ships rocking, this could/was an advantage! since the guns could only elevate so high before making a new gun port for itself, a rocking ship could easily gain range by the increased angle.
Without rifling, I'd say no, their accuracy at such a range would be crap. Taking into account resistance, winds, powder quality, and timing, the odds of hitting a ship at a mile out is slim at best. Not to say luck couldn't play a part!
"Effective" is of itself an ambiguous term, and not defined in any of the literature I've read, so each to his own imagination, no?
As to mortars and howitzers, unless they were using explosive shot (or exploding grape/canister shot), a round shot would have not more success at hitting a ship at range than the long gun! Since the same degrading problems apply.