The British and French nearly did intervene in the Civil War, and that's what turned it into a war over Slavery: the Emancipation Proclaimation was released to foil any British or French intervention as they'd lose any moral high ground.
Now, according to
here, the UK had 87,000 men in Canada (20,000 regulars, 67,000 militia) and an a total of some 220,000 regular soldiers, with 100,000 based in the UK.
At the end of 1862 the Union Army had some 700,000 soldiers and the Confederates some 280,000. It should be noted that the Union Army loses some 80,000 over the next year fighting the Confederates alone.
According to
this source, the USN numbered 90 ships of which only 20 were truly warships. It had some dozen Ironclads. The Royal Navy had some
9 Ironclad Warships, which were ofcourse much larger than Monitors.
So, how does all this come into play?
Well, the Royal Navy could easily maintain control of the High Seas and could effect a blockade of the Union in that manner: by concentrating warships along trade routes they could seriously impact Union imports and exports without having to dominate the coastline. If the RN Ironclads were deployed to North America they could feasibly engage the US Coastal Ironclades, whom they massively outgunned.
The British Army could reinforce Canada with relative ease due to their control of the seas, with 20,000 soldiers present already, which could be upped easily to some 60-80,000 using only reserves from the UK. Were the UK to enter a war footing to fight the war seriously, they could easily deploy a far greater number of soldiers (for example, 250,000 soldiers were deployed in the Crimean War of 1853-56, Canada is easier to access than the Crimea) to the region.
The Civil War went on a further 3 years before the Union was able to finally pacify the Confederacy. With Confederate ports largely or completely re-opened and the Union being forced to fight a second front in Canada against several hundred thousand British soldiers (ontop of the militia), the war would get much more difficult for them. The French would likely also join the war in aid of Britain and the Confederacy, as they, like Britain, had great tension with the Union.
I do think, eventually, the Union could have been overpowered by a two-front war and a naval-blockade, but I doubt it would come to that. If the British and French became involved and the Canadian front began to see the Confederate Front stall I expect the Union would concede and allow the Confederacy to exist.