The Ambarkanta map, or perhaps sketch is a proper word hehe, I always flag as to be handled with greatest care. It's our one version of the entire world, sure, but it's a relevant hint that Tolkien did not repeat the attempt after the 1930ths, 40 years before he died.
Obvious incompatibilities between it and the world he later imagined are for example the non-existing Misty Mountains, a sea where Gondor and Mordor lies, that Harad is not at all in it's today known position and an Africa-form not really matching what the LotR-map confirm, and the existance of continents later excluded from the stories.
It's technically about as useful as many
antique or
medieval maps would prove to us in practice today, hehe. That said, I do deem it a basis for some unbound artistic freedom.
Sounds you have a fair vision of what you want to create.
You could also play with [effects of] either of the two Blue Wizards unless you used them already (or unless you want to avoid to use them, or unless you trust the opposit fate of them suggested in HoME 12);
Even if not you can add something related to the Black Arts, as told by Faramir, and to that we know Queen Berúthiel (with her cats) were up to some sorcery.
In DCI we use 'Sorcerers' for the Hill-men of the North, giving them the ability to frighten enemies and chant to improve own units morale, and something like that can be an idea for a Far Harad unit.
[edit: I do not bring in the Mouth of Sauron into this, because he learned sorcery first when he entered Sauron's service in Mordor]
Gandalf were in the South, though how far south is unclear, unlikley as far as Aragorn; check UT, the last part of the chapter "Istari" for a little discussion on that topic.
If you lack UT atm I can dig the quote up.