If the AI has access to a super cheap and endlessly available unit, it will prioritize that, especially if your AI factions have no monetary help via scripting. Some of the ways of forcing the AI to choose a more diverse army are changing recruitment costs, upkeep and availability of units. You can still have a ladder of costs, from the cheapest and worst units to the top units, but the larger the gap between those bad units and the better units, the more the AI will prioritize them. If you balance recruitment priority, unit costs and upkeep you should see a difference. Also try changing AI behaviour in descr_strat, where you can set what the AI should prioritize when recruiting. Read this:
http://rtw.heavengames.com/rtw/mods/...at/index.shtml It's for Rome, but Med2 uses the same coding.
Overall best way is to add in AI money scripts since the AI does not manage its funds very well. A combination of this and all the above can be fine tuned to see some truly diverse and interesting armies built by the AI.