In the real world, Heresy is a Christian term for those who follow Christianity but not a major orthodox sect. Those who believe and spread Heresy are deemed Heretics. For example in the early times of Christianity the Arianist sect was deemed heresy due to the fact that it claimed that Christ was a separate entity from God and not his Son.
 Heretics in M2TW
In M2TW Heretics randomly spawn where there is Heresy (particularly if there is a lot of it) and increase the amount of Heresy that is followed by the people as a religion. Heretics are agents. You can order your holy men(Priests or Imams depending on your faction) to denounce (thereby killing) Heretics by right clicking on them while a holy man is selected. Doing this has a chance of simply not working or even converting your holy man to a Heretic. This is based on the piety of both your holy man and the enemy Heretic. You want your holy man's piety as high as possible and the enemy Heretic's piety as low as possible so as to avoid the denouncing failing or worse... your holy man becoming a Heretic.
Heretics work similarly to Witches in general. Denouncing Heretics can give your priests traits that improve piety... although if you get a trait for denouncing Heretics you can easily lose it from failing to denounce heretics too many times.