When you order infantry to charge, only some soldiers from the first line charge, while rests of the unit slowly sneaks toward he enemy. Even those in first rank do not keep integrity of the line. So , this leave only small group of soldiers fighting the enemy while rest will slowly sneak until making contact with some of the enemy warriors. This is often too late and too little. Not to mention whole integrity of the unit is in total mess. Whole idea of the charge – hitting the enemy hard with masses of your force is ridiculed by this. Not to tell that first thing the even levy units will learn is to keep integrity of the unit/line while charging/fighting.
I usually overcome this by ordering unit to run in some point behind the enemy which will make most of the unit making fighting contact. Of course, by doing this you will lose charge bonus…