Experiments with units and results

[Taiji]

Eh... with just 2 attack, the effect is minimal since they're near the lowest edge of the scale.

100% difference from 1 to 2, 50% difference from 2 to 3, but you expect the effect is minimal?

That is a bit difficult for me to understand, can you explain what you mean?

[alhoon]

100% difference from 1 to 2, 50% difference from 2 to 3, but you expect the effect is minimal?

That is a bit difficult for me to understand, can you explain what you mean?

What I mean is that when making tests in any kind of system you are better by using near-average values and avoiding outliers.

For example, when you test/calibrate a new piece of lab equipment, you run a simulation that fits well within the margin of the machine's specifications. If it's supposed to go from 1 - 1000, you test values of 300 - 700 not 2-5 or 980-1000.

In our particular case, attack 2 vs defence 15 for example is very much the same as attack 3 vs defence 15. Attack 10 vs defence 12 and attack 11 vs defence 12 should make a bigger difference.
There are also penalties to take into account. But penalties (from I have seen) don't count properly when they would drop the attack too low. I.e if you use spears you can't properly identify the penalty to attack since ... the attack is too low.

[Lü Bu]

Interesting...

[Taiji]

It seems to me that penalties of any kind are irrelevant if we're just trying to test the weapon upgrade's bonus.

In this situation you make the lab equipment suit the subject matter. You adjust the sensitivity to produce a more pronounced result.

It's effectively like using a microscope, the effect is magnified.

And I said everything else minimal, so again the increase in attack is more visible.

[alhoon]

It seems to me that penalties of any kind are irrelevant if we're just trying to test the weapon upgrade's bonus.

In this situation you make the lab equipment suit the subject matter. You adjust the sensitivity to produce a more pronounced result.

It's effectively like using a microscope, the effect is magnified.

And I said everything else minimal, so again the increase in attack is more visible.

And still, our results differ a lot. You see a change of 1 I see a change of 2.5. That's why I suggest a test like the one I did.

[Taiji]

Indeed they do differ. And I just checked again with 12 attack and 10 defence vs 10 attack and 10 defence, and I think I'd still rather have +2 attack than a weapon upgrade.

Do you test with yourself using both units? I mean one batch of tests with the player using the weapon upgrade, and another batch with the AI using the weapon upgrade.

Also do you check that things are roughly equal like that before beginning actual testing?

[alhoon]

Indeed they do differ. And I just checked again with 12 attack and 10 defence vs 10 attack and 10 defence, and I think I'd still rather have +2 attack than a weapon upgrade.

Do you test with yourself using both units? I mean one batch of tests with the player using the weapon upgrade, and another batch with the AI using the weapon upgrade.

Also do you check that things are roughly equal like that before beginning actual testing?

I do the tests in the following way:

I clone a unit, so that things are exactly the same.
I use 1 vs 1 units.
I run a few tests with team A and then a few with team B.
I let the test run till one side or the other wins.
I play in difficulty medium for battles.

Test
150 dismounted crusader knights with armor 4, defence 5, shield 5 and attack 13 vs 150 dismounted crusader knights with attack 11 + weapon upgrade.

First battle: Close Defeat for the upgraded weapons. They killed 117 and the un-upgraded killed 124. The upgraded weapons swordsmen have lost their general first (and I didn't time the charge very well).
Second battle: The upgraded weapons won. They killed 140 and the un-upgraded killed 108. If the un-upgraded have broken earlier the difference in ration wouldn't be as high.
Third battle: Upgraded won. Killed 131, unupgraded killed 113.
Fourth battle: Again, upgraded won. They killed 103 and the unupgraded killed 77.

Now, I change sides and play the un-upgraded weapons.
Fifth battle: Unupgraded lost. They killed 101 and the upgraded weapons killed 133.
Sixth battle: Unupgraded lost. They killed 124 and the upgraded killed 128. Quite close this time.
Seventh battle: Unupgraded lost again. They killed 125 and the upgraded killed 145.

See the above test? Upgraded weapons win most of the time with a small difference against the same unit with a +2 attack.

I'll test and post a few more results later, after I make some more tests.

[Taiji]

I do the tests in the following way:

I clone a unit, so that things are exactly the same.
I use 1 vs 1 units.
I run a few tests with team A and then a few with team B.
I let the test run till one side or the other wins.
I play in difficulty medium for battles.

I would question the stuff in blue.

1 vs 1 means the tested AI unit is behaving like a general. I'd avoid that.
I would suggest a larger sample size than 'a few', like at least 10-15 or so just to get a rough idea, but probably 50+ if you're going to start making a statistical analysis. The potential for results to vary is high even with identical units.
I don't see any benefit in letting the combat end naturally, the numbers on the result screen are always wrong, and the breakdown in cohesion will cause a more and more skewed analysis of the results as time goes on. Just let them fight for 30 seconds (long enough to reduce some of the impact from the charge) and count the losses.
I have no idea why you would use medium difficulty, or make a point of doing it for testing. VH is what I expect users of my work to use, so I do all my testing there.

Also, to me it's more sensible to use armor, rather than shield or defense skill, because armor is always applied and so loss of cohesion plays a lesser part. And it is no different in it's effect when applied, 1 armor is 1 defense, just like defense skill.

Looking at your results for spear, I'd suggest you check out Aradan's EDU guide for RTW. I retested since we last spoke and found it to be roughly the same situation as with RTW.

[alhoon]

I would question the stuff in blue.

Thanks for the suggestions!

1 vs 1 means the tested AI unit is behaving like a general.

I know, but with multiple units, tactics and surrounding etc play a part too instead of pure combat stats.
What I liked best was your idea for the general standing back while the non-generals fight it out! That's a good idea. The BAI I use (Lusted's) does that, so I'll take advantage.
I'll propably use peasants for generals so the AI won't put them in the fight.

would suggest a larger sample size than 'a few', like at least 10-15 or so just to get a rough idea, but probably 50+ if you're going to start making a statistical analysis. The potential for results to vary is high even with identical units.

If I start using more units, where tactical movement is important etc then yes, 20 experiments should be good. For general vs general ... yeah, 10 experiments should be done (and usually I do as many), but truth be told, I don't find experiments that interesting.

I don't see any benefit in letting the combat end naturally, the numbers on the result screen are always wrong, and the breakdown in cohesion will cause a more and more skewed analysis of the results as time goes on. Just let them fight for 30 seconds (long enough to reduce some of the impact from the charge) and count the losses.

Ehhh... for me the numbers are always right. I have used the pointer over retreating and victorious armies just before the end of a battle and the numbers remain the same... except the units in your army that heal.

The break down in cohesion is indeed skewing the results but when 9/10 times a side wins, I consider that side more powerful.
I used to do that to reduce impact from charge and 30 seconds aren't enough IMO, and I always let it go to the end to see exactly how much the results change from fight to fight, but thinking back... yes, there's no reason to sit and wait for the other side to break.

PS. the end results though in my experiments are usually very close which means the battle runs in much the same way.

Again, a good suggestion. I'll use it in some experiments.

As for cohesion, I'll increase the numbers of both sides. After all, if I break the battle in 40-60 seconds there's no reason to NOT put 250 men on each unit.

I have no idea why you would use medium difficulty, or make a point of doing it for testing. VH is what I expect users of my work to use, so I do all my testing there.

Because 1. it's the default
2. there are no hidden bonuses for AI or player

I disagree with VH in tests, since it gives bonuses to the AI = changed stats.
The stats you test are NOT the stats in the EDU!

Also, to me it's more sensible to use armor, rather than shield or defense skill, because armor is always applied and so loss of cohesion plays a lesser part.

Excellent suggestion!
I'll use increased armor and less defence and shield!

Looking at your results for spear, I'd suggest you check out Aradan's EDU guide for RTW. I retested since we last spoke and found it to be roughly the same situation as with RTW.

Hmm?
A link please? Is there a difference from what I suggest?

[Taiji]

Thanks for the suggestions!

You're welcome of course, it's a pleasure to be of assistance

I know, but with multiple units, tactics and surrounding etc play a part too instead of pure combat stats.
What I liked best was your idea for the general standing back while the non-generals fight it out! That's a good idea. The BAI I use (Lusted's) does that, so I'll take advantage.
I'll propably use peasants for generals so the AI won't put them in the fight.

Yeah, good points. I always stop combat the instant either general becomes engaged, and I hold my mouse pointer over the AI unit that is meant to be fighting to make sure it never says 'idle' or 'marching' while in combat.

If I start using more units, where tactical movement is important etc then yes, 20 experiments should be good. For general vs general ... yeah, 10 experiments should be done (and usually I do as many), but truth be told, I don't find experiments that interesting.

Hah me neither. In fact I find it really gets on my nerves

Ehhh... for me the numbers are always right. I have used the pointer over retreating and victorious armies just before the end of a battle and the numbers remain the same... except the units in your army that heal.

Oh, I see healed guys and extra casualties so often that I figured it always happens. I could be over generalising to be on the safe side.

The break down in cohesion is indeed skewing the results but when 9/10 times a side wins, I consider that side more powerful.
I used to do that to reduce impact from charge and 30 seconds aren't enough IMO, and I always let it go to the end to see exactly how much the results change from fight to fight, but thinking back... yes, there's no reason to sit and wait for the other side to break.

PS. the end results though in my experiments are usually very close which means the battle runs in much the same way.

I often just have them race to a desired percentage, just noting who gets to 30% first. It has the added bonus of being faster and so it's annoyingly repetitive for less time... Still annoying though..

As for cohesion, I'll increase the numbers of both sides. After all, if I break the battle in 40-60 seconds there's no reason to NOT put 250 men on each unit.

That seems like a good idea, I never thought of that. Thanks

Because 1. it's the default
2. there are no hidden bonuses for AI or player

I disagree with VH in tests, since it gives bonuses to the AI = changed stats.
The stats you test are NOT the stats in the EDU!

Actually they do seem to be the same stats. In fact the only thing I've noticed with AI on VH compared to other settings is that the general's zone of command is smaller.

I know RTW had +attack and MTW had +morale, but there doesn't seem to be anything stat related in M2TW's difficulty settings.

Anyway, here is Aradan's guide, and it really is still very useful in M2TW. Some stuff is wrong for M2TW, like 'lethality', but a lot of stuff seems right. And you'll probably find that it correlates with what you find in testing for the 'spear' and 'light_spear' weapon attributes.

edit:

BTW here's something I discovered which you might find interesting:

The rumor about spear being vulnerable to slashing weapons probably relates to animations and not stats at all. Spear lack certain animated responses, those they lack would be effected in response to slashing attacks. So rather than occasionally blocking they always perform a knocked back animation. This seems to be a disadvantage because that anim appears to run a little longer than a defense, in other words a unit being knocked make makes less attacks than one who is defending properly.

[nedensiz1]

I always like experiments Nice.

[Taiji]

Where you say a 'max 90%', that suggests to me that you are talking about a limit for a range of possibilities

So better stats just means a higher probability of hitting and killing, it doesn't totally dictate the situation.

[alhoon]

Where you say a 'max 90%', that suggests to me that you are talking about a limit for a range of possibilities

So better stats just means a higher probability of hitting and killing, it doesn't totally dictate the situation.

Well... that's from Aradan's equation.

[Taiji]

It didn't look that way:

Also, I think there is propably a cap to this, like max 90% to kill the enemy, or 95% or 99% or something.
I have seen peasants surviving a blow from 2 handers. Few, but I have.

That says 'I think' and 'I have seen', it doesn't say 'Aradan told me and I believed it'. Forgive me if I am mistaken

[alhoon]

My idea is the propability cap. According to Aradan's equation, for great difference is attack/defence a hit would always be a death. I think there's a cap.

[Taiji]

You're saying that there might be a cap like 90% for example, so logically making 95% impossible?

If you mean the cap exists independantly of stats or animations then I don't think so, and I've so far no reason to suspect so either. But if you mean it is affected by unit stats, animations and circumstances then yes, I agree.

[alhoon]

You're saying that there might be a cap like 90% for example, so logically making 95% impossible?

If you mean the cap exists independantly of stats or animations then I don't think so, and I've so far no reason to suspect so either. But if you mean it is affected by unit stats, animations and circumstances then yes, I agree.

I just say that rarely, you see an archer or something (defence 2 or something) surviving an attack by a dismounted feudal knight with attack 12 or something. The difference of 10 = 2,6 x base lethality chance.
Even if the base lethality chance is 0.4 instead of 0.5 like in RTW, that difference of 10 it would be a guaranteed death.

[Taiji]

Something that would definitely need covering by the equation would be animations. With M2TW compared to RTW that point is much harder to address.

Something that I think would also need to be included would be a 'random' element, and possibly at more than one location in the process. Anyone who looks at the way attacking animations play out long enough, and also understands descr_skeleton.txt, can see that they don't behave the same way each time they are presented by the same stimulus. There are exceptions, like where animations have been set up badly causing the soldier to get stuck with only one move to use, but otherwise it's obvious that visible circumstances are not fully dictating which move is used. And also obvious that they influence it, like we will see a higher proportion of combos and fatalities with a high attack vs low defense. Fatalities are a waste of time for a soldier, he could be killing someone else. Combos can see him kill 3 men in under 1 second.

And then there's hit effects, a man is hit and he:
A) Gets killed
B) Gets knocked back and bloodied
C) Just gets knocked back
D) Executes a successful block/parry/evade response

And he might not be hit! I guess I am saying, it looks tricky to judge. Difficult to test, but also difficult to think of a test that doesn't need other components testing before it starts, and on and on.... lol

[alhoon]

Something that would definitely need covering by the equation would be animations. With M2TW compared to RTW that point is much harder to address.

Something that I think would also need to be included would be a 'random' element, and possibly at more than one location in the process. Anyone who looks at the way attacking animations play out long enough, and also understands descr_skeleton.txt, can see that they don't behave the same way each time they are presented by the same stimulus. There are exceptions, like where animations have been set up badly causing the soldier to get stuck with only one move to use, but otherwise it's obvious that visible circumstances are not fully dictating which move is used. And also obvious that they influence it, like we will see a higher proportion of combos and fatalities with a high attack vs low defense. Fatalities are a waste of time for a soldier, he could be killing someone else. Combos can see him kill 3 men in under 1 second.

And then there's hit effects, a man is hit and he:
A) Gets killed
B) Gets knocked back and bloodied
C) Just gets knocked back
D) Executes a successful block/parry/evade response

And he might not be hit! I guess I am saying, it looks tricky to judge. Difficult to test, but also difficult to think of a test that doesn't need other components testing before it starts, and on and on.... lol

There's always the rule of the great numbers. And let's not forget, a battle involves large numbers.

I believe we could have some battles and experiments with different animations and check the differences.
Like you said in spears, the animation thing is something like -3 attack or something.