Bovine M2TW Checker

[Gigantus]

Much obliged for the 'switch'.

Edit:

It will not attempt to find any file reference in the voice files if this is set to YES.

Does this mean you only process the export_descr_unit_voice file or did you mean sound files?

[bovi]

It suppresses checking of existence for file references in these files:

export_descr_sounds_stratmap_voice.txt
export_descr_sounds_units_voice.txt
export_descr_sounds_soldier_voice.txt
export_descr_sounds_narration.txt
export_descr_sounds_advice.txt

References towards text/diplomacy_speech.txt are still checked as those are not file references.

[Gigantus]

Is it possible that your checker can't handle more then 31 faction entries in the modelDB file? Because for the life of me I can't find anything wrong with this entry, all the header and animation stuff is the same as in EBII, it's just 35 factions (SSHIP sub mod adding more entries because of changed factions):
Cannot handle the model DB syntax for this mod. Expecting an integer marking segment length, but instead got "unit_models/Mounts/Medium_Horse/mount_pony_lod1.mesh". Check your syntax around line 5!

22 serialization::archive 3 0 0 0 0 1331 0 0 
10 mount_pony 
1.12 0 0 3 0 0 
52 unit_models/Mounts/Medium_Horse/mount_pony_lod0.mesh 121
52 unit_models/Mounts/Medium_Horse/mount_pony_lod1.mesh 1225 
52 unit_models/Mounts/Medium_Horse/mount_pony_lod2.mesh 10000 
0 0 35 0 0 
7 england
69 unit_models/Mounts/Medium_Horse/textures/medium_horse_england.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
42 unit_sprites/england_Mount_Pony_sprite.spr 
8 scotland 
70 unit_models/Mounts/Medium_Horse/textures/medium_horse_scotland.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
43 unit_sprites/scotland_Mount_Pony_sprite.spr 
6 france 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_france.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
41 unit_sprites/france_Mount_Pony_sprite.spr 
3 hre 
65 unit_models/Mounts/Medium_Horse/textures/medium_horse_hre.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
38 unit_sprites/hre_Mount_Pony_sprite.spr 
7 denmark 
69 unit_models/Mounts/Medium_Horse/textures/medium_horse_denmark.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
42 unit_sprites/denmark_Mount_Pony_sprite.spr 
5 spain 
67 unit_models/Mounts/Medium_Horse/textures/medium_horse_spain.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
40 unit_sprites/spain_Mount_Pony_sprite.spr 
6 aragon 
67 unit_models/Mounts/Medium_Horse/textures/medium_horse_spain.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
40 unit_sprites/spain_Mount_Pony_sprite.spr 
8 portugal 
70 unit_models/Mounts/Medium_Horse/textures/medium_horse_portugal.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
43 unit_sprites/portugal_Mount_Pony_sprite.spr 
7 abbasid 
67 unit_models/Mounts/Medium_Horse/textures/medium_horse_egypt.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
40 unit_sprites/egypt_Mount_Pony_sprite.spr 
4 pisa 
74 unit_models/Mounts/Medium_Horse/textures/medium_horse_papal_states.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
47 unit_sprites/papal_states_Mount_Pony_sprite.spr 
6 venice 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_venice.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
41 unit_sprites/venice_Mount_Pony_sprite.spr 
12 papal_states 
74 unit_models/Mounts/Medium_Horse/textures/medium_horse_papal_states.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
47 unit_sprites/papal_states_Mount_Pony_sprite.spr 
6 sicily 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_sicily.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
41 unit_sprites/sicily_Mount_Pony_sprite.spr 
6 poland 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_poland.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
41 unit_sprites/poland_Mount_Pony_sprite.spr 
6 russia 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_russia.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
39 unit_sprites/merc_Mount_Pony_sprite.spr 
10 kievan_rus 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_kievan.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
41 unit_sprites/russia_Mount_Pony_sprite.spr 
7 hungary 
69 unit_models/Mounts/Medium_Horse/textures/medium_horse_hungary.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
42 unit_sprites/hungary_Mount_Pony_sprite.spr 
15 knights_templar 
74 unit_models/Mounts/Medium_Horse/textures/medium_horse_papal_states.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
47 unit_sprites/papal_states_Mount_Pony_sprite.spr 
9 byzantium 
71 unit_models/Mounts/Medium_Horse/textures/medium_horse_byzantium.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
44 unit_sprites/byzantium_Mount_Pony_sprite.spr 
6 zengid 
69 unit_models/Mounts/Medium_Horse/textures/medium_horse_kwarezm.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
40 unit_sprites/moors_Mount_Pony_sprite.spr 
5 moors 
67 unit_models/Mounts/Medium_Horse/textures/medium_horse_moors.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
40 unit_sprites/moors_Mount_Pony_sprite.spr 
5 egypt 
67 unit_models/Mounts/Medium_Horse/textures/medium_horse_egypt.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
40 unit_sprites/egypt_Mount_Pony_sprite.spr 
5 turks 
67 unit_models/Mounts/Medium_Horse/textures/medium_horse_turks.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
40 unit_sprites/turks_Mount_Pony_sprite.spr 
3 rum 
67 unit_models/Mounts/Medium_Horse/textures/medium_horse_turks.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
40 unit_sprites/turks_Mount_Pony_sprite.spr 
7 mongols 
69 unit_models/Mounts/Medium_Horse/textures/medium_horse_mongols.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
42 unit_sprites/mongols_Mount_Pony_sprite.spr 
7 georgia 
69 unit_models/Mounts/Medium_Horse/textures/medium_horse_mongols.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
42 unit_sprites/mongols_Mount_Pony_sprite.spr 
6 cumans 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_aztecs.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
41 unit_sprites/aztecs_Mount_Pony_sprite.spr 
5 slave 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_rebels.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
40 unit_sprites/slave_Mount_Pony_sprite.spr 
4 merc 
67 unit_models/Mounts/Medium_Horse/textures/medium_horse_mercs.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
39 unit_sprites/merc_Mount_Pony_sprite.spr 
7 ireland 
69 unit_models/Mounts/Medium_Horse/textures/medium_horse_ireland.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
42 unit_sprites/ireland_Mount_Pony_sprite.spr 
6 norway 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_norway.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
41 unit_sprites/norway_Mount_Pony_sprite.spr 
6 serbia 
76 unit_models/Mounts/Medium_Horse/textures/medium_horse_teutonic_order.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
49 unit_sprites/teutonic_order_Mount_Pony_sprite.spr 
9 jerusalem 
71 unit_models/Mounts/Medium_Horse/textures/medium_horse_jerusalem.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
44 unit_sprites/jerusalem_Mount_Pony_sprite.spr 
9 lithuania 
71 unit_models/Mounts/Medium_Horse/textures/medium_horse_lithuania.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
43 unit_sprites/timurids_Mount_Pony_sprite.spr 
7 normans 
69 unit_models/Mounts/Medium_Horse/textures/medium_horse_england.texture 
68 unit_models/Mounts/Medium_Horse/textures/medium_horse_normal.texture 
42 unit_sprites/normans_Mount_Pony_sprite.spr 0 0 0 1 0 0 
5 Horse 
8 fs_horse 0  
0 0 0 0 -1 0 0 0 0 0 0 0 0

The log checker also goes pretty nuts over regular conditions in the EDCT (nearly 500 entries) but I am guessing that's because the modeldb error as in RotK those conditions don't get flagged. Example:
Unrecognized token in EDCT 1896: hidden. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 16567: factionreligion. Typo in EDCT or fault in the checker?

Trait BiologicalSon
 Characters family
 Hidden 
 
 Level BiologicalSon
   Description BiologicalSon_desc
   EffectsDescription BiologicalSon_effects_desc
   Threshold 1 
-----------------
Trigger CoA_NewNoble_Starter
 WhenToTest CharacterComesOfAge

 Condition I_EventCounter DifficultyLevel = 4
    and not FactionReligion pagan
    and IsGeneral

 Affects NewNoble 1 Chance 100

I am going to ignore the above for the time being until I can get the checker to start without error message. Will have to figure out which factions to remove from the 'faulty' modelDB section.

Edit: nope, reducing that section to 31 factions didn't help.
Edit2: Jesus wept! It was missing spaces at the end of lines and there are literally hundreds of lines like that. Trying to get some info from withwnar how to get Notepad++ to add a SPACE at the end of each line, it got that nifty 'extended' function that I haven't got a clue about.

Here is my Xmas wish: do a little magic by adding that Space to each line and then do a simple find\replace action to remove double Spaces. And THEN check the file.
If that's too awkward because of saving and stuff can you come up with a little script that can do that? After an hour of running your checker dozens of times I have reached line 3000, only 55000 more lines...
By now I have done a find\replace action adding a space to all of the common entries, like the texture, mesh, spr and faction name lines but there simply is no way to do that with faction count numbers and unit names. It amazes me that the game acknowledges a carriage return as replacement for the missing space, how else can a modeldb like that function, considering that the original format of the file is a single line with a space between each entry part.

[Gigantus]

Managed to add a space to each line with the help from the guy here and was left with a sprite line where none was supposed to be (attachment section).
Another one where that long numerical line was preceded by a animation line
Another with a missing zero at the end of a attachment line (line 45888 of 59117)
After then I got hit with this:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Unknown Source)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
        at java.lang.AbstractStringBuilder.append(Unknown Source)
        at java.lang.StringBuilder.append(Unknown Source)
        at java.lang.StringBuilder.append(Unknown Source)
        at sources.unit.ModelDbReader.readSingleToken(ModelDbReader.java:520)
        at sources.unit.ModelDbReader.readTorch(ModelDbReader.java:398)
        at sources.unit.ModelDbReader.readModelDb(ModelDbReader.java:87)
        at sources.unit.UnitReader.readAll(UnitReader.java:30)
        at sources.M2twDataReader.readItAll(M2twDataReader.java:37)
        at main.BovineM2twCheck.read(BovineM2twCheck.java:108)
        at main.BovineM2twCheck.main(BovineM2twCheck.java:21)

The checker.ouput.log ends with this line: unit_models/battle_models.modeldb
Turns out it was a corrupted numerical sequence in the very last line of the file (dumb luck finding that one):
0 -1 0 0 0 0 0 000
instead of
0 -1 0 0 0 0 0 0

No further error messages!! The nearly 500 lines of EDCT stuff remain. they are mostly seldom used conditions, so maybe they are not in your reference file? The 'hidden' error however puzzles me because it's the only flagged instance while the it (obviously) exists in a far greater number.

[Gigantus]

The above adding of a Space was done with the fine and replace action of Notepad++ using the 'extended' search method with these entries:
Search: \n
Replace: \n
the 'replace' entry having a space in front of it

To remove the excess spaces I used the 'normal' search option with two spaces in 'search' and one space in 'replace'. I ran that twice to remove any possible multiple spaces (there were another 3800 operations), a third operation turned up 62. Never mind 1 operation on a fourth ruin.

I then run the check again and basically got back to square one: Cannot handle the model DB syntax for this mod. Expecting an integer marking segment length, but instead got "sicily". Check your syntax around line 4291!

I do quote: "for the life of me I have no idea what it is (this time)" Didn't I just pass line 45888?
Here is the section with the mentioned line marked in red. All lines have only one space at the end.

27 italian_cavalry_militia_ug1 
1 1 
77 unit_models/sandy_units/EN_Pplate_Plate/italian_cavalry_militia_ug1_lod0.mesh 12000 
5 
4 pisa 
71 unit_models/sandy_units/en_pplate_plate/textures/gs_pplate_pisa.texture 
73 unit_models/sandy_units/EN_Pplate_Plate/textures/gs_Pplate_normal.texture 
57 unit_sprites/milan_Italian_Cavalry_Militia_ug1_sprite.spr 
6 venice 
73 unit_models/sandy_units/EN_Pplate_Plate/textures/gs_Pplate_venice.texture 
73 unit_models/sandy_units/EN_Pplate_Plate/textures/gs_Pplate_normal.texture 
58 unit_sprites/venice_Italian_Cavalry_Militia_ug1_sprite.spr 
12 papal_states 
79 unit_models/sandy_units/EN_Pplate_Plate/textures/gs_Pplate_papal_states.texture 
73 unit_models/sandy_units/EN_Pplate_Plate/textures/gs_Pplate_normal.texture 
65 unit_sprites/papal_states_Italian_Cavalry_Militia_ug1_sprite.spr 
6 sicily 
73 unit_models/sandy_units/EN_Pplate_Plate/textures/gs_Pplate_sicily.texture 
73 unit_models/sandy_units/EN_Pplate_Plate/textures/gs_Pplate_normal.texture 
58 unit_sprites/sicily_Italian_Cavalry_Militia_ug1_sprite.spr 
5 slave 
73 unit_models/sandy_units/EN_Pplate_Plate/textures/gs_Pplate_rebels.texture 
73 unit_models/sandy_units/EN_Pplate_Plate/textures/gs_Pplate_normal.texture 
57 unit_sprites/slave_Italian_Cavalry_Militia_ug1_sprite.spr 
5 
4 pisa 
66 unit_models/sandy_units/AttachmentSets/gs_attachments_pisa.texture 
68 unit_models/sandy_units/AttachmentSets/gs_attachments_normal.texture 0 
6 venice 
68 unit_models/sandy_units/AttachmentSets/gs_attachments_venice.texture 
68 unit_models/sandy_units/AttachmentSets/gs_attachments_normal.texture 0 
12 papal_states 
74 unit_models/sandy_units/AttachmentSets/gs_attachments_papal_states.texture 
68 unit_models/sandy_units/AttachmentSets/gs_attachments_normal.texture 0 
6 sicily 
68 unit_models/sandy_units/AttachmentSets/gs_attachments_sicily.texture 
68 unit_models/sandy_units/AttachmentSets/gs_attachments_normal.texture 0 
5 slave 
67 unit_models/sandy_units/AttachmentSets/gs_attachments_slave.texture 
68 unit_models/sandy_units/AttachmentSets/gs_attachments_normal.texture 0 
1 
5 Horse 
13 MTW2_HR_Lance 
13 MTW2_HR_Sword 
2 
21 MTW2_HR_Lance_Primary 
14 fs_test_shield 
2 
18 MTW2_Sword_Primary 
14 fs_test_shield 
16 -0.090000004 0 0 -0.34999999 0.80000001 0.60000002

I checked the preceding model's entry in case there is weird formatting like the stuff I mentioned in the previous post, nada.

Edit: apparently it's now the turn of the line count check. If these are indeed distinctive checks then some indicative messages would be very helpful.
Before I run the checker another couple dozen times for line count errors I'll give it a shot if Gracul's checker accepts the file now - that editor provides a formatted copy of the file with correct line count numbers. Unfortunately it's known to mess up the animation formatting if it is done in an unorthodox manner so caution will be advised. Nothing a 'compare' action won't be able to pick up.

How on earth did the game ever allow this file to run?

Edit: well, Gracul didn't like the file and bugged out after the fourth model. So back to bovi....
Surprisingly there was only one more error message, but that was a real curve ball - the line count error was a dozen lines above the indicated line. It was actually a correct number but a letter was missing at the end of the sprite line, the 'r' in '.spr'.

And now there is no error anymore and the file still works in the mod, Gracul accepts the file as well. Now to enable all the log files and figure out the EDCT stuff. Did I mention that at some point the EOL system changed from Windows to UNIX? (Probably me messing things during the expanded operation). There were also TABs instead of SPACEs.

This is entering the realm of the philosophical: when is the point of 'enough is enough' reached?

[bovi]

This was a lot.

Could you upload somewhere the EDCT and model DB file you're smashing your head with, so I can go through them? Please give both the original model DB and the one you've been hammering into shape, this will hopefully allow me to give better reports than what is obviously sending you on a wild goose chase.

I'm pretty sure I didn't get ALL the different tokens you could use in EDCT, I only went through ROTK and EB2 (unless I forgot to check EB2's, but I think I did). As the mod you're holding to the fire apparently uses others, the checker is freaking out. I'll be able to expand the repertoire fairly easily, having the file in question.

Not sure just what SSHIP is. If you link me to the submod I can probably get the files I need from there.

[Jurand of Cracow]

You are welcome, @bovi! Here is the link to the mod and the descriptions what it's about.
cheers
JoC

[Gigantus]

I am having some rather odd log lines as well so it might be best if you installed the mod yourself. The 'fixed' modeldb (missed a small number of tabs because I didn't check the log after the checker eventually accepted it) can be had here.

Install process of mod:
1. install Stainless Steel
2. set to early campaign and choose Savage AI
3. start mod all the way to the start of the campaign
4. Exit to desktop
5. Install SSHIP and run once to the start of the campaign
6. Exit to desktop
7. Do your checker thingy

Edit: just saw that the SSHIP download has been updated, which will include my fixed modeldb. Here the modeldb file I struggled with.

Example odd log lines (script_error.log):

CS 29491: Event counter "aggressor" is set but never checked.
CS 29731: Event counter "mil_assistance" is set but never checked.
Event counter "weak_leader_becomes_king_info" is checked but never set.
null -1: Event counter "weak_leader_becomes_king_info" is also a declared counter.
CS 30674: Event counter "resistance" is also a declared counter.
CS 30713: Event counter "new_house_mongols" is set but never checked.

Unrecognized token (checker.output.log) - the most comprehensive way would be to enter all conditions listed in the docudemons into your list

Unrecognized token in EDCT 16567: factionreligion. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 18865: originalfactiontype. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 24766: factionbuildingexists. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 25666: charfactiontype. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 28599: factionexcommunicated. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 32180: isregioncrusadetarget. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 34922: settlementhasplague. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 35851: settlementpopulationmaxedout. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 42755: missionid. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 42764: paybackid. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 43861: numcapturedsoldiers. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 46263: i_numberofheirs. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 46831: settlementstaken. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 54521: iscrusade. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 54584: isjihad. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 54575: isregioncrusadetarget. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 54607: isregionjihadtarget. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 56262: numkilledgenerals. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 56459: percentageroutedofffield. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 59473: diplomaticstancefactions. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 59976: probabilitysuccess. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 61367: tradingexotic. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 61677: factionincome. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 62684: timesinceheresy. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 64929: diplomaticstancefromcharacter. Typo in EDCT or fault in the checker?

[bovi]

A snag. My computer ran out of hard drive space installing SS6.4. I'll need to clear out some other mods.

Edit: Done, I've installed SS, SSHIP and the .97 patch F on top. I need to probably wait until at least tomorrow before tackling it though.

[bovi]

Unrecognized token in EDCT 1896: hidden. Typo in EDCT or fault in the checker?

Here is my Xmas wish: do a little magic by adding that Space to each line and then do a simple find\replace action to remove double Spaces. And THEN check the file.
If that's too awkward because of saving and stuff can you come up with a little script that can do that? After an hour of running your checker dozens of times I have reached line 3000, only 55000 more lines...
By now I have done a find\replace action adding a space to all of the common entries, like the texture, mesh, spr and faction name lines but there simply is no way to do that with faction count numbers and unit names. It amazes me that the game acknowledges a carriage return as replacement for the missing space, how else can a modeldb like that function, considering that the original format of the file is a single line with a space between each entry part.

That "hidden" directive should already be handled, really. Curious, I'll see why it's complaining.

I believe I can do that "assume a space is at the end of the line" - by reading in all the lines at once and then operating on the memory representation after that. There may be better options though:
* I could report when a line has none or more than one space at its end, so it's easier to see why it might be going haywire in reading the file. I thought I already did, actually, but possibly that got tossed out as part of reading segment by segment instead of line by line.
* I could also consider a line break the same as a space when working through the file, like the game itself apparently must do. The parsing of that file is pretty complicated already, so I'm making no promises, but it doesn't sound impossible at least.

[Gigantus]

That "hidden" directive should already be handled, really. Curious, I'll see why it's complaining.

That was not the regular 'trait is hidden but uses non-hidden description' log entry, THAT one I disable by default because while every mod uses a dummy template for hidden text it usually doesn't fit your search parameter which you must have taken from EBII. It is immaterial what description is used (in every trait actually) as long as it is there and referenced in export_vnvs - THAT test you do already. My suggestion: remove that 'trait is hidden but uses non-hidden description' check sequence, it's superfluous.

As to the actual log entry: it was the only one, coming after a number of prior 'hidden' traits, and it was also the first line in the 'Unrecognized token' list. I am suspecting a formatting issue in the EDCT just prior to that line, my example were the two first lines of that list:

Unrecognized token in EDCT 1896: hidden. Typo in EDCT or fault in the checker?
Unrecognized token in EDCT 16567: factionreligion. Typo in EDCT or fault in the checker?

I could report when a line has none or more than one space at its end

Don't report multiple spaces while checking, that's done already in the log. Rather something like 'expected SPACE at end of line, check lines preceding line 5897'
While you are at it: another fail happens if the line count number is incorrect, any chance to have a particular message for that as well? Everything else is rather esoteric, for that you could have the generic message. Just these three messages will be enough to vastly expedite the bug hunting - my brain was getting slightly scrambled having to keep all these different possibilities in mind while scanning through the error preceding lines in the modelDB.

I could also consider a line break the same as a space when working through the file

That would be the best solution where the zero SPACE issue is concerned - I trust it will still show in the log afterward?

In all my checks I have never come across a modelDB that routinely misses SPACE at the end of the line. The odd occasion, but not hundreds, never. I suspect someone was in charge of coding the models in and wasn't aware of the formatting. And then got away with it, even with wrong line count numbers.

[bovi]

The messed up model DB you linked to does crash as is. However, before it does that, it has spouted a lot of reports to the graphics_n_units log:

model DB 4: Trailing space count is wrong: 0.
model DB 8: Trailing space count is wrong: 0.
model DB 149: Trailing space count is wrong: 2.
model DB 298: Trailing space count is wrong: 2.
model DB 299: Trailing space count is wrong: 0.
model DB 302: Trailing space count is wrong: 0.
model DB 306: Trailing space count is wrong: 0.
model DB 307: Trailing space count is wrong: 0.
model DB 310: Trailing space count is wrong: 0.
model DB 311: Trailing space count is wrong: 0.
model DB 314: Trailing space count is wrong: 0.
model DB 315: Trailing space count is wrong: 0.
model DB 318: Trailing space count is wrong: 0.
...

Considering that it says it crashes around line 5 ("Cannot handle the model DB syntax for this mod. Expecting an integer marking segment length, but instead got "unit_models/Mounts/Medium_Horse/mount_pony_lod1.mesh". Check your syntax around line 5!"), the combination would be a good indication for where to start tinkering - there is an error on line 4, and line 5 subsequently gets misinterpreted.

That said, I will still change to interpret a line break as a space if there was none before it, unless I find something else blocking that.

[bovi]

Actually I already did try to do that and forgot. Except I did it on the wrong part of the line break. It expects a line break to be Windows style with CRLF (ASCII 13,10, or "\r\n" in string literals), so converts the CR into a space if there is none before it:

if(character == '\r')
{
  //Try to handle a missing trailing space at end of line
  if(previousCharacter != null && previousCharacter != ' ')
    character = ' ';
}
if(character == '\n')
  currentModelDbLine++;

I now changed it to ignore completely the CR and instead do the conversion to space on the LF, which will work with both Windows and Unix style line breaks. Mac style will no longer work, but I don't think anyone is using that, and anyway the line counting would be borked with that anyway as is.

Still crashes on line 4291, so it's not perfect yet.

------------------------

Edit: That comes from this preceding line:

12 papal_states
79 unit_models/sandy_units/EN_Pplate_Plate/textures/gs_Pplate_papal_states.texture
73 unit_models/sandy_units/EN_Pplate_Plate/textures/gs_Pplate_normal.texture
65 unit_sprites/papal_states_Italian_Cavalry_Militia_ug1_sprite.spr
6 sicily
73 unit_models/sandy_units/EN_Pplate_Plate/textures/gs_Pplate_sicily.texture
73 unit_models/sandy_units/EN_Pplate_Plate/textures/gs_Pplate_normal.texture
58 unit_sprites/sicily_Italian_Cavalry_Militia_ug1_sprite.spr

That text is 64 characters long rather than 65, so the checker (knowing it needs to read AT LEAST 65 characters) consumes also the length marker for the next line and then fails to find the segment length marker in turn. It does report the discrepancy, but does not flush the report immediately (rather doing that every 25 reports), so this fails to be written by the time it crashes.

I have now changed to immediately flush reports to the units log, so that even if it crashes, it will be able to report its latest findings. Thus, the last output for your original model DB before crashing says:

model DB 4291: (Entry italian_cavalry_militia_ug1) Line length for faction papal_states's sprite "unit_sprites/papal_states_italian_cavalry_militia_ug1_sprite.spr 6" of length 66 says it's 65 long.
(Entry italian_cavalry_militia_ug1) Reference "unit_sprites/papal_states_italian_cavalry_militia_ug1_sprite.spr 6" does not exist.



I think this should be a good indication of what needs to be corrected.

[bovi]

Crashing somewhat later in the model DB at line 45888, reading the attachment set textures:

5 turks
66 unit_models/AttachmentSets/Final Middle Eastern_turks_diff.texture
66 unit_models/AttachmentSets/Final Middle Eastern_turks_norm.texture 0
3 rum
66 unit_models/AttachmentSets/Final Middle Eastern_turks_diff.texture
66 unit_models/AttachmentSets/Final Middle Eastern_turks_norm.texture 0
6 zengid
68 unit_models/AttachmentSets/Final Middle Eastern_kwarezm_diff.texture
66 unit_models/AttachmentSets/Final Middle Eastern_turks_norm.texture 0
7 abbasid
91 unit_models/_Units/ME_Mail_Plate_Hvy_Mplate/textures/ME_Mail_Plate_Hvy_Mplate_turks.texture
92 unit_models/_Units/ME_Mail_Plate_Hvy_Mplate/textures/ME_Mail_Plate_Hvy_Mplate_normal.texture
42 unit_sprites/turks_iqta_dar_ug1_sprite.spr

This looks like a copy paste error, where the texture assignment for the faction is copied into the attachment sets section. The checker does not expect any sprite assignment in this section, and crashes.

Edit: Another crash at line 51799, another wrong length nomination saying "pisa" is 5 long:

model DB 51800: (Entry halberd_maa) Reference to nonexistent faction "pisa 60"
model DB 51800: (Entry halberd_maa) Line length for faction pisa 60 says it's 5 long.


I'm concluding that the checker with these changes supports anything it could reasonably tackle, and gives a good enough pointer to where you need to fix stuff when it does not survive. Moving on to the model DB present in the distributed version.

[bovi]

The reaction to "hidden" is because I check that the ENTIRE line is "hidden" in order to set that flag. In this particular instance, there is a trailing space ("hidden "), so it did not recognize it. I will trim any trailing whitespace so it will successfully understand even if there are whitespace AFTER as well as BEFORE the text.

-----------

I am adding the other directives from SSHIP EDCT, quite a few religion related that are not in use in EB2 or ROTK. I notice there are both IsCrusade and IsOnCrusade, and both IsJihad and IsOnJihad. They seem to be interchangeable, what is the difference? Perhaps one is whether there EXISTS a crusade/jihad, and the other is whether this particular character has JOINED it?

-----------

Right, all the directives present in the mod are now parsed. There are probably yet more in the docudemons, but I prefer having a concrete example to test on when coding them. Even though the full extent of references are parsed, there remains a ton of checking to do on them:

//Reported when equals "BecomesFactionLeader" due to the related crashing bug
//TODO Check for valid event names
public String triggeringEvent;
//TODO Check for valid settlements
public ArrayList<SimpleReferenceRecord> referredSettlements = new ArrayList<SimpleReferenceRecord>();
//Checked for existing buildings or building levels at TraitsAncillariesScript.checkTraitBuildingReferences
public ArrayList<SimpleReferenceRecord> referredBuildings = new ArrayList<SimpleReferenceRecord>();
//TODO Check for valid attributes
public ArrayList<SimpleReferenceRecord> referredAttributes = new ArrayList<SimpleReferenceRecord>();
//Checked for existing traits at TraitsAncillariesScript.traitChecks
public ArrayList<TraitReferenceRecord> referredTraits = new ArrayList<TraitReferenceRecord>();
//TODO check for valid faction
public ArrayList<SimpleReferenceRecord> referredFactions = new ArrayList<SimpleReferenceRecord>();
//TODO check for valid culture
public ArrayList<SimpleReferenceRecord> referredCultures = new ArrayList<SimpleReferenceRecord>();
//TODO check for valid religion
public ArrayList<SimpleReferenceRecord> referredReligions = new ArrayList<SimpleReferenceRecord>();
//TODO check for valid ancillary
public ArrayList<SimpleReferenceRecord> referredAncillaries = new ArrayList<SimpleReferenceRecord>();
//TODO check for valid mission
public ArrayList<SimpleReferenceRecord> referredMissions = new ArrayList<SimpleReferenceRecord>();
//TODO check for valid mission penalty/reward setting
public ArrayList<SimpleReferenceRecord> referredMissionPaybacks = new ArrayList<SimpleReferenceRecord>();
//TODO check for valid loyalty level
public ArrayList<SimpleReferenceRecord> referredSettlementLoyaltyLevels = new ArrayList<SimpleReferenceRecord>();
//TODO check for valid diplomatic stance
public ArrayList<SimpleReferenceRecord> referredDiplomaticStances = new ArrayList<SimpleReferenceRecord>();

[bovi]

That was not the regular 'trait is hidden but uses non-hidden description' log entry, THAT one I disable by default because while every mod uses a dummy template for hidden text it usually doesn't fit your search parameter which you must have taken from EBII. It is immaterial what description is used (in every trait actually) as long as it is there and referenced in export_vnvs - THAT test you do already. My suggestion: remove that 'trait is hidden but uses non-hidden description' check sequence, it's superfluous.

I know you don't find the check useful - but it is useful for EB2 as that wants to know if traits have mismatched hidden/non-hidden flag and descriptions (a common hidden text also reduces the export_vnvs.txt file size significantly, by the way). With the program switch HIDDEN_TRAIT_DESCRIPTIONS_SUPPRESSED turning off that check, is it not already working well for other mods as well?

Just to be clear, that report of the hidden flag was as part of parsing the EDCT, not as part of crosschecking texts.

[bovi]

export_descr_character_traits 15484: Target character type "inquisitor" is not recognized. Typo?export_descr_character_traits 15530: Target character type "inquisitor" is not recognized. Typo?
export_descr_character_traits 15664: Target character type "heretic" is not recognized. Typo?
export_descr_character_traits 15789: Target character type "heretic" is not recognized. Typo?
export_descr_character_traits 15789: Target character type "inquisitor" is not recognized. Typo?
export_descr_character_traits 15814: Target character type "heretic" is not recognized. Typo?
export_descr_character_traits 15814: Target character type "inquisitor" is not recognized. Typo?
export_descr_character_traits 16265: Target character type "inquisitor" is not recognized. Typo?
export_descr_character_traits 16290: Target character type "inquisitor" is not recognized. Typo?
export_descr_character_traits 16308: Target character type "inquisitor" is not recognized. Typo?
export_descr_character_traits 16326: Target character type "inquisitor" is not recognized. Typo?
export_descr_character_traits 16338: Target character type "inquisitor" is not recognized. Typo?
export_descr_character_traits 16361: Target character type "witch" is not recognized. Typo?
export_descr_character_traits 16400: Target character type "witch" is not recognized. Typo?
export_descr_character_traits 16428: Target character type "heretic" is not recognized. Typo?
export_descr_character_traits 16453: Target character type "heretic" is not recognized. Typo?

These characters are not part of ROTK/EB2. They are now recognized as valid character types.

[bovi]

descr_engines.txt 27: (Entry huge_bombard) Model group dead references nonexistent skeleton "nulldescr_engines.txt 97: (Entry rocket_launcher) Model group dead references nonexistent skeleton "null
descr_engines.txt 166: (Entry monster_ribault) Model group dead references nonexistent skeleton "null
descr_engines.txt 234: (Entry culverin) Model group dead references nonexistent skeleton "null
descr_engines.txt 303: (Entry serpentine) Model group dead references nonexistent skeleton "null
descr_engines.txt 369: (Entry catapult) Model group dead references nonexistent skeleton "null
descr_engines.txt 432: (Entry bombard) Model group dead references nonexistent skeleton "null
descr_engines.txt 501: (Entry mortar) Model group dead references nonexistent skeleton "null
descr_engines.txt 567: (Entry basilisk) Model group dead references nonexistent skeleton "null
descr_engines.txt 634: (Entry ballista) Model group dead references nonexistent skeleton "null
descr_engines.txt 700: (Entry great_bombard) Model group dead references nonexistent skeleton "null
descr_engines.txt 767: (Entry trebuchet) Model group dead references nonexistent skeleton "null
descr_engines.txt 832: (Entry mangonel) Model group dead references nonexistent skeleton "null
descr_engines.txt 899: (Entry cannon) Model group dead references nonexistent skeleton "null
descr_engines.txt 966: (Entry ribault) Model group dead references nonexistent skeleton "null
descr_engines.txt 1029: (Entry great_cross) Model group dead references nonexistent skeleton "null
descr_engines.txt 1082: (Entry great_standart_milan) Model group dead references nonexistent skeleton "null
descr_engines.txt 1133: (Entry great_standart_venice) Model group dead references nonexistent skeleton "null
descr_engines.txt 1184: (Entry siege_tower) Model group dead references nonexistent skeleton "null
descr_engines.txt 1278: (Entry middle_tower) Model group dead references nonexistent skeleton "null
descr_engines.txt 1373: (Entry heliopolis) Model group dead references nonexistent skeleton "null
descr_engines.txt 1491: (Entry tortoise_ram) Model group dead references nonexistent skeleton "null
descr_engines.txt 1559: (Entry town_ladder) Model group dead references nonexistent skeleton "null
descr_engines.txt 1609: (Entry medium_ladder) Model group dead references nonexistent skeleton "null

These come because there is no skeleton defined for the model group:

type huge_bombardculture all
class huge_bombard
pathfinding_data none
reference_points siege_engines/siege_huge_bombard.modelReferencePoints




engine_model_group normal
engine_skeleton huge_bombard
engine_bone_map siege_engines/BoneMaps/huge_bombard.xml
engine_collision siege_engines/collision_models/siege_huge_bombard_collision.CAS
engine_mesh siege_engines/siege_huge_bombard.mesh, 40.0
engine_mesh siege_engines/lods/siege_huge_bombard_med.mesh, 80.0
engine_mesh siege_engines/lods/siege_huge_bombard_low.mesh, max




engine_model_group dying
engine_skeleton huge_bombard_dying
engine_bone_map siege_engines/BoneMaps/huge_bombard_destroy.xml
engine_collision siege_engines/collision_models/siege_huge_bombard_collision.CAS
engine_mesh siege_engines/destroyed/siege_huge_bombard_destructuction.mesh, max


engine_model_group dead
engine_collision siege_engines/collision_models/siege_huge_bombard_collision.CAS
engine_mesh siege_engines/destroyed/siege_huge_bombard_destroyed.mesh, max


engine_shadow none

I don't think I have seen that before. Are you sure you can omit the skeleton for a model group? I guess you would define no skeleton if you wanted it to disappear completely on death? Depending on the answer, I will either skip the test if there is no skeleton defined, or give a better text like "model group (name) is missing its required engine_skeleton clause".

[Jurand of Cracow]

I am adding the other directives from SSHIP EDCT, quite a few religion related that are not in use in EB2 or ROTK. I notice there are both IsCrusade and IsOnCrusade, and both IsJihad and IsOnJihad. They seem to be interchangeable, what is the difference? Perhaps one is whether there EXISTS a crusade/jihad, and the other is whether this particular character has JOINED it?

Frankly speaking, I don't know.

[bovi]

As far as I can see, the EDCT and model DB are now properly handled. There are still some parsing errors in EDB due to directives not in use by EB2/ROTK which I will look at before uploading a new version.

There are a MASSIVE amount of reports though (18199, though before I fix the EDB parsing). I assume some of the reports regard Stainless Steel itself though, rather than your submod. This will make it harder for you to sift through the list. Anyway, with such a lot, I would expect some of these reports to be false positives, so when working through them please let me know about stuff it gets wrong.