Bovine M2TW Checker

[bovi]

Oh boy, that's pretty much 20 months whizzing by. Been super busy with work and no TW. I hope to remedy that now.

Edit: Oh, and my PMs tell me apparently I won an award for this tool. Many thanks for the support, even it if took me more than a year to recognize it .

[_Tartaros_]

well deserved!!

[bovi]

I think this is long enough ago to warrant a ping, but apparently I have forgot how to do that. I thought it was like @Gigantus, but that doesn't seem to work. Guess I'll need to use a PM. Thanks for reporting the problems.

The problem with the checker reading the battle_model reference is not precisely that it needs to be on the end of the line, but yes that is the symptom. The code finds the battle_model token and then picks the next token as the reference, but it neglects to remove the comma so cannot find it. I believe I've fixed that now, although I don't know just what mod you saw this in. It should now also report if the reference it points to doesn't exist, which I apparently forgot about back when.

By the way, when reading the campaign script, the comma was handled already.

"change_battle_model" is in theory already handled in the campaign script, but I would not guarantee there is no bug there. Surely that is not part of descr_strat? But perhaps in show_me scripts I guess? Could you point me to an instance where it is used but not recognized by the checker?

file look up for models

- the checker reads the 'soldier' entry and then looks for supporting faction files in the modeldb file. This leads to false positives as this is the entry that defines the animation used. Usually it's the same as the first entry in the armour_ug_models line, but occasionally those are different resulting in the mentioned false positives. I would suggest to exclude that line from referencing.
- understandably the checker has an issue when in the ownership line cultures are listed instead of factions. Can a note to that effect be made or the wording of the message be changed?

I fear I may have forgotten a bit about this, but I recall having to read the soldier entries to check for silver surfers. I don't think I have ever seen any references for soldier that were not battle models. Looking at ROTK, there are soldier references to "ZYzhongqibing", for instance. That does not exist anywhere other than in the model DB - if it was an animation reference, I would expect it to rather be defined in descr_skeleton?

Yes, I think you're right that cultures are not recognized correctly as factions. I should be able to fix that though, picking up which factions belong to which cultures and replacing any such references with its list of factions.

[Gigantus]

battle_model for characters - you have the 'preset' in descr_strat which gives a character a custom battle model and then you have the script that chnages the battle model for the leader\heir. There are also battle model definitions for the 'named character' and 'general' in descr_characters:

type            named character

actions            moving_normal, moving_quickmarch, garrison, assault, attack, besiege, entrench, ambush, diplomacy, bribe, exchange, building_fort, building_watchtower
wage_base        200
starting_action_points    80

faction            venice
dictionary        15
strat_model        southern_general
battle_model    Northern_General
battle_equip    gladius, chainmail shirt helmet and rectangular shield

soldier reference - this an entry in EDU looking up a model and then applying it's animation to the particular unit (you could use a twohander model's animation for a sword and buckler unit). Textures or models are not involved with that entry. Only the entries in the 'armour_ug_models' line are used for the actual model, eg model and textures.

Edit: just a request - is it possible to disable the texture check for ships? The models obviously never see the day of light so it's not required for those models to have textures. The keyword would the category of 'ship'.

[bovi]

Sorry about the wall of text. For clarity I have marked questions I'd like answered with bold and red.

Descr_character:

                else if(line.startsWith("battle_model"))
                {
                    String modelName = line.split(" ")[1];
                    unusedBattleModels.remove(modelName);
                    ModelDbEntry model = selectModelDbEntry(modelName);
                    if(model == null)
                        BovineM2twCheck.writeGraphicsLog(lineNumber, modelName, "DC " + lineNumber + ": Character type \"" + characterType + "\" for faction " + characterFaction + " in descr_character refers to nonexistent battle model " + modelName);
                }

So, what it does:
- Find any line that starts with battle_model and deem the next token as the model name.
- Consider this referenced so it will not be listed as unused.
- Look up the model db entry for that name and report if it does not exist.

I think this should be fine?

------

I'm not 100% sure I get you about descr_strat, but what is done is pretty much the same (although corrected from the released version of the checker!):

                if(line.startsWith("character"))
                {
                    for(int i = 0; i < tokens.length; i++)
                    {
                        if(tokens[i].equals("battle_model"))
                        {
                            String foundBattleModel = tokens[i + 1].replaceAll(",", "");
                            if(selectModelDbEntry(foundBattleModel) == null)
                                BovineM2twCheck.writeGraphicsLog(lineNumber, "", "DS references nonexistent battle model \"" + foundBattleModel + "\".");
                            else
                                unusedBattleModels.remove(foundBattleModel);
                        }
                    }
                }

- For every line starting with "character", go through the tokens and look for "battle_model" among them. When "battle_model" is found, consider the next token as the model name. However, (unlike the released version) shed any comma included in it.

The rest is basically the same as for descr_character, except it doesn't care to mark used a battle model it cannot find (which then cannot possibly be listed as unused).

It does not however look for any change_battle_model instances in descr_strat. What I fail to interpret is if you mean that can happen there too, as well as in the campaign script.

------------

campaign_script:

                if(line.startsWith("character"))
                {
                    String[] tokens = line.split(" ");
                    for(int i = 0; i < tokens.length; i++)
                    {
                        String foundBattleModel = null;
                        if(tokens[i].equals("battle_model"))
                            foundBattleModel = tokens[i + 1].replaceAll(",","");
                        if(tokens[i].equals("change_battle_model"))
                            foundBattleModel = tokens[i + 3].replaceAll(",","");
                        if(foundBattleModel != null)
                        {
                            unusedBattleModels.remove(foundBattleModel);
                            if(selectModelDbEntry(foundBattleModel) == null)
                                BovineM2twCheck.writeGraphicsLog(lineNumber, "", "CS references nonexistent battle model \"" + foundBattleModel + "\".");
                        }
                    }
                }

- For every line starting with "character", go through the tokens and look for "battle_model" OR "change_battle_model" among them. When "battle_model" is found, consider the next token as the model name except any comma. When change_battle_model is found, consider the THIRD following token as the model name except any comma.

The rest is the same as for descr_character.

I think this should be fine?

Edit: Although I suddenly got unsure if change_battle_model commands are really part of a line starting with "character"...

show_me scripts are NOT checked for battle model references. Do any such changes happen in those?

-----------

Thanks for clarifying about the soldier clause. I now changed to omit checking for textures based on this clause, but keep to only do that for armour_ug_models and officer. It will still check that the soldier reference is a valid model, and consider any such references as sufficient use that it should not be reported as unused.

Is the officer clause also not indicating required textures? Those are presently still checked for faction texture assignments, and I believe I have seen silver surfer officers in the past where the rest of the unit were properly textured, so I think they do indeed need them.

-----------

That leaves me with this list of remaining items to do, I think:
* Correctly check texture assignments where cultures are listed instead of distinct factions
* Stop checking texture assignments for ship units

Possibly more given answers to my questions above.

There is probably some stuff requested in the past I have forgot, so a prod about that would be appreciated.

[Gigantus]

battle_model entry exists only in descr_strat and descr_characters - in descr_strat it will only occur in a line starting with 'character', example from teutonic campaign below, blue the term, red the model:

descr_strat

character    Gunther von Schwarzenburg, named character, male, leader, age 45, battle_model Teutonic_Hochmeister, x 60, y 36

descr_characters - green is the faction reference

type            named character

actions            moving_normal, moving_quickmarch, garrison, assault, attack, besiege, entrench, ambush, diplomacy, bribe, exchange, building_fort, building_watchtower
wage_base        200
starting_action_points    120

faction            teutonic_order
dictionary        2
strat_model        teutonic_general
strat_model        teutonic_general                    ; heir
strat_model        teutonic_grand_master                ; leader
strat_model        northern_general_fra                ; crusading noble (french model)
strat_model        northern_general_eng                ; crusading noble (england model)
battle_model    Teutonic_General
battle_equip    gladius, chainmail shirt helmet and rectangular shield

change_battle_model is a command syntax and only present in the script, not descr_strat and there are no preceding words before this command in a line, you can therefore simply check for lines starting with that command. Example from teutonic campaign script, blue being the faction and red the model, note that 'leader' can be replaced by 'heir'.
I cannot recall that I have ever seen a show_me script that uses the syntax, given that these scripts are hardly used in a mod other then for AI scripts.

    monitor_event BecomesFactionLeader FactionType teutonic_order
            change_battle_model teutonic_order leader Teutonic_Hochmeister
        end_monitor

officer - that entry requires the same check as the armour_ug_models entry as the officer model has to have the relevant faction's texture. These are up to 3 separate 'units' (individual soldiers) attached to the main unit. Example from 1648:

type             Buergerwehr
dictionary       Buergerwehr     ; Halberd Militia
category         infantry
class            spearmen
voice_type       Heavy
banner faction   main_spear
banner holy      crusade
soldier          buergerwehr, 100, 0, 1
officer         Drummer_Helm
officer         Flagger_Hat
officer         1648_officer
attributes       sea_faring, hide_forest, can_withdraw, pike, free_upkeep_unit
formation        1.2, 1.2, 1.8, 1.8, 10, schiltrom, square
stat_health      1, 0
stat_pri         5, 2, no, 0, 0, melee, melee_blade, piercing, axe, 25, 1
.../...

[bovi]

battle_model occurs in the campaign script as well as descr_strat and descr_character, like so in ROTK:

            spawn_army
                faction     hre
                character    J-3302-ZhugeJin, named character, age 20 , x 160 , y 96, portrait A017, battle_model wenren
                traits JbAAAA0100 1 , LoyaltyStarter 1 , Lz-3302-0 1 , Jn1000 5 , Jn2000 9 , Jn3000 6 , Jn4000 4 , Jn5000 8 , JnA1000 1 , JnA2000 2 , JnA3000 2 , JnA4000 2 , JnA5000 1 , JnA6000 2 , JnA7000 6 , JnA8000 6 , JnA9000 2 , Jnxg3000 2 , B2050JnRZ-A 2 , B2060JnAM-A 2 , B2140JnSJ-A 1 , Jn-orthodox 3
                unit    JiangJun_JiangDong_Da        exp 0 armour 0 weapon_lvl 0
            end

As an aside, I fail to see why anyone would choose that human unfriendly names for traits, but I'm sure there are reasons.

I'm going to upload a new version now (it should be up if this post was posted more than 15 minutes ago). I have made a number of changes:

"soldier" clauses in EDU are no longer checked for valid faction textures, as referencing those is handled with the armour_ug_models clause instead, according to Gigantus.
Reading "battle_model" clauses in descr_strat now checks that the model exists.
Reading "battle_model" references in the campaign script now removes any comma in the reference. This makes it successfully identify it even when the reference is not at the end of the line.
Reading "change_battle_model" fixed, this is no longer attempted when part of the character command, which it never is, but rather when it is an own command, which it always is.
Expanded check between EDU and unit descriptions. Now reports if name, short description or long description are not all there as a set, and whether the description is used at all, in addition to the previous duplicate check and missing description check.
New log for file references added. The meaning is to move any reports of missing files from the others into this, but only factions' file references are there as of yet.
Ship entries in the EDU are no longer checked for texture assignments as these will not enter battle.
Unit ownership is now read also from "era" clauses. Any missing textures will be reported in duplicate for each time a faction's ownership is included across the eras.
Unit ownership assigned by culture is now read and interpreted to mean the set of factions in those cultures.

Restructured the code somewhat. This work is far from complete:
Classes are now split in folders according to their areas.
Evacuated a good chunk of the code from the program entry class and the Graphics class into the new ConfigurationLists class - endeavouring to read all the lists in one place.
Started moving reading of files into the new M2twDataReader class, storage of everything in the new DataRoot class and leaving only the logic for checks in the processing classes.
So far, reading EDU, export_units and descr_sm_factions are moved.
As part of separating responsibilities like this, slightly more details are added in reports where possible, such as which file an error report is arising from in the graphics and units log.

[Gigantus]

The trait names is some coding by the original, Chinese modders, certainly takes out intuition from the equation
And my bad - I had forgotten about scripted spawns which are essentially formatted like the character entries in descr_strat

Gave the new version a spin with RotK and it looks fine, will come back if I come across stuff with different mods.

[Gigantus]

Came across an odd one: in the EDU's ownership line the comma after a faction name was set after a space, eg slave , - this resulted in a silver surfer error even though the slave entry existed in the modelDB file. Simply removing the space removed the error.

EDU:1159: (Entry northern_captain_early_flag) Silver surfer: EDU entry "demi lancers" requires officer texture assignments for faction slave

officer          northern_captain_early_flag
ownership        england, lithuania, slave , teutonic_order, jerusalem, sweden

Can confirm that ship 'crews' are no longer flagged for silver surfers, nor are 'soldier' entries. Culture entries in the EDU's ownership line appear to be processed correctly as well.

Edit: a similar issue with a comma after the last ownership entry, the correct model exists:

EDU:12775: (Entry reysen_knights_ug2,) Missing model: EDU entry "christ knights" references nonexistent armour model "reysen_knights_ug2,"

armour_ug_models reysen_knights, reysen_knights_ug1, reysen_knights_ug2,

[bovi]

I'll look into that, thanks.

Right now I'm restructuring a lot on the way the checker works, trying to make it less of a hodgepodge of reading stuff, crossreferencing and writing errors, causing a bunch of requirements to process stuff in a particular order. I'm now trying to make it first read everything, then crossreference everything and finally writing out any findings.

This involves a lot of rewriting existing checks and I imagine there will be some mistakes, so please bear with any sudden resurgence in false reports. Possibly even previous checks suddenly no longer reporting at all, although I am trying to move all the functionality into the new structure piece by piece. There are also a number of new points being checked now that some files are read more thoroughly.

I've been testing my changes on ROTK as it runs a lot faster than on EB2's giant files, and I happened upon this case when testing that the event counters are still identified correctly:

wait 0.1 ; historic_event recruiting_start factions { england, }

This is repeated a LOT, with more than 15000 instances for "recruiting_start" and "recruiting_stop". I interpret it as doing nothing at all for 0.1 seconds? The command for the historic event is commented out by the semicolon, right, it's not a weird way to make that command happen slightly later while the rest of the script runs along?

I imagine running into a single of these waits doesn't matter, but potentially running into thousands of them could make the game slow down significantly. Why add waiting at all?

[bovi]

The reason it doesn't like the space before the comma in ownership is because it splits by ", ". This means the name of the faction is interpreted as "slave ", which doesn't exist. Now I trim away whitespace in each ownership entry.

Also on the armour_ug_models, I now cut away any extraneous commas and whitespace.

Tested the changes by introducing the situations in question to my version of ROTK. Should be alright once I release the next version.

[Gigantus]

The 'wait 0.1' issue is in RotK's recruit limit script, similar to the monster (220K lines) I wrote for EBII. It's all in 'IF' sequences which means the 'wait' will only happen if the conditions are met, which is once out of all those 15K sequences, if the player is that faction with a very specific condition - thereby not noticeable where a time delay is concerned.
The semi-colon part was supposed to be in the next line, a commented out (not processed) line. I suppose it happened when we disabled the message line later with a 'find\replace' action.

EBII has over 15K instances of 'wait' as well, most of them in script outside the recruit limit script. It's a tool used in complex monitors to allow previous conditions\commands to finish processing before the next command, especially when it comes to message display.

That said - 'wait' does not delay the processing of the script, merely the processing of the individual monitor. else this 'playtime' monitor of mine in EBII would totally mess things up:

    ; === Play Time Reminder ===
    monitor_conditions TrueCondition
        wait 3600.0                                  ; time in seconds until first reminder (one hour)
        historic_event HE_reminder_one               ; initial reminder after one hour
        if I_EventCounter HE_reminder_one > 0        ; initial reminder has fired
            set_event_counter repeat_reminder 1      ; setting repeat counter
        end_if
        if I_EventCounter repeat_reminder > 0        ; first repeat, two hours elapsed
            and I_EventCounter repeat_reminder < 2   ; first repeat, two hours elapsed
            wait 3600.0                              ; time in seconds until reminder (one hour)
            historic_event HE_reminder_two           ; repeating reminder
            inc_event_counter repeat_reminder 1
        end_if
        if I_EventCounter repeat_reminder > 1        ; second repeat, three hours elapsed
            and I_EventCounter repeat_reminder < 3   ; second repeat, three hours elapsed
            wait 3600.0                              ; time in seconds until reminder (one hour)
            historic_event HE_reminder_special       ; special reminder
            inc_event_counter repeat_reminder 1
        end_if
        if I_EventCounter repeat_reminder > 2        ; more then three hours elapsed
            wait 3600.0                              ; time in seconds until reminder (one hour)
            historic_event HE_reminder_two           ; repeating reminder
            inc_event_counter repeat_reminder 1      ; optional, for use with additional special reminders
        end_if
    end_monitor

This is unlike the 'campaign_wait' command which does delay processing of the whole script and is used very sparingly, here it is meant to allow strat map movement\actions to catch up before the next command. Very prominent in my 'movie' scripts in EBII where the script moves the map and characters around in a cut scene like manner. In RotK I have used it in faction intros, eg Cao Cao's 'intro tour' when you get to the campaign map.

Edit: EBII's script has 306K lines, RotK has 245K. Both are pretty monstrous where the sheer size is concerned due to the recruit limit script. For comparison my 1648 script has 17K lines.
What determines turn times is the number and type of monitors, example: while a FactionTurn monitor will max get tested 31 times per turn a CharacterTurn monitor can be tested several hundred times per turn, only limited by the number of characters on the map. The worst culprit is the monitor_condition - it gets rested about 20 times per second as long as the script runs.
The above is also the reason why the recruit limit script has hardly any impact on turn times - it only has 30 monitors in total all of which have limited trigger times. Compare that to the total of 3200 and 3300 monitors of RotK and EBII respectively. 1648 has only 1000.
This is the reason why scripters try to use 'IF' sequences instead of multiple monitors - also the reason for the so called 'faction counters' which allow faction related scripting in those IF sequences which otherwise would require individual monitors.

[bovi]

Thanks a lot, that was an excellent explanation.

I'll be uploading a new version in the coming minutes. Some changes from the SVN log:

Now reading more fields from descr_character.
Reading descr_skeleton now stores the line number for reporting.
Text files are now properly read as UTF-16, rather than read using the OS' default encoding and then hammered only mostly successfully into readable characters.
Started writing tool output to file instead of standard output.
Started reading buildings in a more thorough manner.
Faults in resources' file references are now reported in the file reference log.
Vocal reference files are now read using a common method.
Files being read are now in some cases reported in the tool's output, others remain silently read for now.
Consolidated storage, reading and checking of event counters, except declared counters for now. Decision, historic and general event counters are now the same but merely with flags for whether they are historic/decision counters.
References to event counters are now stored directly in the main.data.code rather than across the different entities referencing them, for common processing.
Started commenting in data classes when a particular unit of data is checked against where it references, as well as reminders for making checks where they don't exist yet.
New output file stratmap.error.log (instead of building.error.log).
Checking for unused mount models now recognizes the acceptedUnusedBattleModels list.
descr_strat_model faction references (on texture assignment) are now checked.
Trait trigger affect clauses are now checked that chance is between 1 and 100.
Traits are now checked for valid character type.

Restructuring continues:
No game files are being read in the constructor anymore, allowing rearranging order of reading so whatever I'm working on now is read first.
Moved more file reading into reader classes and putting what they read into data classes, rather than having it all crunched together in the checking logic classes.
Factions remain under the main reader class in lieu of fitting group.
Unused strat models reporting moved from graphics & units to stratmap.
Checking file system for unused files moved from Graphics to FileReferences processing.

[Gigantus]

I used RotK to run the new checker.

It looks like the tool has an issue with reading descr_cultures when it comes to EDB requirements, from stratmap log:

export_descr_buildings.txt 9: Requirement of non-existent faction/culture barbarian ; yes, we created that culture - Gig
export_descr_buildings.txt 9: Requirement of non-existent faction/culture eastern_european
export_descr_buildings.txt 9: Requirement of non-existent faction/culture northern_european
export_descr_buildings.txt 9: Requirement of non-existent faction/culture greek
export_descr_buildings.txt 9: Requirement of non-existent faction/culture southern_european

Picked up an error of using colon instead of semi colon when commenting an empty line:
descr_model_strat.txt parsing error, unrecognized directive in line 1685, ":"
descr_model_strat.txt parsing error, unrecognized directive in line 1687, ":"

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
:
; These entries are needed but do not need textures assigned to them
:
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

I checked a great number of log lines and it would appear that there is a reading\referencing issue as the listed models (full complement of file) all exist in descr_characters:
[code]descr_model_strat.txt 1646: Strat model "chinese_navy" is never used by strat characters.
descr_model_strat.txt 1353: Strat model "korea_diplomat" is never used by strat characters.[code]
Invalid factions in descr_model_strat were correctly identified.
Surplus CAS models in data\model_strat were correctly identified (I am assuming descr_sm_resources was referenced)

-- graphics_n_error_log --
Definetely a referencing error from descr_characters to descr_model_strat, the beggining of the log is basically a list of all models used in descr_characters, I am starting to wonder if there is a format error in my files somewhere:

descr_character 15: (Entry new_han_general)  DC 15: Character type "named character" for faction aragon in descr_character refers to nonexistent strat model new_han_general

Basically these 3 lines contradict the error lines from the strat map log:

descr_character 1681: (Entry heretic)  DC 1681: Character type "heretic" for faction slave in descr_character refers to nonexistent strat model heretic
descr_character 1693: (Entry witch)  DC 1693: Character type "witch" for faction slave in descr_character refers to nonexistent strat model witch
descr_character 1705: (Entry inquisitor)  DC 1705: Character type "inquisitor" for faction slave in descr_character refers to nonexistent strat model inquisitor

stratmap log:

descr_model_strat.txt 1610: Strat model "heretic" is never used by strat characters.
descr_model_strat.txt 1622: Strat model "witch" is never used by strat characters.
descr_model_strat.txt 1634: Strat model "inquisitor" is never used by strat characters.

-- filereferences.error.log --
Correctly identified reference to missing sound file:

export_descr_sounds_units_voice.txt 19644 (general): ch_neutral_023.wav under folder sounds/voice/human/localized/battle_map/ does not exist.
export_descr_sounds_units_voice.txt 20537 (heavy): ch_neutral_023.wav under folder sounds/voice/human/localized/battle_map/ does not exist.
export_descr_sounds_units_voice.txt 21364 (light): ch_neutral_023.wav under folder sounds/voice/human/localized/battle_map/ does not exist.

I am assuming this will resultin a ton of error messages if the sound files are not provided in the orignal form, eg as DAT (packed) file only. As the vast majority of mods do not provide their custom sound files a switch in the CFG will come handy here.

[bovi]

I used RotK to run the new checker.

It looks like the tool has an issue with reading descr_cultures when it comes to EDB requirements, from stratmap log:

export_descr_buildings.txt 9: Requirement of non-existent faction/culture barbarian ; yes, we created that culture - Gig
export_descr_buildings.txt 9: Requirement of non-existent faction/culture eastern_european
export_descr_buildings.txt 9: Requirement of non-existent faction/culture northern_european
export_descr_buildings.txt 9: Requirement of non-existent faction/culture greek
export_descr_buildings.txt 9: Requirement of non-existent faction/culture southern_european

Actually, I read the set of cultures from descr_sm_factions.txt, not descr_cultures.txt. In that, no factions are assigned to any of these cultures - rather, all factions are part of the middle_eastern culture. I did not check descr_cultures, so did not know they were present there. I agree that the message is unclear about what is wrong here. I think this is still not really an invalid report? I'll change it to say it could be a memberless culture though. Please let me know if I am mistaken about this.

I checked a great number of log lines and it would appear that there is a reading\referencing issue as the listed models (full complement of file) all exist in descr_characters:
[code]descr_model_strat.txt 1646: Strat model "chinese_navy" is never used by strat characters.
descr_model_strat.txt 1353: Strat model "korea_diplomat" is never used by strat characters.[code]

I'll look into this, thanks. At first glance, it DOES check references from descr_character, but it may be doing so too early, as it's happening while reading the file instead of after all reading is finished.

Surplus CAS models in data\model_strat were correctly identified (I am assuming descr_sm_resources was referenced)

That's right! The tool reads descr_model_strat.txt and descr_sm_resources.txt. It does not yet read any of the other files referencing files in the subfolders below data/model_strat (such as descr_cultures.txt, which has a ton of references in it and should be checked at some point). Despite not finding references to the files in the subfolders, the checker does not report any surplus files as they are listed in acceptedNonReferencedFiles.txt for now:

models_strat/residences
models_strat/resource
models_strat/textures
models_strat/wall
models_strat/wonders

-- graphics_n_error_log --
Definetely a referencing error from descr_characters to descr_model_strat, the beggining of the log is basically a list of all models used in descr_characters, I am starting to wonder if there is a format error in my files somewhere:

descr_character 15: (Entry new_han_general)  DC 15: Character type "named character" for faction aragon in descr_character refers to nonexistent strat model new_han_general

Basically these 3 lines contradict the error lines from the strat map log:

descr_character 1681: (Entry heretic)  DC 1681: Character type "heretic" for faction slave in descr_character refers to nonexistent strat model heretic
descr_character 1693: (Entry witch)  DC 1693: Character type "witch" for faction slave in descr_character refers to nonexistent strat model witch
descr_character 1705: (Entry inquisitor)  DC 1705: Character type "inquisitor" for faction slave in descr_character refers to nonexistent strat model inquisitor

stratmap log:

descr_model_strat.txt 1610: Strat model "heretic" is never used by strat characters.
descr_model_strat.txt 1622: Strat model "witch" is never used by strat characters.
descr_model_strat.txt 1634: Strat model "inquisitor" is never used by strat characters.

Damning evidence! I'll fix.

-- filereferences.error.log --
Correctly identified reference to missing sound file:

export_descr_sounds_units_voice.txt 19644 (general): ch_neutral_023.wav under folder sounds/voice/human/localized/battle_map/ does not exist.
export_descr_sounds_units_voice.txt 20537 (heavy): ch_neutral_023.wav under folder sounds/voice/human/localized/battle_map/ does not exist.
export_descr_sounds_units_voice.txt 21364 (light): ch_neutral_023.wav under folder sounds/voice/human/localized/battle_map/ does not exist.

I am assuming this will resultin a ton of error messages if the sound files are not provided in the orignal form, eg as DAT (packed) file only. As the vast majority of mods do not provide their custom sound files a switch in the CFG will come handy here.

Yes, there will be tons of reports if used without the individual sound files present. But this tool is meant to be used by the developers of the mod, not players. At the development stage, the sound files will surely be present as unpacked files, no? As for the lines reported, I believe it has correctly identified a missing sound file (in my ROTK version, there are present ch_neutral_001.wav, ch_neutral_002.wav and so on to 21, but not to 23.

[Gigantus]

I think you should split the look up of valid factions and valid cultures. In principle the cultures exist and are valid and therefore a condition referencing it will be valid - just not applicable if those cultures are not used. It would be equivalent to looking up factions in descr_strat and then labeling EDB entries as invalid even if the faction exists in descr_sm_factions. In other words - the reading leads to wrong assumptions as it is not the actual, most basic source of reference.
Less preferable would be to change the wording to something indicating that the culture is not used, similar to other messages of this nature. Thus making clear that it is not an error per se, but also potentially creating an endless list (like now) of log entries.

The only default sub directories of data\model_strat are 'residences' with a 'textures' subfolder and the 'textures' subdirectory. All others are custom, like RotK's 'wall' and 'wonders' (which contain referenced files from descr_sm_factions). A 'standard' custom sub directory is by the use of 'faction variants', best example the custom settlement strat models of the teutonic faction in the campaign of the same name: ...\data\models_strat\residences\faction_variants\teutonic_order\textures. This can however be ignored as the models used have to have the same name as the models listed in descr_cultures for that faction's culture (if that didn't do a scramble then nothing will ). It is possible that the dependent textures have a different name, but I don't think you are testing for those.

I think the principle of generating packed sound files is sufficiently different from animations that one can live with endless entries when checking a third party mod given the otherwise beneficial output. It would just be great if there was a simple switch, I have recently repacked about a dozen mods after running your tool on each of them (and none of them bar one had unpacked sound files), so regard this as a personal request, not an essential issue. Alternatively an automatic suppression could work, eg if there are no sub directories to be found in the data\sounds directory. Note: the look up certainly works as the missing files in RotK were correctly identified, sorry if I wasn't clear.

[bovi]

About descr_cultures. Yes, it should discern whether the culture is actually nonexistent rather than whether there are any factions in the culture. I'll have to parse descr_cultures before I can do that. By the way, why would you have a ton of requirements for cultures that are not in use?

I'm not sure I get all of what you are saying about models_strat here, but I conclude that it's not a good idea to include subfolders in checking for surplus files at all, until any files referencing stuff in them are also parsed and checked. Rather than listing those as "whatever, ignore them", rather the check should not go into subfolders at all. Once the other files ARE parsed, there can be some weirdness where files are indirectly referenced.

I see I made the wrong assumption about usage of the tool then. I was thinking the tool would be used by the developers in order to validate and correct any potential mistakes, not good samaritans like you who help out a myriad of teams. I'll make a switch for whether sound files integrity should be ignored.

------------

Another thing I'm running into now is parsing capabilities in EDB. Sometimes these include "bonus" before the value and sometimes they don't. My understanding is that some of these MUST have the keyword, some MUST NOT have the keyword, and some MAY have the keyword. According to this, when a capability may and may not have it, it should, in order to improve how it's displayed. However, it does not exhaust which capabilities are in each of these three groups. Is that listed somewhere else?

[Gigantus]

The culture requirements were 'inherited' as the mod was ported from RTW to M2TW (hence the 'barbarian' culture), basically it was vanilla RTW and then build upon - not like EBII were everything was rebuild from scratch. And then at some stage we decided that it's nonsense to make our life difficult to use a variety of culture definitions when there wasn't any difference at all. Those later fringe factions excluded.

Once you have parsed descr_cultures, descr_model_strat, descr_sm_factions and descr_sm_resources you will have parsed all CAS files that should be in the model_strat folder and it's possible sub directories. Which means if you THEN run a cross check if CAS files present are utilized you should be fine.
I also remember why I created the wall and wonders sub directories: I had one heck of a time to remember the (for me) non descriptive names for the CAS models that we used for the Chinese Wall stretching over the map and the wonders (which both were repurposed trading resources).

Spoiler for wall, wonders: 

As to the 'bonus' issue - way I understand it is that if the preceding word contains 'bonus' then it has to have that additional 'bonus' entry. And then there are the obvious exceptions like free_upkeep. Basically any 'capability' that multiplies\adds to an ability has the bonus entry before the value. I reckon you will need a list of those terms as a formula is not likely to pick up whether its required or not. For what purpose would you parse that?
I am not sure if you can get the source code - but the EDB validator does a pretty good on the EDB in general so it should have an approach how to handle it, it's Java based.

The article you link points out that while it does work without the additional bonus entry it tends to mess up display and\or effect which means a missing entry should be interpreted as unintentional omission. As to negative bonus - it only reduces already existing bonus but will never result in a negative total.

[bovi]

I guessed right about the strat model lookup, it was done too early. Additionally, there was a copy-paste error making it try to find the models among the battle models rather than the strat models. Should be fixed now.

Added a new configuration item SOUNDFILE_INTEGRITY_SUPPRESSED. It will not attempt to find any file reference in the voice files if this is set to YES.

Checking usage of files under models_strat no longer enters subfolders, for now.

---------

After parsing the building capabilities more properly, these warnings are put out for ROTK (in my version of it, at least):

export_descr_buildings.txt 3155: Capability recruits_exp_bonus expects keyword "bonus" after it: recruits_exp_bonus 3
export_descr_buildings.txt 4872: Capability cavalry_bonus expects keyword "bonus" after it: cavalry_bonus 3 requires factions { emirs, }
export_descr_buildings.txt 4893: Capability cavalry_bonus expects keyword "bonus" after it: cavalry_bonus 4 requires factions { emirs, }

I have set these capabilities to expect its value directly after its name:

wall_level
tower_level
gate_strength
recruitment_slots
farming_level
road_level
trade_fleet
mine_resource
weapon_melee_simple
weapon_melee_blade
weapon_missile_mechanical
armour
amplify_religion_level
stage_games

Meanwhile, these capabilities expect "bonus" before the value:

free_upkeep
population_health_bonus
population_growth_bonus
income_bonus
trade_base_income_bonus
happiness_bonus
law_bonus
religion_level
recruits_exp_bonus
recruits_morale_bonus
cavalry_bonus
retrain_cost_bonus

----------

The reports about the unused cultures now reads like this, until I can get descr_cultures parsed:

export_descr_buildings.txt 9: Requirement of faction/culture barbarian, which does not exist, or possibly is a culture with no member factions.
export_descr_buildings.txt 9: Requirement of faction/culture eastern_european, which does not exist, or possibly is a culture with no member factions.
export_descr_buildings.txt 9: Requirement of faction/culture northern_european, which does not exist, or possibly is a culture with no member factions.
export_descr_buildings.txt 9: Requirement of faction/culture greek, which does not exist, or possibly is a culture with no member factions.
export_descr_buildings.txt 9: Requirement of faction/culture southern_european, which does not exist, or possibly is a culture with no member factions.

[bovi]

Wow, simultaneous posting there. Note I have amended the previous post.

I'm trying to parse files more thoroughly in order to:

1. Get all the requirements stored into a data structure.

This allows me to make checks happen after parsing instead of during reading. This allows easier switching around of what happens when, which saves me a ton of time testing my changes as I can set the bit I'm changing to happen first. Additionally, it makes it a lot easier to spot any checks that could be made but aren't done yet.

2. Enabling reporting of probable mistakes. When just scanning the file for various keywords like I did before to find requirements, it would be impossible to spot any typos or other errors. Now it can report when unexpected syntax occurs.