Bovine M2TW Checker

[Gigantus]

Jihad - your assumption is correct:
IsJihad - checks if a jihad is in progress
IsOnJihad - checks if the character has signed up for the jihad

'regular' hidden trait text - I tried to suggest to remove the check solely to cut on processing time. The suppression works just fine otherwise.

heretic, witch etc - their absence in RotK and EBII is due to both mods being ported from RTW where these characters don't exist. The mods do not use the default religion mechanism but rather use it to represent culture which in turn does not require these characters. This is possible by suppressing the otherwise automatic spawn of them in descr_campaign_db.

ModelDB general - being able to continue the check after encountering zero space at the end of the line is the most important issue solved here, the other errors can now be much easier identified simply by the absence of this error and the flush to the log adds to the understanding.

huge bombard - the descr_engines entry is standard, at least we have the same in 1648. None of the siege engines actually have a skeleton group for the 'dead' section, I am guessing it's an extension of the 'dying' group.

Looking forward to testing the new version of the checker!

Edit: suggestion regarding the download page: you have to open the whole page fro editing to update the download, so why not have a line at the top like 'latest version dated ....' alternatively use version numbers and edit the page title accordingly, that's my preferred method, eg for Bare Geomod.

[Gigantus]

New case: this time from Wrath of the Norsemen.

The modelDB file is error free according to the old python checker but your checker pops this. There is barely a time gap between starting the checker and the error:

Exception in thread "main" java.lang.NumberFormatException: For input string: "grants"
        at java.lang.NumberFormatException.forInputString(Unknown Source)
        at java.lang.Integer.parseInt(Unknown Source)
        at java.lang.Integer.parseInt(Unknown Source)
        at data.code.trait.AffectClause.<init>(AffectClause.java:15)
        at sources.code.TraitReader.readTraits(TraitReader.java:137)
        at sources.code.TraitReader.readAll(TraitReader.java:27)
        at sources.code.CodeReader.readAll(CodeReader.java:22)
        at sources.M2twDataReader.readItAll(M2twDataReader.java:35)
        at main.BovineM2twCheck.read(BovineM2twCheck.java:108)
        at main.BovineM2twCheck.main(BovineM2twCheck.java:21)

EOL is 'Windows (CR LF)' with UTF8 - the regular format.
Log output:

Mon Nov 30 11:23:26 IST 2020: Starting up the checker.
Initializing components.
Mon Nov 30 11:23:26 IST 2020: Constructor finished. Starting reading process.
Reading configuration files...
acceptedMissingFiles.txt
vanillaFiles.txt
acceptedNonReferencedFiles.txt
acceptedMissingMounts.txt
acceptedMissingSkeletons.txt
acceptedUnusedMounts.txt
acceptedUnusedSkeletons.txt
acceptedUnusedBattleModels.txt
acceptedUnusedStratModels.txt
genericSounds.txt
unquotedDiplomacySounds.txt
Mon Nov 30 11:23:27 IST 2020: Configuration files reading complete. Starting reading game files.
descr_sm_factions.txt
text/export_vnvs.txt
export_descr_character_traits.txt

Gracul's editor came up with a duplicated faction entry in one of the mounts but that wasn't it either. Will replace the required TXT files with vanilla files to see if that might be the issue as the 'For input string: "grants"' entry seems to suggest it's not related to the modeldb file.

Edit: Replacing the EDCT with the one from BG allowed a full processing of the checker. Now to figure out what the issue is. Introducing the 'hidden with trailing space' error shows up in the log so the file is definitely being parsed. But this is like spear fishing in the dark so I am just going to upload the EDCT.

[bovi]

This does not concern the model DB, but rather EDCT. You can see this both in the output and in the stacktrace:

Mon Nov 30 11:23:27 IST 2020: Configuration files reading complete. Starting reading game files.
descr_sm_factions.txt
text/export_vnvs.txt
export_descr_character_traits.txt


Exception in thread "main" java.lang.NumberFormatException: For input string: "grants"
at java.lang.NumberFormatException.forInputString(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at data.code.trait.AffectClause.<init>(AffectClause.java:15)
at sources.code.TraitReader.readTraits(TraitReader.java:137)
at sources.code.TraitReader.readAll(TraitReader.java:27)
at sources.code.CodeReader.readAll(CodeReader.java:22)
at sources.M2twDataReader.readItAll(M2twDataReader.java:35)
at main.BovineM2twCheck.read(BovineM2twCheck.java:108)
at main.BovineM2twCheck.main(BovineM2twCheck.java:21)


So, the last file it is trying to read is EDCT, and it crashes while reading an affects clause including "grants" somewhere. I cannot be more precise without the EDCT / whole mod to see what it's doing.

[bovi]

I have now uploaded the new version of the tool. Changes from the SVN log:

Added output message when starting to read more files.
Moved reading ancillary descriptions from checker to reader class.

Reading the Model DB, now LF is used as starting a new line rather than CR. This makes Unix style line endings work properly. It also makes Mac style line endings fail, but those are unlikely to be in use.
As reading the Model DB and other related files are crash-prone when encountering unrecognized syntax, it is more important to get all information written than having it run fast. Graphics / unit reports are therefore now flushed immediately on each report rather than every 25 reports.

Reading EDB, now recognizes more directives. Anything within plugins is now ignored completely (as opposed to complaining a lot about syntax it fails to understand). If this proves to be widely used, I can make it read it properly at a later stage.

Reading EDCT, "hidden" flag is now recognized even if there is trailing whitespace.
Reading EDCT, now recognizes more directives. There remains a lot of reference checking to be done on the data read here.

Reading descr_skeleton and descr_engine_skeleton no longer include the source path as part of the filename when reporting issues.

Checking trait character types now recognizes inquisitor, heretic and witch as valid character types.

When reporting event counters that are ALSO declared counters, even if the event counter set location is not recorded, the first location it is used is reported, to avoid saying it is located in the "null" file.

Edit: Oops, forgot about the reports on missing "dead" skeletons. I'll fix this for the next version.

[bovi]

New case: this time from Wrath of the Norsemen.

Edit: Replacing the EDCT with the one from BG allowed a full processing of the checker. Now to figure out what the issue is. Introducing the 'hidden with trailing space' error shows up in the log so the file is definitely being parsed. But this is like spear fishing in the dark so I am just going to upload the EDCT.

I tried to download the mod from ModDB but 7-zip could not read the file. I'll just check the EDCT you uploaded plonked into SSHIP.

Edit: Found it immediately.

Trigger Become_Land_Grants ;Killen_triggeri
WhenToTest CharacterTurnEnd


Condition IsGeneral
and EndedInSettlement
and Trait Chief >= 1
and Attribute Loyalty >=6


Affects Farm Grants 1 Chance 20

Reading an affects clause, I expect "Affects" <trait_name> <amount> "Chance" <chance>

In this case, there is a space in the trait name. I suggest calling the trait either Farm_Grants or FarmGrants (the point is to avoid the space). This goes also for "Warrior Retinue". As the rest of the traits appear to use no underscores, those should follow suit.

[Gigantus]

It was an issue of a space within a trait's name: 'Farm Grants'. Fixed all instances and a similar issue (retinue) after that. Have a clear run now but will update the checker before proceeding.

In hindsight I should have really checked for that 'grants' entry once I figured it was the EDCT.

After updating I am now getting this issue:

Exception in thread "main" java.lang.NumberFormatException: For input string: "1.5"
        at java.lang.NumberFormatException.forInputString(Unknown Source)
        at java.lang.Integer.parseInt(Unknown Source)
        at java.lang.Integer.parseInt(Unknown Source)
        at sources.stratmap.building.BuildingReader.readBuildingCapability(BuildingReader.java:321)
        at sources.stratmap.building.BuildingReader.readBuildings(BuildingReader.java:176)
        at sources.stratmap.StratMapReader.readAll(StratMapReader.java:31)
        at sources.M2twDataReader.readItAll(M2twDataReader.java:38)
        at main.BovineM2twCheck.read(BovineM2twCheck.java:108)
        at main.BovineM2twCheck.main(BovineM2twCheck.java:21)

I am taking a guess and will say that it's an EDB issue with an invalid multiplier value in a capabilities section? Found this entry: 'amplify_religion_level 1.5' - corrected to '1' and no further issue. I am not sure if decimal values are valid in non bonus capabilities, requires follow up.

Looking now at the new EDB parsing messages:

export_descr_buildings.txt 1561: Parsing failure, expected "requires" at the start of this section to read requirements: "city" ("castle" or "city" before "requires" is allowed)

While I haven't come across the total absence of requirements before I would assume that it will simply mean that there are no requirement, eg it's a 'free for all'. The only limitation here is that it can only be constructed in a city: "charcoal_burner city" A very rare case and the message is fairly clear - no need for further refinement

export_descr_buildings.txt 2523: Parsing failure, encountered unexpected building capability named "trade_level_bonus": trade_level_bonus bonus 2
export_descr_buildings.txt 2523: Parsing failure, expected "requires" after recruitment pool details.

valid capability - see vanilla 'caravan' building, the parsing failure apparently triggers the next message as there is no recruit_pool line

Parsing failure, encountered unexpected building capability named "gun_bonus": gun_bonus 1

valid capability - see vanilla building guild_alchemists_guild

Parsing failure, encountered unexpected building capability named "weapon_artillery_mechanical": weapon_artillery_mechanical 1

valid capability - see vanilla building guild_spearsmiths_guild

Reference to faction/culture mesoamerican, which does not exist, or possibly is a culture with no member factions.

This is a good wording, correct in this case.

Still from stratmap_error_log:

descr_model_strat.txt 70: Skeleton reference "strat_named_with_army" does not exist.

and from graphics_n_units_error_log

model DB 2: (Entry mount_pony)  Assigned non-existent mount skeleton fs_horse

The mod has no descr_skeleton file which leads to a long list of these false positives: Time for a skeleton exception list?

Here is the related message that appears in graphics_n_units_error_log:
"descr_skeleton.txt: Unable to find list of skeletons/animations (descr_skeleton.txt). This is okay only if you are using only vanilla models, else you may have given a wrong path to your mod's datafolder."
Maybe based on that those above messages could be suppressed?

[bovi]

Yes, I am expecting all capability bonuses to be an integer, not a decimal number. Is this incorrect?

I should be able to skip the requirements when the line ends, will fix. Haven't ever seen no requirements before.

Added recognized capabilities:
trade_level_bonus (expects bonus keyword)
gun_bonus (expects bonus keyword)
weapon_artillery_mechanical (simple integer)

[Gigantus]

Yes, I am expecting all capability bonuses to be an integer, not a decimal number. Is this incorrect?

In the case of non 'bonus' capabilities like this one I am not sure if decimals are valid. Would require some follow up but I would suggest to leave the parsing unchanged until proven otherwise.

I did continue editing - in case you didn't note the skeleton file issue at the bottom of the post.

Looks like I got my work cut out before this repack gets published: quite the number of missing descriptions, some model files, too. And tons of surplus files.

[bovi]

I fixed the requirements reading - it now only demands "requires" if the line is not already completely read already.

About the descr_skeleton thing, I think I'll have to make a list of vanilla skeletons as part of the configuration lists, yes. When descr_skeleton.txt does not exist, it would read that one instead for checking references there. Similar for descr_engine_skeleton.txt, presumably.

The thing is, I don't have the vanilla files, at least not on the computer I'm doing this work on. Do you have them, or even better, a list of their entries?

Edit: Oh wait, I have the skeletons.idx file, that should do the trick.

[bovi]

Looks like I got my work cut out before this repack gets published: quite the number of missing descriptions, some model files, too. And tons of surplus files.

Don't let perfection be the enemy of good. It's a ton of work to make everything match up, and any improvement is probably worth putting out there. Surplus files is definitely not worth holding up release over, at least.

[Gigantus]

I usually only do surplus file removal if I have time on my hands or the number of files is simply staggering in comparison to the total size of the mod. In the case of this mod the 'is not referenced in the Model DB' entries number slightly over 2700. Allowing for duplicate entries that's still about 15% of the total and might be the reason why the mod is requiring 5GB hard drive space,which is a tad high as it has no custom movies.
My ulterior motive is however to pack the mod into one installer: the one I am using can pack up to 2GB into one installer and the zipped size of the mod is slightly above that.
The mod appears to be build on 'the last Kingdom' as I recognize some of the naming terms, eg normans-baptized' from the time I worked on the mod with danova, that should make things a bit easier to follow up.

Edit: one for the LOLs - It appears I had the 1.5 version while the official one is 3.0 beta, certainly was a good exercise

[bovi]

I'm checking off one of the many todo items: validating that effects of traits are among valid values. As part of that, I am running into these kinds of effects I have not been looking into before:

combat_v_faction_portugal
combat_v_faction_poland
combat_v_faction_byzantium
combat_v_religion_catholic
combat_v_religion_orthodox

How do these work? Are they a soft reference to the religion / faction in question, so if you mod a new faction "myfaction" it recognizes that with effect combat_v_faction_myfaction? Or do they refer to faction through some index that just happened to match up with the factions in vanilla, so you'd have to remember the order of factions to match up when modding?

I'm thinking it is most likely a soft reference because the aztecs have their own such, and those are not part of "vanilla vanilla", only the one kingdoms variant. If I'm right, that means I need to parse these as references and check that they are correct - but can not make an exhaustive list of valid effect names.

Edit:

Another thing I ran into when trying to run the tool on EB2 again after such a long time, was that some trait triggers are now programmed using negation like:

and Trait Spt != 2

Does the game handle that? It is in Java or C# programming, but I believe the usual method of writing that in EDCT would be "and not Trait Spt = 2". My checker does not handle it like that, it only skips over "<>=" signs to find the level it is being checked for. That makes it think "!" is the level and fails to turn that into an integer. I can easily add the exclamation mark to the characters to skip, but that would only bury the problem, unless the game does cope with it.

[Gigantus]

Are they a soft reference to the religion / faction in question, so if you mod a new faction "myfaction" it recognizes that with effect combat_v_faction_myfaction?

Correct.

The use of an exclamation mark instead of 'not' is recognized by the game and used reasonably often by seasoned modders. As you noticed there are unorthodox uses\placements that otherwise wouldn't be done with 'not'.

[bovi]

Uploaded a new version. Changes:

Trait effects are now checked against a list of valid effect names. Ones referring to factions or religions are now parsed as such references, but not yet checked to be valid.
Historic events are no longer reported as never triggered, if they have a set date in descr_events.txt.
When descr_skeleton.txt is not present in the mod, a list of vanilla skeletons is now applied instead for checking valid skeleton references.
Reading descr_skeleton.txt and descr_engine_skeleton.txt extracted from checking class to the new SkeletonReader.
Reading descr_engines.txt extracted from checking class to the new EngineReader.
Reading export_descr_buildings.txt no longer requires "requires" keyword if the line is exhausted (meaning there are no requirements).
More building capabilities are now recognized while reading export_descr_buildings.
religion_level and amplify_religion_level capabilities are now read as decimal numbers rather than integers.
Reading trait conditions now handle comparisons on the form "!=".

[Gigantus]

Uh, nice.

[_Tartaros_]

i came across the combat bonus thing too, but tend to ignore it, as it was "ok".

[bovi]

Parsing descr_cultures.txt now. I'm running into these kinds of references below. The file references are easy enough to process, but I'm having a harder time with the bold parts:

fishing_village data/models_strat/residences/ebstrat_roman_fishing_village.CAS, port_eastern_level_1
port_land data/models_strat/residences/ebstrat_roman_port_01_land.CAS, port_eastern_level_2
normal data/models_strat/residences/ebstrat_roman_huge_city_aba.cas, settlement_eastern_level_6
watchtower data/models_strat/residences/ebstrat_wgreek_watchtower.CAS, watchtower_roman



I cannot find where these point to. Is this possibly where you define a name other files can reference, and it's just not in use in EB2? Which files would make such references?

-------------

Additionally, there are references to agent images:

spy spy.tga spy_info.tga spy.tga 1350 1 1
assassin assassin.tga assassin_info.tga assassin.tga 1500 1 1
diplomat diplomat.tga diplomat_info.tga diplomat.tga 1250 1 1

When I'm looking for these, apparently, these can refer to files under ui/units/<faction> and ui/unit_info/<faction>. Then why is this defined on the culture instead of the faction ? Also, can you tell me what the difference is between the first and third such file?

[Gigantus]

It's the 'base' under the strat models, see descr_aerial_map_bases:

type        port_eastern_level_1            
filename    bases/ports/ground_underlay_small.tga                    
range        1.0

No idea why the agent pictures are defined in descr_cultures and not by an executable mechanic like regular units - but my bet is firmly on 'RTW left over'. There are agent pics with the hash sign like for units in the ui folders, but they do not show and can be removed from vanilla.
No idea for the second, identical tga entry, maybe RTW guys might know more? Maybe it's for the display in messages, like priest got recruited, or priest got trait? But only in collective messages, individual messages use the info pic.

Edit: did a little test and can confirm that the second entry is for the unit pic in the recruitment panel, not clear yet what the first instance is for other then the above speculation. When selected an agent will display a portrait, so it's certainly not that.

[bovi]

I'm confused. We don't even have that descr_aerial_map_bases.txt in EB2, yet we have such references that seem clearly custom to me, like roman_watchtower. Maybe we don't use the base in any way, and just have random strings in there that aren't meant to do anything but need to be present? Anyway, that's the reason why I couldn't find those anywhere else, I only looked in EB2.

Curiously, I see that SSHIP (or possibly SS itself) does have that particular file, and watchtower_roman IS in there . Perhaps we're simply using vanilla references after all (and they're simply carried over into SS), even though I'm surprised there would be anything "roman" in the vanilla game.

Anyway, thanks for pointing me to the file. I'll mark it as a TODO item for later, so I don't forget about it entirely.

[_Tartaros_]

come across a special one in the edb

export_descr_buildings.txt 2663: Capability navy_bonus expects keyword "bonus" after it: navy_bonus 1
export_descr_buildings.txt 2681: Capability navy_bonus expects keyword "bonus" after it: navy_bonus 2

the tool (version01/12/2020) requires a "bonus" for navy_bonus like in "recruitment_cost_bonus_naval bonus 1" = "navy_bonus bonus 1"
it´s the first time this finding appears, but i think navy_bonus 1 look pretty good