Bovine M2TW Checker

[Belovèse]

For EB2, most units seem to be fine except possibly 4, so that speaks in favour of it being accurate. If you're not surprised by the great number of dangly bits, I'll assume it's accurate for now. I'll make a check for surplus files in those folders using the results from the existence checks.

Actually I used your last version and it seems there are false reports for missing units info pictures in SSHIP. I looked at the "italian cavalry militia" for instance, here was the report:

export_descr_unit.txt 1135 (italian cavalry militia - Faction pisa): italian_cavalry_militia_info.tga under folder ../SSHIP_098/data/ui/unit_info/pisa/ does not exist.
export_descr_unit.txt 1135 (italian cavalry militia - Faction venice): italian_cavalry_militia_info.tga under folder ../SSHIP_098/data/ui/unit_info/venice/ does not exist.
export_descr_unit.txt 1135 (italian cavalry militia - Faction papal_states): italian_cavalry_militia_info.tga under folder ../SSHIP_098/data/ui/unit_info/papal_states/ does not exist.
export_descr_unit.txt 1135 (italian cavalry militia - Faction sicily): italian_cavalry_militia_info.tga under folder ../SSHIP_098/data/ui/unit_info/sicily/ does not exist.
export_descr_unit.txt 1135 (italian cavalry militia - Faction slave): italian_cavalry_militia_info.tga under folder ../SSHIP_098/data/ui/unit_info/slave/ does not exist.

It turns out the only folder it was missing in was the slave folder, but even after putting the tga in it your tool still reported it as missing.

Anything more I can do to help? Do you want a link to our current beta version?

EDIT: Also, second report: I have a lot of unused mount and missing mount reports that are false positive. It seem to me like the tool does not recognize spaces in the mount name, and so it searches for example it the mount is named "barded horse" in descr_mount it search for "barded" only in EDU.

[bovi]

Yeah, I expected there couldn't be that many missing ones... A download for your current dev build would be helpful, yes, as I would then have the same line numbers as you. I'll look into why it's not recognizing the files for the italian cavalry militia, and that will probably wipe out a lot of other false reports as well.

Thanks also for mentioning the mounts. I'll see what it's doing - probably it's only looking at the second token, while it should look at all the tokens until a comma or similar as there can be spaces in their names.

[Belovèse]

I'll put together a new testing version tomorrow I think, I'll send you the link by PM then.

[bovi]

I got to looking at this today. On my computer, on the version of your mod I have, these are the messages output by it:

export_descr_unit.txt 1135 (italian cavalry militia - Faction slave): italian_cavalry_militia_info.tga under folder ../../../../M2TWOtherMods/mods/ss6.3/data/ui/unit_info/slave/ does not exist.
export_descr_unit.txt 1135 (italian cavalry militia - Faction venice): #italian_cavalry_militia.tga under folder ../../../../M2TWOtherMods/mods/ss6.3/data/ui/units/venice/ does not exist.
export_descr_unit.txt 1135 (italian cavalry militia - Faction papal_states): #italian_cavalry_militia.tga under folder ../../../../M2TWOtherMods/mods/ss6.3/data/ui/units/papal_states/ does not exist.
export_descr_unit.txt 1135 (italian cavalry militia - Faction sicily): #italian_cavalry_militia.tga under folder ../../../../M2TWOtherMods/mods/ss6.3/data/ui/units/sicily/ does not exist.

So, only the slave one for the info picture, while three other factions for the unit card picture. After copying pisa's info picture into slave's folder, that report disappeared.

The three unit cards are also truly missing, as far as I can see in my version of your mod, and it's not reporting the two factions where that file is present (slave and pisa).

So, it looks correct to me for that unit at least. For better or for worse, I guess... I'm not sure what kind of changes I could have been making in the meantime that makes it better than in your version of the checker, but there should at least be a chance that it's fine for you as well once I release a new version. Will do so when I have looked at the mount references.

[bovi]

Uploaded a new version.

New checks:
All present files under data/ui/unit_info are in use for the unit detail page.
All present files under data/ui/units are in use for unit cards in the army composition panel.

Bugfixes:
Reading mount usage in EDU now reads the full line rather than only the first token of the mount name, as these can contain spaces.
When dynamically encountering a file reference starting with "data/", this part is now omitted, as the search for any files start in the data folder as is. Dynamic here means a constructed path, such as from unit ownership to necessary unit card pictures. Static file references would be straight file names encountered in various files. Static file references already got shaved off this part upon storing the reference, but dynamic references go directly to checking the reference rather than being stored first. It would be better to just not include the data bit in the first place, will probably fix that later.

Technical:
Converted a number of data storage classes to Kotlin.

Edit: Oops, did a mistaken rename of a java file that was not yet converted to Kotlin. While the compiled classes should work correctly, the sources would not have compiled. Amended now.

[Belovèse]

Thanks a lot Bovi, I'll double-check this unit cards business when I get on my comp.

Just thought of one thing: does your checker take into account the vanilla unit cards and Info's pics, if they are not present in the mod folder? If the names are right the game will look for them in the vanilla folder, so there's no need for them to be in the mod folder in this case.

[bovi]

Yes, it should do so. In general:

1. Look in the mod folder for the exact file in question.
2. Look in the mod folder for the file with an additional extension that it could have.
3. Look in the mod folder for the file with an alternative extension that it could have.
4. Check if the exact filename is contained in config/acceptedMissingFiles.txt.
5. Check if the exact filename is contained in config/vanillaFiles.txt.
6. Check if the filename with additional extension is contained in config/vanillaFiles.txt.
7. Check if the filename with alternative extension is contained in config/vanillaFiles.txt.

About additional and alternative extensions, texture files often have ".dds" tacked on their names in addition to the reference stated in the model files, and model files are typically referenced as filename.cas, but sometimes these files are actually called filename.cmi instead. The game successfully finds these files, so the checker must too.

About acceptedMissingFiles, you would add filenames here to suppress reports about them missing. This is useful if there is some piece of the game that needs to be present, but it is actually never in use. In EB2, for instance, the "peasants" unit ostensibly needs to exist for the game to run, but the unit cannot ever be created. Thus, it's fine even if the files referenced in their definition don't exist.

[Belovèse]

Thanks a lot, so if I understand correctly it doesn't check the vanilla folder himself, but rather the lists contained in those 2 files? So I'll have to add a list of the vanilla unit cards and pics in it to suppress those reports.

I also was wondering about animations, I get a lot of missing animations reports but maybe it is the same thing here. I didn't tackle it yet because I don't know anything about animations!

[bovi]

Thanks a lot, so if I understand correctly it doesn't check the vanilla folder himself, but rather the lists contained in those 2 files?

Yes.

So I'll have to add a list of the vanilla unit cards and pics in it to suppress those reports.

No. The vanilla unit cards and pictures should already be in vanillaFiles.txt, and only those missing from your mod and not part of a vanilla installation should be reported as is. If this is not true, please point out a concrete example.

I also was wondering about animations, I get a lot of missing animations reports but maybe it is the same thing here. I didn't tackle it yet because I don't know anything about animations!

Have you considered this option in the checker configuration file?

;The checker can be configured to require vanilla animations to be present in
;the mod's own animations folder. As this is the case for EB2, this is the
;default setting even if most mods can rely on vanilla animation files being
;available to the mod. Chances are that you want to uncomment the setting
;below for any other mod than EB2.
;VANILLA_ANIMATIONS_REQUIRED=NO

[Belovèse]

No. The vanilla unit cards and pictures should already be in vanillaFiles.txt, and only those missing from your mod and not part of a vanilla installation should be reported as is. If this is not true, please point out a concrete example.

Thanks, when I asked this I was not on my comp, but I saw this morning that the files are indeed well listed in vanillaFiles.txt, and I have no error to report about this!

Have you considered this option in the checker configuration file?

;The checker can be configured to require vanilla animations to be present in
;the mod's own animations folder. As this is the case for EB2, this is the
;default setting even if most mods can rely on vanilla animation files being
;available to the mod. Chances are that you want to uncomment the setting
;below for any other mod than EB2.
;VANILLA_ANIMATIONS_REQUIRED=NO

This option is already uncommitted on my config, here is the file just in case I messed something up (even if this config file is straight-forward and well documented!)

;Set DATAFOLDER to the data folder for your mod, such as C:/Games/M2TW/mods/my_mod_name/data
;C:/Games/M2TW needs to be changed to wherever your M2TW is installed to.
;my_mod_name needs to be changed to whatever your mod folder is called.
DATAFOLDER=../SSHIP_098/data

;Set STRAT to the folder containing your campaign script and descr_events.
;Probably this is the default imperial_campaign.
STRAT=world/maps/campaign/imperial_campaign

;When encountering a file (animation, sound, model etc) where it cannot see
;any usage, it gets reported as unused. You will probably want to actually
;reference it, but possibly you instead wish to remove it to reduce mod size.
;Provide commands for what to do with these below.
;Suggested variants below, not all of them will be relevant for you.
;Use this with caution, and ONLY after checking for possible typos in
;references and files that you really want to use in the mod after all!
;You can define more than one - this will cause each of them to be executed,
;in sequence. If you want to move to a backup folder, ensure that works
;properly before enabling the deletion command!
;REMOVE_UNUSED_FILES=COPY_TO unused_backup
;REMOVE_UNUSED_FILES=CONSOLE_COMMAND svn delete %DATAFOLDER%/%FILENAME%
;REMOVE_UNUSED_FILES=CONSOLE_COMMAND git rm %DATAFOLDER%/%FILENAME%
;REMOVE_UNUSED_FILES=CONSOLE_COMMAND del %DATAFOLDER%/%FILENAME%

;The checker can be configured to require vanilla animations to be present in
;the mod's own animations folder. As this is the case for EB2, this is the
;default setting even if most mods can rely on vanilla animation files being
;available to the mod. Chances are that you want to uncomment the setting
;below for any other mod than EB2.
VANILLA_ANIMATIONS_REQUIRED=NO

;The checker expects all hidden traits to use descriptions "hidden_desc" and 
;"hidden_effects_desc". Any hidden traits with specific descriptions are
;considered to be intended to be shown to the player and assigned as hidden by
;mistake. If your mod wishes to keep trait specific text even for hidden
;traits, uncomment this line below to suppress those reports.
HIDDEN_TRAIT_DESCRIPTIONS_SUPPRESSED=YES

;The checker tries to match all sound file names with the accent it is used for.
;Any file names not matching the defined accent prefixes will be reported. If
;you are not doing voice modding or otherwise don't want these reports, you can
;suppress them by uncommenting the line below.
ACCENT_PREFIXES_SUPPRESSED=YES

;The checker tries to match all referenced sound files with the file system.
;If all your sound files are packed rather than available as individual files,
;you want to suppress reports about missing files by uncommenting the line below.
;SOUNDFILE_INTEGRITY_SUPPRESSED=YES

;Some agent types may not be in use in your mod, despite them being part of
;some of your files. In this case you are probably encountering a bunch of
;reported issues that are not really relevant for you. You can enter any number
;of these agents here and their file references should not be checked. If you
;still get some reports concerning them, I may have forgot to check the config
;in some instances. Let me know in the forum.
; UNUSED_AGENT=merchant
; UNUSED_AGENT=priest
; UNUSED_AGENT=princess
; UNUSED_AGENT=inquisitor
; UNUSED_AGENT=witch
; UNUSED_AGENT=heretic

[bovi]

In that case, you'll have to bring up a concrete example if you think there's something wrong with the animation checks. There certainly could be, but it seems right to me.

[bovi]

Got your PM. No worries about the delay - we at EB2 are no strangers to stuff cluttering up plans of releases, to be sure.

[bovi]

I've got the download and have unpacked it, but could I ask what the "data XII" and "data XIII" folders are meant to be? Should I patch the data folder with their contents?

[Belovèse]

ah yes, sorry, these are part of the Kostic Chronology Costumes: it is used to swaps textures when you progress in the game years, to reflect historical evolution. Well it is not automatic, the player has to do it.
There is no need for you to do anything at first, the countents in data XII are the same that are in the regular data folder.

[bovi]

In that case I'll just delete those folders, the main data folder should be sufficient to run my tests of checks on.

[Nickolas "The Lit" Romanov]

Hey Bovi, I don't know if I ever mentioned this to you or I have forgotten, but I got it working awhile ago, I also just Made a steam guide if anyone else is having troubles with setting it up. here it is, to those who are struggling even with Bovi's help: https://steamcommunity.com/sharedfiles/filedetails/?id=2503791410

[bovi]

I don't think you mentioned it, no. I may have forgot about it if you did.

I have to say your descriptions there indicate a bit of confusion on your part about how to set it up, which points a finger at my failure to write understandable instructions. Glad you got it working in the end, but please point out confusing parts of the instructions so I can try to amend.

SET EDITOR=notepad++

set DATAFOLDER=E:\Steam\steamapps\common\Medieval II Total War\mods\banana\data
set STRAT=E:\Steam\steamapps\common\Medieval II Total War\mods\banana\data\world\maps\campaign\imperial_campaign

REM java -jar target\BovineM2twCheck.jar configuration.txt
java -cp target main.BovineM2twCheckKt configuration.txt
start "%EDITOR%" eda.error.log
start "%EDITOR%" edct.error.log
start "%EDITOR%" filereferences.error.log
start "%EDITOR%" unit.error.log
start "%EDITOR%" script.error.log
start "%EDITOR%" sounds.error.log
start "%EDITOR%" stratmap.error.log
start "%EDITOR%" sundry.error.log
start "%EDITOR%" checker.output.log

This is not correct. You are not supposed to set the data folder or the strat folder in this file anymore - it was like that back in 2018, but not now. You should point to your mod in the configuration file instead. That is at least one place where I apparently need to change the instructions, will see if I can spot where I said it.

Note that the runCheck.bat file will be changing considerably in the next release as I've pulled the Kotlin libraries to a common location used by several tools in EB2 I've converted into Kotlin by now. It currently looks like this on my computer:

SET EDITOR=notepad

if not exist lib (
mkdir lib;
copy ..\common_kotlin_library\*.* lib
)

SET LIBRARIES=lib/kotlin-reflect.jar
SET LIBRARIES=%LIBRARIES%;lib/kotlin-reflect-sources.jar
SET LIBRARIES=%LIBRARIES%;lib/kotlin-stdlib.jar
SET LIBRARIES=%LIBRARIES%;lib/kotlin-stdlib-sources.jar
SET LIBRARIES=%LIBRARIES%;lib/kotlin-stdlib-jdk7.jar
SET LIBRARIES=%LIBRARIES%;lib/kotlin-stdlib-jdk7-sources.jar
SET LIBRARIES=%LIBRARIES%;lib/kotlin-stdlib-jdk8.jar
SET LIBRARIES=%LIBRARIES%;lib/kotlin-stdlib-jdk8-sources.jar
SET LIBRARIES=%LIBRARIES%;lib/kotlin-test.jar
SET LIBRARIES=%LIBRARIES%;lib/kotlin-test-sources.jar

java -cp "target;%LIBRARIES%" main.BovineM2twCheckKt configuration.txt

start "%EDITOR%" eda.error.log
start "%EDITOR%" edct.error.log
start "%EDITOR%" filereferences.error.log
start "%EDITOR%" unit.error.log
start "%EDITOR%" script.error.log
start "%EDITOR%" sounds.error.log
start "%EDITOR%" stratmap.error.log
start "%EDITOR%" sundry.error.log
start "%EDITOR%" checker.output.log

I have marked the ONLY two locations where you should need to make any changes in it with bold and blue.

---------------

Also, it's been a while since my last update. I have some changes pretty much waiting to be released, but have been busy with some other tools for EB2 for a time. I will also have a paid gig starting now and going for some weeks, which is highly likely to make progress on this tool slow.

---------------

I've been thinking about maybe making a GUI for the tool. It's quite a bit of work to set this up and requires some redesign of the structure of the code, so it's NOT coming in the near future, but it's been on my mind.

As part of this, I could also try to make it easier to pick up - I could make it so you would not have to edit any files to run it, but rather that you told it where your mod is while running the tool, and it could save your choices for later.

The tool is getting quite big and taking a while to do some of its parts, most prominently parsing some specific files, so if you could direct it to only do certain checks and read only the files relevant for those checks, it would be an improvement. I imagine different people are using the tool to check different aspects of the mod. If I'm working on sound file integrity at the moment, for instance, I don't really want to wait for it reading and checking the model DB, traits, script and what have you.

[bovi]

I couldn't find where I still stated that you should modify runCheck.bat to give it your data/strat folders, but adjusted the instructions a bit in other respects.

I also marked as obsolete (with big red text) two of the earliest responses to calls for help getting the tool to run, that is some of the earliest posts in this thread. I think those may be where you got your misconceptions.

[Nickolas "The Lit" Romanov]

I don't think you mentioned it, no. I may have forgot about it if you did.

I have to say your descriptions there indicate a bit of confusion on your part about how to set it up, which points a finger at my failure to write understandable instructions. Glad you got it working in the end, but please point out confusing parts of the instructions so I can try to amend.


This is not correct. You are not supposed to set the data folder or the strat folder in this file anymore - it was like that back in 2018, but not now. You should point to your mod in the configuration file instead. That is at least one place where I apparently need to change the instructions, will see if I can spot where I said it.

Note that the runCheck.bat file will be changing considerably in the next release as I've pulled the Kotlin libraries to a common location used by several tools in EB2 I've converted into Kotlin by now. It currently looks like this on my computer:


I have marked the ONLY two locations where you should need to make any changes in it with bold and blue.

---------------

Also, it's been a while since my last update. I have some changes pretty much waiting to be released, but have been busy with some other tools for EB2 for a time. I will also have a paid gig starting now and going for some weeks, which is highly likely to make progress on this tool slow.

---------------

I've been thinking about maybe making a GUI for the tool. It's quite a bit of work to set this up and requires some redesign of the structure of the code, so it's NOT coming in the near future, but it's been on my mind.

As part of this, I could also try to make it easier to pick up - I could make it so you would not have to edit any files to run it, but rather that you told it where your mod is while running the tool, and it could save your choices for later.

The tool is getting quite big and taking a while to do some of its parts, most prominently parsing some specific files, so if you could direct it to only do certain checks and read only the files relevant for those checks, it would be an improvement. I imagine different people are using the tool to check different aspects of the mod. If I'm working on sound file integrity at the moment, for instance, I don't really want to wait for it reading and checking the model DB, traits, script and what have you.

Nah, your instructions aren't confusing Bovi, I said: " to those who are *still* struggling even *with* Bovi's (your) help" and even though that way might be (or is) technically outdated, it still works pretty well or atleast it works (and is good enough) for me, though i'll see if i can get this newer way to work for me, eventually. oh and the modifying the runbat was from another persons' comment here in this thread (their example didn't work for me, but I fiddled around with it and made it work). Also I wish you luck in whatever your new paid gig is.

[Gigantus]

Made a fresh install after downloading the latest version.

Some false positives:
Unrecognized token in EDCT 19314: factionscorerank. Typo in EDCT or fault in the checker? - Valid condition, correct spelling: FactionScoreRank
Unrecognized token in EDCT 21478: numcapturedcharacters. Typo in EDCT or fault in the checker? Valid condition, correct spelling: NumCapturedCharacters
Unrecognized token in EDCT 26125: targetreligion. Typo in EDCT or fault in the checker? Valid condition, correct spelling: TargetReligion

I am guessing 'mounted' is the issue here, it's that weird artillery elephant of the timurids:
export_descr_unit.txt 12977: Unrecognized directive on line: mounted_engine elephant_rocket_launcher


Still looking at the other stuff that is new