TWTECV: TW Text Editor, Cleaner & Validator v1.20

[Withwnar]

CRITICAL UPDATE

WARNING: versions prior to v1.9 have a serious flaw: semicolon ";" characters are treated as comment characters. That character and any that follow it in that line are ignored by the tool and therefore discarded when the tool updates the file. My sincere apologies if this has affected you. This has been fixed as of v1.9.

TWTECV: TW Text Editor, Cleaner & Validator v1.20

M2TW/Kingdoms
Requires: Java 6 (1.6) or later.

This tool provides functions to edit and clean up (restructure and reformat) various M2TW text files and to validate some aspects of them. The files are:

• export_buildings.txt (EB)
• export_units.txt (EU)
• export_vnvs.txt (VNV)
• export_ancillaries.txt (EA)
• historic_events.txt (HE)

..and also, optionally, their associated enum files (e.g. export_descr_buildings_enums.txt).
PLUS: some image tools.

NOTE: it is not an EDB/EDU/EDCT/EDA editor. It does provide some special validation but not of these files themselves.

Features:

• A graphical user interface for viewing and editing text entries. For buildings: the ability to copy entries from one culture to another or remove them from a culture.
• Orphan text: text entries that are not associated with buildings/units/traits/ancillaries. The tool lets you select which ones to delete.
• Filtering options to display only problem entries: missing names, descriptions & incorrect effects descriptions.
• Usage check (validation): search various files (commands, conditions, etc.) for usages of buildings/units/traits/ancillaries/HEs that do not exist, plus report any unused units/traits/ancillaries/HEs: ones that have no usage in those files. HE also reports missing images (culture aware) and videos.
• Reformatting options to help reduce text file size and increase readability.
• Buildings: auto-reduce function to restructure a building's culture entries to reduce duplication of text.
• Traits/Ancillaries: effects description auto-builder.
• HEs: useful functions for adding, renaming & removing entries, duplicating entries and copying images.
• Effects Description Config: define the text, format and other parameters of each Effect type to facilitate effects description auto-build and checking for descriptions that do not match the trait's/anc's actual effects.
• Image Tools
  
  • Portrait Viewer/Replicator: view portraits, duplicate to extend pool counts, create dead portraits.
  • Building Image Replicator: quickly duplicate, name and position in the folder tree building image(s) for one or multiple cultures.
  • Missing Shadows: find and optionally autofix images that have a missing shadow effect in their alpha channel.

Details are in the manual which may be downloaded separately if you wish to read it without downloading the tool. (The manual is also included in the tool.)

For example, using this tool to cleanup EB/EU on TATW 3.2 the line counts went from...

EB: 15,777
EU: 6,271

...to...

EB: 3,039
EU: 1,419

That cleanup included: all keys/values on same line, remove all blank lines and duplicated comments, Auto-Reduce all buildings, remove all orphan EB/EU entries. The result is much shorter, consistently formatted and, therefore, easier to work with.

Download:
EXE - Windows application (this is not an installer)
JAR - for other operating systems or an alternative to using the EXE for Windows
Manual

Installation:
Just unzip. Into the mod's folder is most convenient...

The tool lives in its own "TWTECV" folder which also contains some other files that the tool requires.
You may place this folder anywhere on your computer.
...
When the tool starts up it will auto-read the relevant TW files if it can find them. To find them the TWTECV folder must be under the folder that contains the unpacked files. e.g. The M2TW installation folder or, for a mod, the mod folder. NOTE: the tool cannot read packed files.

Change Log

Spoiler Alert, click show to read: 

v1.20

• Units/HEs: images no longer shrink when container too small to fully show them.
• Saving CombatV config now also saves the "gameName". Was getting "CombatV might have changed" message every time I started the tool for a mod, if a faction game name had changed in that mod.
• No more flashing "there are errors" - was a CPU killer. Instead the Errors tab title is just coloured red.
• Anc image display ignores alpha channel (as game does).
• Fixed a window refresh issue: some images weren't always displaying (e.g. HEs, units).
• Added AntiTraits to Traits Usage Checker. A trait is now counted as "accessed" if it is an AntiTrait of anything.

v1.15 .. v1.19 - personal versions

v1.14

• Portrait Viewer/Replicator - added.
• Unit images: now shows both info and card images. Images displayed via a background thread: faster to change the current unit selection (no longer waits for images to be displayed).
• Units: added filters for missing info and card images. Added image checks to Usage Check.
• Units: displays mercenary_unit, general_unit and no_custom attributes.
• Effects Config: added option to sort alphabetically on effect description (instead of effect order in EDCT/EDA).
• Usage Check: Historic Events: comments were not being ignored.
• More helpful error description for illegal colour values in descr_sm_factions.txt.
• Application errors now written to a log file.

v1.10 .. v1.13 - personal versions

v1.9

• Critical Bugfix: semicolons were being treated as comment characters.
• Can also rebuild enum files.

v1.8.1

• HE: description field labeled as "Body" instead of "Desc".
• HE: Usage Check: ignores EMERGENT_FACTION event type.
• HE: Usage Check: custom campaigns incorrectly listed descr_strat.txt instead of descr_events.txt (just a display mistake: descr_events.txt was the file being checked).
• "Custom Campaign" renamed to "Custom/Provincial Campaign".
• More info added to missing custom campaign and historic battle folder warning messages.
• Now reports missing advisor scripts.

v1.8

• Historic events: new.
• Usage Checks can now also check any advisor scripts.
• Text editor popup window now remembers last-used size and position. Text that should not contain line breaks (e.g. titles) now use a single-line text editor popup window.

v1.7

• Traits/ancs now have the option to be excluded from effects description "auto-build all".
• Now checks for these issues upon source data read: non-unique trait level names, same text key assigned to multiple traits/ancs.
• Effects Description Walkthrough added to manual.

v1.6.2

• Building Image Replicator - new.
• Missing Shadows - new.
• Added files to usage checks: descr_faction_standing.txt & export_descr_advice.txt.
• BUGFIX: usage checker would break app if no custom campaigns present.
• BUGFIX: if no source could be found on startup and user closes app window instead of finding the source then app window would not close.
• BUGFIX: building icons were never shown in previous version.

v1.5

• Usage Check: now also (optionally) checks files in Custom Campaigns and, for units, Historical Battles.
• Usage Check: BUGFIX: v1.4 was only checking condition usage for units in export_descr_guilds.txt.

v1.4

• Displayed descriptions etc. now preserve repeated spaces. e.g. Previously two spaces in a row would be displayed as only one space. \n is shown as a line break instead of "\n".
• Text editor now has option to display "\n" as either "\n" or as actual line breaks.
• Effects Config: effects and Combat_V tabs now have a config load/save function, allowing multiple configs per tool copy to be saved and reloaded.
• Effects Config: Combat_V tab: added faction/religion game names.
• Last-used source location now automatically used when tool restarts.
• No longer reads some TW files that it never needed to, including maps; slightly faster startup times and no more halting of the reading process due to map errors.
• Some minor cosmetic changes including: "short" descriptions no longer italics, some gap added between border and text in text boxes.

v1.3

• Building orphan text: now excludes generic_repair, generic_dismantle & settlement_upgrade_xxx types.

v1.2

• Added Traits, Ancillaries and Effects Config.
• Added text manager for Units.
• Building images: blue border now signifies clickable (to show large image). "From M2TW" is now written as text rather than signified by red border.
• Buildings: building tree names now editable and removable.
• Names in lists now reflect edits made in tool.
• Lists retain current selection after writes.
• Building text manager: added more sort options, including no-tree options. Now filterable: missing text is now found from there; "missing text" tab removed.
• Units: "missing text" tab removed. Can now be found via text manager filters.
• Fixed bug: only "not" conditions were being checked in Unit usage check.
• Unit usage check: more elaborate check for unused units. Added battle script conditions.
• Added Building usage check.
• Various UI changes: text, layout, etc.
• Renamed to TWTECV: TW Text Editor, Cleaner & Validator.

v1.1

• "missing units": renamed to "usage check"; now also checks for non-usages; now also checks EDB, descr_mercenaries.txt and descr_rebel_factions.txt; condition checks can now handle "not" cases.
• Bug fix: if non-comment/non-blank/non-key line was found during read then it would use it as the value for last-found key, even if the value had already been found. Any non-comment, non-key line is now treated as part of the value of the latest key.
• Now reports presence of ";" comments as errors.
• Building "remove blank lines" now just removes blocks of blank lines. Removing blank lines within a building section is now already done as part of the value reading.

[Ngugi]

Awesome Withwnar, very handy tool; already helped me hunt up some units, deleted in the EDU everywhere were I could find them, that hid themselves still in the files, ready to cause issues when no one expected - but no more! Hoho

[Withwnar]

Update: v1.1.

Mainly increasing the scope of the "missing units" check but also fixes a bug...

In v1.0 this...

{baghlah_descr_short}
Arabic long distance sailing ship, with mounted swivel guns, useful for transporting troops and supplies.
Some other text on a different line.

...would be read by the tool as...

{baghlah_descr_short}
Some other text on a different line.

v1.1 reads it as...

{baghlah_descr_short}
Arabic long distance sailing ship, with mounted swivel guns, useful for transporting troops and supplies. Some other text on a different line.

[Withwnar]

I just found a bug in the Units usage checker. When checking conditions it is only checking "not" conditions. This will be fixed in an upcoming version.

[Withwnar]

v1.2 released. It now also handles traits and ancillaries, including a nifty effects description config builder. Plus some other changes: see the Change Log.

[OPASEH]

Look at generetic_repair in export_buildings.txt. Generetic_repair does not have any buildings description and cant be alone.

[Withwnar]

v1.3 excludes generic_repair, generic_dismantle & settlement_upgrade_xxx types from building orphan list.

[Gigantus]

Looks promising - I'll give it a spin.

[Gigantus]

Looks promising - I'll give it a spin.

Edit: any chance to get the application to 'remember' the last used mod?
Although it's easy to fix in map regions, the port error when the pixel touches a second region color diagonally only (just the corner) isn't really an error I believe, only if it is with at least one side, eg one side red region and another green region.

[Withwnar]

any chance to get the application to 'remember' the last used mod?

That is possible to add. The problem is that the Effects Config part will probably need to be configured differently for different mods, at least the Combat_V section (see the manual). Therefore the recommended usage of this tool is to have one copy under each mod, which in turn means that there's no need for it to remember the last used one because it (this copy) is only used for one mod anyway.

Although it's easy to fix in map regions, the port error when the pixel touches a second region color diagonally only (just the corner) isn't really an error I believe, only if it is with at least one side, eg one side red region and another green region.

So you're saying: the port belongs to the region that owns the tiles to the N, S, E & W of the port (not counting those tiles that are sea or off-map, i.e. has no owner.). If any of those tiles are NOT owned by the same faction then, and only then, it is an error. Yes?

[Gigantus]

Remembering: I don't think it is worth the effort to change - most people don't work on multiple mods like myself

If any of those tiles are NOT owned by the same faction then, and only then, it is an error. Yes?

That is correct.
Connection at the corners (NE, NW, SE, SW) are not taken into consideration. Your application is checking those corners as well which leads to false error messages, I discovered that with the 1648 map.
I am not sure if the same is true for settlement pixel, but I would assume it is.

[Withwnar]

Remembering: the effort would be in making the tool support more than one Combat_V and Effect config. Something that I was hoping to avoid but it would be better. I will look into it.

Ports: okay, thanks. My Map Reader tool must be saying the same thing. I'll fix that. Actually, I don't think that there's any need for this tool to be reading maps - behaviour that it has inherited from a core framework that most of my tools use - so I'll look at that too.

[Murfmurf]

Very nice! This'll save us a lot of time

+Reppage to you sir.

[Withwnar]

Thank you Murfmurf. Saving time is the intention. As is avoiding duplication of effort and tedious manual text editing.

~~~~

v1.4 released.

1. Displayed descriptions etc. now preserve repeated spaces. e.g. Previously two spaces in a row would be displayed as only one space. \n is shown as a line break instead of "\n".
2. Text editor now has option to display "\n" as either "\n" or as actual line breaks.
3. Effects Config: effects and Combat_V tabs now have a config load/save function, allowing multiple configs per tool copy to be saved and reloaded.
4. Effects Config: Combat_V tab: added faction/religion game names.
5. Last-used source location now automatically used when tool restarts.
6. No longer reads some TW files that it never needed to, including maps; slightly faster startup times and no more halting of the reading process due to map errors.
7. Some minor cosmetic changes including: "short" descriptions no longer italics, some gap added between border and text in text boxes.

3 & 5 mean that one copy of the tool can be used for multiple mods (more easily) and that the tool no longer needs to live under a mod folder to have the auto-find-and-load behaviour.

Personally I would still have one copy per mod, so that the copy always knows which mod folder to read and which effects configs to load. But now these copies can live anywhere instead of under the mod folder.

PLEASE NOTE: if you have any effects config files from a previous version and you wish to keep using them then you will need to rename them:

EffectConfigs.cfg --> Effects_xxx.cfg
CombatVConfigs.cfg --> EffectsCombatV_xxx.cfg

...where xxx is any name you like. To use them in the new version click the "load" button on the "effects" and "Combat_V" tabs. They will appear in the list of available configs named whatever you chose for xxx.

[Withwnar]

v1.5 released. This should be the last one for a while, unless it needs fixing in some way.

The Usage Checks now also check Custom Campaigns and Historical Battles. The fact that they were not checked was only ever mentioned for the unit checker.

It is important that such things are included in the check because a building/unit/trait/anc might exist only for the sake of a Custom Campaign; previous versions of the tool would have reported that they are not used, implying that they could be removed. Similarly for units: a unit might exist purely for the sake of an Historical Battle.

P.S. See also the previous post - if you missed it - for important changes.

[Ngugi]

Chill
More green!

[Silver Legionary]

I've always loathed cleaning up those 4 files(more than 4, but those 4 ESPECIALLY) by hand, you have my unending thanks for
making it easier.

[Withwnar]

v1.6.2 released.

This adds a couple of handy image tools and fixes some bugs. See the OP and its changelog for details.

[Gigantus]

Nice feature.
The TWC download section is working correctly now - for safety sake (remember MegaUpload?) and easy update without having to change links that's the way to go.

[MIKE GOLF]

Brilliant work Withwnar!!! I wish I would have had this tool a few months back. +rep


EDIT: I have just read the manual, and it seems pretty straight forward. I have recently redone all the text entries in export_vnvs and export_ancillaries for MOS 1.62, it would have been good to be able to verify all the effects with the corresponding export_descr_XXX entries. Whenever my new computer arrives I am sure I will use this tool right away and wil be sure to give you some feedback on your tool. Once again, great job!