CTD after editing "text" contents

[Tellos Athenaios]

I don't know what you mean by lookup table, not in this context at least. Is it something that should be embedded in the strings.bin somehow?

Yes, depending on strings.bin type. There are 3 different types produced/consumed by M2TW:

• Ordered. This format does not contain keys, so the mapping between keys and values (text) depends on the order of the entries. No lookup tables. Example: strat.txt.strings.bin. Get the order of entries wrong and you're looking at oddly misplaced text. Miss a few entries or have 'too many' and you'll likely CTD. This format is generally a pain if you want to generate the strings.bins yourself without the engine.
• Keyed. The basic format for strings.bins which are not ordered. Some strings.bins use this format, but you can safely ignore it and use the next one instead:
• Keyed with lookup table. The same as keyed, except a list of keys is also appended to the other contents of the strings.bin file. This format is used by export_units.txt.strings.bin and export_buildings.txt.strings.bin. Picking the keyed format without the lookup table does appear to work but comes with performance regressions. Note that in the case of export_units.txt the game may not, in fact, be able to generate the correct lookup table. Instead you might get a dump which looks suspiciously like a listing of RTW Vanilla units.

How do you create yours? BinEditor?

No, we use an internal release of the CUF tool with a few other bits of glue to build them from XML sources instead. That gets us a much better specified file structure in which comments are not some afterthought, with proper support for arbitrary character encodings.

Yes, that was one of my first suspicions but I found that removing them didn't help. The latest export_units.txt (which caused the CTD) had introduced some long texts but when I changed them to be shorter than any in the previous export_units.txt (which did not CTD) the CTDs persisted.

Yeah I didn't expect those lines to be exceeding 64KiB. I don't think the text file format matters much, except for line endings and line comments (¬ instead of ; from RTW).

[makanyane]

Note that in the case of export_units.txt the game may not, in fact, be able to generate the correct lookup table. Instead you might get a dump which looks suspiciously like a listing of RTW Vanilla units.

That dump looks even more suspiciously like the contents of M2TW/data's vanilla export_descr_unit_enums.txt

which is interesting, maybe we should be updating that... Withwnar do you fancy extending the scope of your TW Text Editor tool a bit?

EDIT;
and a game generated buildings.strings.bin ends up with the contents of export_descr_buildings_enums.txt at the end so those are used/read to an extent

though CA obviously didn't bother updating the units one - but I wonder if it would make a difference when you start getting into problems with these files.

Tellos what did you notice about "performance regressions" with not having a lookup table?

[Tellos Athenaios]

That dump looks even more suspiciously like the contents of M2TW/data's vanilla export_descr_unit_enums.txt

which is interesting, maybe we should be updating that... Withwnar do you fancy extending the scope of your TW Text Editor tool a bit?

EDIT;
and a game generated buildings.strings.bin ends up with the contents of export_descr_buildings_enums.txt at the end so those are used/read to an extent

though CA obviously didn't bother updating the units one - but I wonder if it would make a difference when you start getting into problems with these files.

Tellos what did you notice about "performance regressions" with not having a lookup table?

At one point the CUF tool did not yet write them (it only knew about the Ordered & Keyed formats), this resulted in long loading times. The most obvious example was the quotes strings.bin, loading would take a long time yet the loading screen and quote would only appear very briefly towards the end.

You can observe the difference in loading times quite easily using ProcMon and enabling tracing of additional low level file system operations -- you'll be able to see the time between open & close of the file changes quite a bit.

As for lookup tables in general, some files are more sensitive to them than others. For example, the events files will trigger CTDs if the game is unhappy about the lookup tables and order can be significant as well. The strings.bin format is simply not a very nice and well specified thing...

... Problem is I'm going by memory here and I've successfully managed not to let the specific horrors of each instance scar me too much so my recollection is not as vivid as it could be...

[Withwnar]

Thanks for the info. That also answers two other questions of mine:

1) Why does the game bother to read the enum files if they aren't used?
2) Why is the strings.bin file so much larger than the .txt one?

That dump looks even more suspiciously like the contents of M2TW/data's vanilla export_descr_unit_enums.txt

which is interesting, maybe we should be updating that... Withwnar do you fancy extending the scope of your TW Text Editor tool a bit?

EDIT;
and a game generated buildings.strings.bin ends up with the contents of export_descr_buildings_enums.txt at the end so those are used/read to an extent

though CA obviously didn't bother updating the units one - but I wonder if it would make a difference when you start getting into problems with these files.

My game-generated ones contain the mod's enum file contents, not vanilla M2TW's. TATW has modded the enum files but I haven't been for the submod I'm working on.

Okay, I'll try that. And yes, Mak, I'll make my tool do it. No way I'm doing the files by hand!

[Withwnar]

Hmm, yes, well, this is interesting.

It seems that if I increase the size of a .txt file then the problem disappears too. I added 2800 lines like this...

{test_key_1}This is filler line of text. This is filler line of text. This is filler line of text. This is filler line of text. This is filler line of text. This is filler line of text.
{test_key_2}This is filler line of text. This is filler line of text. This is filler line of text. This is filler line of text. This is filler line of text. This is filler line of text.

...to export_buildings.txt and no CTD. I then put that file back to how it was and tried doing the same to historic_events.txt instead: no CTD.

I did also try the enum idea. While the strings.bin files did appear to contain my new enum lists it didn't help with the CTD problem. I undid the enum changes before trying the above extra-lines tests; the enums definitely had nothing to do with those changes working.

Some other things I noticed along the way...

1) My export_buildings.txt.strings.bin is 1000kB in Explorer. That is a suspiciously neat number. However, its properties actually say this:
Size: 999 KB (1,023,584 bytes)
Size on disk: 0.97 MB (1,024,000 bytes)
Probably a false alarm.

2) Of all five files that I have been tinkering with, both modding wise and in all of these tests - export_buildings, export_units, export_vnvs, export_ancillaries, historic_events - only historic_events has its strings.bin file rebuilt after hitting the start button from the faction selection screen: the time of the CTD. All of the other four are rebuilt prior to the first menu screen. So perhaps historic_events is the real culprit here.

However, neither of those two points make sense in that changing the file size of ANY of those files fixes the problem.

So I'm none the wiser as to the cause but at least another workaround was found.

~~~

As for the enums: I take it, then, that keeping the enum files up to date and correct is a good idea? For performance reasons?

[Vipman]

I'm going nuts... I'm trying to add some unit descriptions but I got in a situation where apparently no matter what I write, to which unit I write, if I add enough words it gives me CTD. From existing 3 words and working I add 5-6 lines (viewing in word wrap), also tried writing random stuff: fkjebsoe cujshefb kdabjk, used bineditor to import, export and save, it still occurs. Just what the ......

[Withwnar]

I'm going nuts... Just what the ......

Yep, I had similar thought processes.

If it is the same problem as mine - which would be a very timely coincidence - then try one of my workarounds: either reduce a file size by chopping out unused entries or increase it using some dummy entries (attached). The "file" does not need to be the same one that is giving you problems (export_units.txt), e.g. doing it to export_buildings.txt or historic_events.txt worked for me and I suspect others would work too.

The attached has 2800 lines. That's a number I pretty much chose at random. Far fewer lines might also work.

EDIT: note that adding 2800 comment lines made no difference and neither did adding 2800 lines with the same key/label. Comments are discarded in strings.bin and so are duplicate key lines it would seem.

EDIT2: please let us know if either of these fixes your problem.

EDIT3: the attachment actually had 5600 lines with duplicate key names (one upper and one lowercase). This has been fixed.

[Vipman]

Thank you for this! It seems to work, I added the lines you uploaded. There was nothing old to remove so I had to add, the file had almost 400 lines and length 100k (characters I guess?). Doesn't make any sense why increasing length of a unit description gives CTD... BTW I was getting it after clicking start of custom battle.

[Withwnar]

Good to have it confirmed.

Doesn't make any sense why increasing length of a unit description gives CTD

There must be some limit on how many characters a description may contain. Tellos mentioned that on the previous page. But clearly that is not the cause of your CTD, nor mine.

The whole thing is bizarre, including the fact that you encountered it at the same time as me. And why does decreasing OR increasing the length of ANY file fix the problem? And why does deleting strat.txt.strings.bin fix the problem? No idea.

I'll change the thread title while I think of it.

[RubiconDecision]

I had a very weird issue where many of the routine adjustment to text and xml files were causing a CTD, and finally I began to suspect Notepad ++, that I'd somehow turned on some strange feature because I was noticing a lot of unusual characters I'd never seen before. As soon as I removed Notepad++ and reinstalled it, I had no further issues with editing files.

It might be totally unrelated to your issues, but might help someone else. The simplest trait changes were causing CTDs with inexplicable log file entries where the edit looked completely fine.

Saving those files as revisions meant I didn't lose all of my work, but could revert to one that worked, so somewhere along the way, Notepad ++ was inserting some kind of strange code into the text files and causing the CTD, but probably because I'd fat fingered a keystroke and turned on a feature.

[Withwnar]

I don't think it is related, no, otherwise the solutions mentioned here still shouldn't work.

I have never seen strange characters before, other than when using non-ANSI characters in ANSI files. e.g. Trying to use the ë character in script, EDCT, etc.

It might be that the files switched to another encoding somehow. They should be UCS-2 Little Endian for data\text files and ANSI for everything else.

Another thing I always check as soon as a file misbehaves for no apparent reason is the EOL characters. That wasn't the problem here though.

[Gigantus]

Yup, EOL and formatting is the first thing I check with ++ when stuff like this pops up.

[Squid]

I don't remember it being an issue in data/text files, but some files do not handle large number of tabs well and having too many can cause CTDs, especially line leading tabs. I believe EDB was one such file and I recall routinely doing a search and replace on tab characters at the start of a line (i.e. ^\t in regex replace)

[MIKE GOLF]

I don't remember it being an issue in data/text files, but some files do not handle large number of tabs well and having too many can cause CTDs, especially line leading tabs. I believe EDB was one such file and I recall routinely doing a search and replace on tab characters at the start of a line (i.e. ^\t in regex replace)

Can you please elaborate on what you are referring to in reference to the tab characters?

Withwnar I also noticed that your crashes have portraits in common with mine. Withwnar's sys log

15:06:41.234 [system.io] [info] exists:   found mods/Third_Age_32_Dcila/data/ui/greek/portraits/portraits/young/generals/000.tga (from: E:\M2TW)
15:06:41.234 [system.io] [info] exists:   found mods/Third_Age_32_Dcila/data/ui/greek/portraits/portraits/young/generals/000.tga (from: E:\M2TW)
15:06:41.234 [system.io] [info] exists:   found mods/Third_Age_32_Dcila/data/ui/greek/portraits/portraits/young/generals/000.tga (from: E:\M2TW)
15:06:41.250 [system.rpt] [error] Medieval 2: Total War encountered an unspecified error and will now exit.

my sys log

Files\Sega\Medieval II Total War)
04:53:03.009 [system.io] [info] exists:   found mods/Third_Age_3_RK/data/ui/custom_portraits/Gothmog/portrait_young.tga (from: D:\Program Files\Sega\Medieval II Total War)
04:53:03.009 [system.io] [info] exists:   found mods/Third_Age_3_RK/data/ui/custom_portraits/Gothmog/portrait_young.tga (from: D:\Program Files\Sega\Medieval II Total War)
04:53:03.029 [bink] [debug] [data/fmv/menu/DUMP0000.BIK]Pause on
04:53:03.265 [system.rpt] [error] Medieval 2: Total War encountered an unspecified error and will now exit.

I am not sure if there is any connection there or not.

[Withwnar]

Tabs: to create indented text like this...

building castle_tower
{
    convert_to tower
    levels c_ballista_towers c_cannon_towers 
    {
        c_ballista_towers castle requires factions { sicily, normans, saxons, }  and hidden_resource towers
        {
            convert_to 0
            capability

...you can either use spaces or the TAB key to create that 'empty space' before the line's actual text. So Squid's saying that if the file contains too many uses of the TAB key (i.e. too many TAB characters) then it can CTD.

In Notepad++ toggle on "show all characters" (toolbar button or View > Show Symbol > ... menu) to see them. Spaces are displayed as a dot, tabs are displayed as arrows.

However, I don't think that this is the problem here. Otherwise adding lines to a file, increasing its size, shouldn't make any difference.

~~~

Portraits:
My gut says no, that sometimes the last reported thing in the log has nothing to do with the error and this is another example of it. The CTD occurs only after changing a text file, not portraits, and is fixed by making other changes to text files, not portraits. So I think it's a red herring.

[MIKE GOLF]

Wow, I am truly baffled but very thankful. Withwnar, I added your "extra lines" to the historic_events file and now the game starts 5 times in a row with no ctd. I am completely at a loss though as to why this is working. Not complaining mind you but baffled is all. THANK YOU for the fix!

Will rep you when I have spread more around.


EDIT: I have also been able to add additional traits and ancillaries with no issues.

[Withwnar]

Good to hear, you're welcome and I'll put you down in the "baffled" column alongside myself.

[kanteljic]

This thread gave me the answer for the same problem! I stumbled upon it and had been trying to solve it for 4 days without success. After reading the thread, all I had to do was get the strat.txt.strings.bin from the original mod folder (Tsardoms, which I am rebuilding to get rid of bugs) and make it read-only. No more CTDs then.


One more useful info that might be handy to anyone who might need it: campaign_descriptions.txt is also among the files that have the weird "entry total" added between them to cause this CTD.

[Titus le Chmakus]

Dunno if it has been said before, but when I make some change to these text files, I sometimes delete all the .bin files in data/text. As the game recreates them every time needed (same as the map.rwm in data\world\maps\base), it sometimes solves crashes.