CTD after editing "text" contents

[Withwnar]

UPDATE: still no idea why this is happening but there seems to be two workarounds:
1) Reduce the size of one of the text files, e.g. discard unused entries. Doing this with export_units, export_buildings or historic_events worked for me (did not try others).
2) Increase the size of one of the text files by adding a bunch of dummy lines (see post 27).
Strangely, increasing/decreasing the size of ANY text file seems to fix the problem (export_units, export_buildings or historic_events at least). That is, if you had only altered export_units when the CTD occurred then you do not necessarily need to increase/decrease export_units to fix the problem: you could alter historic_events instead, for example.

~~~~~~~~~~~~

Help!

I am getting a CTD every time I start a new campaign - as soon as the faction selection menu's "start" button is clicked - UNLESS I delete strat.txt.strings.bin first. That is, I need to delete that file every time I start a new campaign. I don't understand why because I have never had this problem before.

If I delete it and run the game then no problem. If I quit then I get a CTD every time I try starting a campaign from then on (not just the first attempt).

I had changed strat.txt but I reverted it back to how it was and I still have the problem. What really has me stumped is that I took both strat.txt and strat.txt.strings.bin from vanilla Third Age (I'm working on a submod) and put them in my folder and I still get a CTD. So it isn't a problem with strat.txt itself otherwise TATW would have this problem too, right?

If I delete both strat.txt and strat.txt.strings.bin then no CTD any time (the log says found it in the packs and the texts are missing in-game).

I do have some changes in other files but I can't spot any problems and in any case why would they only cause a CTD when strat.txt.strings.bin is present? They don't throw any log errors when the game does start (beyond a few that I have always had).

Any help would be very much appreciated because I'm utterly stumped.

The log is attached. These errors are not the cause because I have had them for a long time:

Spoiler Alert, click show to read: 

15:02:15.593 [script.err] [error] Script Error in mods/Third_Age_32_Dcila/data/world/maps/campaign/imperial_campaign/descr_strat.txt, at line 642, column 1
The character record Vice-Regent Elrond is set as faction heir but is not in the family tree.
15:02:15.609 [script.err] [error] Script Error in mods/Third_Age_32_Dcila/data/world/maps/campaign/imperial_campaign/descr_strat.txt, at line 2068, column 152
You already have a faction heir
15:02:15.625 [script.err] [error] Script Error in mods/Third_Age_32_Dcila/data/world/maps/campaign/imperial_campaign/descr_strat.txt, at line 3065, column 1
The character record Protector Malgalad is set as faction heir but is not in the family tree.
15:02:15.625 [script.err] [error] Script Error in mods/Third_Age_32_Dcila/data/world/maps/campaign/imperial_campaign/descr_strat.txt, at line 3230, column 1
The character record Lord Amonost is set as faction heir but is not in the family tree.
15:02:15.640 [script.err] [error] Script Error in mods/Third_Age_32_Dcila/data/world/maps/campaign/imperial_campaign/descr_strat.txt, at line 4400, column 92
You already have a faction heir
15:02:15.640 [script.err] [error] Script Error in mods/Third_Age_32_Dcila/data/world/maps/campaign/imperial_campaign/descr_strat.txt, at line 4827, column 90
You already have a faction leader

[Withwnar]

UPDATE

Hmm,well, I think that I have tracked down the cause.

It appears to be related to export_units.txt. I went through the changes to the other files I had made and found that the problem reappeared only after reapplying the export_units.txt ones. (Why the presence of strat.txt.strings.bin has anything to do with it I have absolutely no idea.)

Using this tool I removed all of the orphan unit texts and the problem disappeared. None of them seemed to have anything erroneous about them so I tried removing only half of them: problem gone. Then I tried the other half instead: problem gone. So it wasn't any particular entry causing the problem.

So I'm now wondering if there's a file/character size limit for export_units.txt.

The mod has 351 units in EDU and there were 27 orphans. That's nowhere near the 500 limit so perhaps it is the total length of the descriptions and therefore file size.

Have you cleared your cache? Does the CTD occur after cache clearing, again?

Thanks Aik. I should have mentioned that I'm using XP not Vista etc. so virtual stores had nothing to do with it. Is that what you meant by cache?

[Gigantus]

I have an export_unit file with 1504 lines (501 unit entries, 315675 characters) and it runs without a problem. Perhaps a formatting issue?
With Notepad++ I usually remove trailing spaces (edit\blank operations) and make sure it's encoded in USC-2 Little Endian.
If I have problems my first step is to 'import' and then 'export' the TXT version with the BinEditor and then 'save' it to get the STRING.BIN version. That usually solves formatting issues.

[Withwnar]

Thanks for that.

Mine is 1479 lines, 282968 characters, USC-2 Little Endian. Some of the descriptions are long but, yeah, I'd be surprised if there weren't bigger files out there for mods with many more units than this.

I don't think that it is a formatting issue because it didn't matter which lines I removed to solve the problem. Taking out either chunk A or chunk B did the trick. If only A had a formatting problem within it then taking out just B wouldn't have fixed it (but it did) and vice versa. If both had a formatting problem within it then both would have needed to be removed (but they didn't).

I'll try the BinEditor and see if it helps.

EDIT: nope, no luck with the BinEditor.

[Aikanár]

No, I didn't mean the virtual store, but I meant clearing your ram/system cache. Sometimes I get very strange results for absolutely no logical explanation doing the very same edit before and after a clearing the ram/cache.

Can you upload the strat.txt so that we can test if we can reproduce the ctd?

[Withwnar]

Good to know, cheers. It didn't help here though.

Here's the EU file...

EDIT: oops, I just realised you asked for strat.txt. I'll add that too.

However, to get a CTD I'm now finding that I only need to delete export_units.txt.strings.bin: the game crashes first time. But removing strat.txt.strings.bin fixes it.

This is what I'm getting...

1) delete export_units.txt.strings.bin
2) start campaign --> CTD
3) delete strat.txt.strings.bin
4) start campaign --> no CTD
5) shut down the game
6) start campaign --> CTD

Repeating steps 5 & 6 always gives CTD, unless I delete strat.txt.strings.bin.

After removing some entries from export_units.txt and doing all of those steps again I never get a CTD.

So I believe that strat.txt has nothing to do with it - but have no idea why strat.txt.strings.bin plays a part.

[makanyane]

At stage 3 does ddeleting a different strings.bin work?

[Withwnar]

Hmm, still CTD but now at step 4 instead of step 6. I tried export_buildings and event_titles; same result for each.

HOWEVER: I removed entries from export_units up to the point that removing one more would fix the CTD. So still a CTD at this point. Now I removed a couple of entries from historic_events and no more CTDs! I tried two campaign starts after not deleting the EB & HE strings.bin files and two starts after deleting. It's definitely CTD free.

What on earth is going on there? It seems that if the combined size of all of the "text" files (or their strings.bin counterparts) is too large then I get the CTD.

[makanyane]

Don't know - but suggest also testing if you reduce the length of a lot of the unit descriptions or other entries - does that fix the problem in the same way as deleting the entries?
i.e. is it the 'size' of the files in terms of numbers of characters, or the total number of {} entries.

[Withwnar]

It does, yes. Reducing the length of some HE text entries fixed the CTDs.

So it's file size not entry count or line count.

[Gigantus]

Try this pair

[Withwnar]

Thanks. Still a CTD, regardless of using your strings.bin file or letting the game build its own.

I'm sure that it isn't any one particular file causing the problem, given my findings in post 8.

[Zarathos]

Ok, here is my suggestion. First thing: replace "[ \t]+$" with "". Then: set the encoding again to Unicode Little-Endian. Now... perform a binary search for hidden/special characters.

[Withwnar]

Thanks Zarathos but this would suggest a formatting problem with a particular file. I'm convinced that it isn't that because if I reduce the length of any* file then the CTDs disappear (post 8). If there was a formatting problem in file X then reducing the size of file Y, Z or W should still CTD.

* Files I have tried are export_units.txt, export_buildings.txt and historic_events.txt.

It's as if the combined size of my text files has hit some limit. By the way it is not just my PC: same problem on a different PC and a fellow modder's.

[Gigantus]

Have you tried reducing 'dead wood' in the text files with your checker? There is usually a lot in vnvs and in the units file.

[Withwnar]

I have, yes. It was by using that tool to remove a bit of dead wood that I found the 'solution' in the first place.

I can chop away a lot more but my fear is that it might not be enough to offset the amount that still needs to go in. Many descriptions are just short placeholders at the moment and there are more units, buildings, historic events, etc. to come. Hopefully it will be enough and I never need worry about this crazy error again. Although, it might happen to somebody else one day so a real solution would have been nice.

[Tellos Athenaios]

Whatever the root cause, the example export_units.txt.strings.bin which Gigantus posted does not contain a lookup table. The engine really would like to have one for export_units and export_buildings (at any rate).

For EB2 we try not to rely on the M2TW engine to generate our strings.bin files, and we run the mod with (last time I update the strings.bins on that working copy, so it is bound to be bigger by now) some rather bigger strings.bins:

• 998.4KiB for export_units.txt.strings.bin (i.e. nearly twice the size of your example)
• 8.8MiB for export_buildings.txt.strings.bin (somehow I don't reckon...)
• 579.3KiB for names.txt.strings.bin (even our names are bigger than your example...)

All in all, the strings.bins weigh in at 15.9MiB, so let's peg this hypothetical size limit at a round 16MiB.

Does your mod exceed this 16MiB figure by any significant amount?

----

In terms of the file format the strings.bin files use 16bit unsigned values to indicate string length (either key or value, but I don't advise you try using 64kiB keys...) and 32bit unsigned values for total number of entries. So there's an upper limit of 2^16 -1 characters for any given description. I did notice you have some very, very long lines with lots of wrapping going on but I don't reckon you hit that limit either.

[makanyane]

The bit that's confusing me more is why deleting strat.txt.strings.bin and letting the game rebuild it apparently fixed the problem for that game launch. (see With's post #6)

Can only assume there's a chunk of memory allocated for the strings.bin building process which gets added to whatever is used to just read the existing strings.bin files? Or somehow or other it's changing the way it processes them...

[Withwnar]

Thanks for the input. Some interesting points here.

My strings.bin files total ~5MB which I did notice was less than vanilla TATW; that contradicts my theory of a size limit, I know. Unless it's the total size of only some files, not all of them, that is the cause.

the example export_units.txt.strings.bin which Gigantus posted does not contain a lookup table. The engine really would like to have one for export_units and export_buildings (at any rate).

I don't know what you mean by lookup table, not in this context at least. Is it something that should be embedded in the strings.bin somehow?

For EB2 we try not to rely on the M2TW engine to generate our strings.bin files

How do you create yours? BinEditor?

I did notice you have some very, very long lines

Yes, that was one of my first suspicions but I found that removing them didn't help. The latest export_units.txt (which caused the CTD) had introduced some long texts but when I changed them to be shorter than any in the previous export_units.txt (which did not CTD) the CTDs persisted.

[Withwnar]

The bit that's confusing me more is why deleting strat.txt.strings.bin and letting the game rebuild it apparently fixed the problem for that game launch. (see With's post #6)

Can only assume there's a chunk of memory allocated for the strings.bin building process which gets added to whatever is used to just read the existing strings.bin files? Or somehow or other it's changing the way it processes them...

I think it might be because strat.txt is a bit different. If it's strings.bin is deleted then it uses the one from the packs, even though it's rebuilding one in the mod folder. I think. If so then that might have something to do with it.

EDIT: hmm, I just tried it again. After deleting strat.txt.strings.bin I went into the game and looked at some things that strat.txt defines. They were all there, as defined by the mod, not vanilla.

So disregard the above comment. I must have had my wires crossed from trying so many things. I definitely had a case where I wasn't seeing any strat.txt descriptions in game but can't recall how I got to there.