FAO Modders: Changes to ESF/Save format

[CraigTW]

So it depends? Hmmm... that it will load and not CTD is good news. That it will work as previously is not guaranteed, I understand from your answer.

Yes, that's right.

[Akaie]

Oh good. I was wondering whether my question would be answered .

[Plumo]

@Craig

I really wonder why supporting your older games is so hard, nowadays there is steam with auto update (I know it is much more than only that, though). But some small tweaks, like a newer version of speedtree for Empire total war campaign map (like in ROTS)? Just giving an example. Think also about the publicity , and it's a sign of total devotion to your products.

[PietroMicca]

Can we hope to get some information about pathfinding.esf? Everything is made by in-game data or there's some documentation we can read?

[Flikitos]

Hi CraigTW. Thank you for your effort concerning FotS!

Sorry to ask this question again but you forgot to ask it the last time. Have we any chance to have some help from CA's team about the hardcoded files on NTW, ETW and even S2TW?

Thanks again for your efforts.

[alhoon]

I'm very pleased that CA went forth and clean with that.
They changing the game for the better, they know it will cause mods to stop working or working as before and took steps to address it. Thank you very much.

OK, this is going way off topic and I don't want to hijack our own thread, but there would be a number of problems - not least that it would be a legal/IP minefield.

One of my goals is to bring CA closer to the community in any way possible, so we'll explore ways in which we can maybe work closer - but it has to be baby steps to begin with.

I'm all ears if you have ideas on how we can close the gap.

Well, the idea to warn the modders before time that their mods will experience problems was a good start. Please for the next game (Hopefully R2TW) consider doing the same: Tell modders what would work beforehand from the existing tools.
Of course... modding tools to allow putting in regions would be also nice, but I think they're hard to make.

[taw]

New type ids etc are simple enough. LZMA compression might be some pain in the ass since my modding tools all use JRuby,
and that's probably not bundled together with standard Java Runtime Environment or JRuby.

Anyway, if someone emails me a bunch of esf files in this new format, I'll try to add support for them.

[DARTH VADER]

taw
I can send you the new ESF file type to check when my game is downloaded from Steam.

[taw]

taw
I can send you the new ESF file type to check when my game is downloaded from Steam.

Yeah, do that, you have my email.

[King Sama]

Very good action by CA, I'm personally very happy with the increased modder-CA communication, even if it is one step at a time. I am even more happy that Craig took the time to answer the question(s) concerning the petition.

And of course, taw is to the rescue!

[Steph]

I'm all ears if you have ideas on how we can close the gap.

Ok Craig, I send you a PM and email through TWC. I know you usually don't really read them, but try to get this one.

[taw]

I tried to write a decoder, but documentation and reality seem to mismatch.

Documentation says there are no longer sections like that:

Type id high bit set indicates section; (that's bit 0x80)
next bit indicates section block; (that's bit 0x40)
next bit indicates old-style section id and version (written as 16 bit unsigned integer for id then 8 bit unsigned integer for version). (that's bit 0x20)
If this bit is not set, then read next byte.

Version is in the next four bits of the type id – (type_id & 31) / 2

section id is in the remaining bits – (type_id & 1) * 256 + next_byte.

The way I read it it means (aaaa - version, bbbb - type):

110a aaab bbbb bbbb section
111a aaab bbbb bbbb section block
1000 0000 bbbb bbbb bbbb bbbb aaaa aaaa - section
1100 0000 bbbb bbbb bbbb bbbb aaaa aaaa - section block

Here's startpos.esf from the new campaign.

First 16 bytes make perfect sense:

CA AB 00 00 - new magic
00 00 00 00 - 4 byte of 0s
BF 85 47 4F - 4 byte of timestamp (or whatever it was)
A4 D8 2A 00 - address of footer (and footer decodes fine, except there are now extra 00 bytes after footer which parser has to ignore)

Then there's a problem:

80 00 00 07 - by old interpretation this means "section CAMPAIGN_STARTPOS version=7" (s2tw esfs I have here are version=6, so this makes perfect sense)
81 AB B1 0C - by new decoding, block size number 2807948 (0x2AD88C), corresponding to end offset 2807948 (0x2AD8A4), which is perfect.

But new interpretation claims to be that - since it doesn't have 20 bit set, it should be CAMPAIGN_STARTPOS version=0 (makes no sense) followed by 00 (size) and total random garbage!!!

So I tried to continue decoding under assumption that 20 bit is inverted or something, then I get:

88 01 (section SAVE_GAME_HEADER, version 4 - s2tw had version 2 so it's possible number)
88 36 (size 1078 / 0x436)

Under old interpretation this is garbage bytes (there's no 0x88 type id at all).

So each interpretation makes either CAMPAIGN_STARTPOS or SAVE_GAME_HEADER wrong.

And if I let's say interpret bit 0x08 in 0x88 (1000 1000) as magic bit instead of 0x20 (which is not set in both cases), then version of SAVE_GAME_HEADER will be 0, which will be probably incorrect (unless they can move backwards now).

After that mess there's clear part - two strings (0E + 4 byte index each), uint32=1, uint32=1864 (together that's starting date or something, just like in S2TW), two more strings.

Then comes this:

84 02 - section DATE version=2 (perfectly matching S2TW)
07 - size 7
(4 uint32s within, nicely compressed)

Then there's a string (which wasn't there in S2TW savegame).

Then C0 03 = section block, MAPS version=0 (just like S2TW). etc. etc.



Anyway, the big problem is that the root node is encoded as 80 00 00 07 (just like in old ESFs), not as either 9D 00 or A0 00 00 07 as documentation claims it would be.

Am I doing something wrong, or is there an error in documentation?



EDIT:

And at offset 17562 (0x4496) there's 80 0F, which is very much NOT old style 80, it's new style section COMPRESSED_DATA version=0.

Why is 80 00 00 07 root node old style, while 80 0F is new style?

EDIT:

OK, as far as I can tell the new rules apply everywhere except for the root node, and that's the whole story.

[taw]

esfxml has now extremely basic support for the new format.

esf2xml's dividing data into multiple XMLs works, but there are no semantic names only numbered names like "victory_conditions/0001.xml" not "victory_conditions/bos_tokushima.xml". (this is probably easiest to fix). Logic for semantic names like "victory_conditions/bos_tokushima.xml" doesn't handle some complex cases for new files, but it works most of the time.

esf2xml disables all high level conversion and outputs only low level xml for new style ESFs. (some of these would work, but many rely on low level details of encoding - it would need checking one by one which work and which don't at least, in principle it's all fixable)

esf2xml doesn't even attempt to handle compressed data, it just saves it to compressed_data.xml. (so almost nothing is moddable for now...)

xml2esf generates esfs that are not byte-for-byte identical yet. In particular all variable sizes and offsets are encoded as 4 bytes (1 as "80 80 80 01"), except for 4x/5x arrays where we know the size in advance. There are some minor problem like using 4 byte ints for 3-byte ints, stripping 0 padding at end of file etc. Unfortunately fixing them all and achieving byte-for-byte identical ESFs will require rewriting a lot of stuff xml2esf.

As far as I know files esf files recreated by xml2esf might work just fine in spite of these differences in encoding. The only moddable stuff is what's in uncompressed headers - victory conditions etc.

If someone could try unpacking, modding victory conditions, and repacking just to confirm that it works so far, it would be great.

[wis]

Mod tools? Or better yet make your next game more moddable.

[taw]

New format now round-trips esf->xml->esf byte-perfect.

Support for high-level conversion very limited, since a lot of converters relied on low level details.
It would be hard to test it anyway, since the only records in uncompressed part with high level
converters are DATE and REGION_OWNERSHIPS_BY_THEATRE (and MAPS, but its format is changed).

No support for compressed data yet, which is the biggest missing thing obviously.

[taw]

I cannot decompress data.

First, converter extracts it to:
<compressed_data size="9717493" meta="5D 00 00 04 00" path="compressed_data.data"/>

Then (looking at Java code) I tried to write this (5 byte properties, 8 byte size, data):

5D 00 00 04 00 F5 46 94 00 00 00 00 00 (everything in compressed_data.data).

And unpack with lzma command line program.

$ ruby -e 'print "\x5d\x00\x00\x04\x00", [9717493].pack("Q"), STDIN.read' <tmp/startpos_boshin_esf_dir/compressed_data.data | lzma -d >out.esf
lzma: (stdin): Compressed data is corrupt
-e:1:in `write': Broken pipe (Errno::EPIPE)
from -e:1:in `print'
from -e:1

While creating out.esf file with size 9717493 and md5 8b353ded87f54941254ddf23d5eb8243, which seems to start like ESF file, but not quite:

CA AB 00 00 - good header
00 00 00 00 - zeroes
BA 85 47 4F - timestamp
A2 42 F1 00 - this is supposedly index to tag table, but it's out of bounds
80 00 00 07 - root record
etc.

It starts sensible, but it ends in what looks like total garbage.


If anybody has a clue what I might be doing wrong, it would be most helpful now.

[taw]

Ignoring claimed size (9717493), and saving compressed data with unspecified size (-1), I get uncompressed ESF of size 16074388 and md5 1f7bf4f661a6b9975211574fa51108b8, which seems fine enough.

Any idea what is wrong with that size parameter?

Now, for the modders:

First unpack esf normally:

$ jruby esf2xml startpos.esf startpos_dir

In startpos_dir you'll see file compressed_data.esf.xz . Using lzma command line program (at least with OSX/Linux) you can unpack it:

$ lzma -d <startpos_dircompressed_data.esf.xz >compressed_data.esf
lzma: (stdin): Unexpected end of input

Ignore the error for now.

Now unpack esf it extracted:

$ jruby esf2xml compressed_data.esf startpos_dir_nested

So far this data is good mostly for viewing. Until I figure out how exactly this compression works, why I'm getting this error, and why stated size is wrong, that's probably the best we'll have.

EDIT: Wait, is size data of reversed endianness, or are all 3-byte integers (type 24/28) of reverse endianness or what else is going on???

[taw]

OK, the mystery solved, 3-byte (u)int32s are big-endian (like the new variable length coding), while everything else is little endian.
2-byte (u)int32s seem to be little endian as far as I can tell (from DATE fields for example).

If someone wants to get to compressed data, it's:

$ jruby esf2xml startpos.esf startpos_dir
$ lzma -d startpos_dir/compressed_data.esf.xz
$ jruby esf2xml startpos_dir/compressed_data.esf startpos_dir_nested

And I haven't really tested it, but recompressing should be:

$ jruby xml2esf startpos_dir_nested startpos_dir/compressed_data.esf
$ lzma startpos_dir/compressed_data.esf
$ jruby xml2esf startpos_dir startpos.esf

It's not the most convenient interface ever, but at least you get to mod it and no other tool works with new ESFs yet.

Conversion of nested esfs is not byte-exact, but as far as I can tell it's only because they originally use
weird encodings (5 byte sizes where 4 bytes would be enough - 80 87 C5 85 09 vs 87 C5 82 42 etc.).

Other than some inconvenience, it's ready for modding.

[adrian74]

hi Craig

not sure were to put this so forgive me if its in the wrong thread . But at the end of Shogun total war ( first release) there was a video at the end that told about the expulsion of the west from Japan .

Now i know you have done if you like the return of the westen powers with Fall of the samurai, but i wondered seeing as you have most of the units for Japan if you would consider doing another expansion for it . The time when Japan deicided to close the country and expell the west . All you would need is a few new units for the west a few ships and a little bit of work. Think adding this area too most of the TWC would like .

Think it would if you like make a complete shogun total war series .
I am also glad you went for the west time as an expansion as it is a lot better than the mongul invasion which you had for shougun total war though i do have that too and did enjoy it

Just a suggestion but one i think would make sense

takecare
adrian

[♘Top Hat Zebra]

Wow. taw, I don't understand what any of that means, but you seem to have been working on this all day, and I think you deserve some rep.

Er, that is, when Im able to.