Perfect ESF<->XML converter and revolution in ESF modding [Released!]

[taw]

So now that every single DB table except one is moddable, the next step is ESF.

Current ESF architecture makes even the simplest changes really difficult, and you're only allowed one change - there's no esf merging ever (hybrid startpos.esf tricks aside).

I didn't like any of it, so I wrote a pair of scripts that convert ESF to XML, and then XML back to ESF. ESF->XML->ESF round trip is perfect to the very last byte on startpos.esf, regions.esf, and every other esf I tested.

Once data is in XML format, all sort of mass modifications like removing every single fort (db-based no forts mod horribly confuse AI, esf-based no forts mod don't have such problems), flipping factions from minor to major, changing character skills and abilities, giving every minister a mistress, changing towns minor to major, and many more change from long painful effort to trivial find-and-replace-via-xquery, and a brave new world of far more interesting modding becomes wide open - including all CAI stuff that hides inside ESF for no apparent reason, and maybe even new campaign map because why the hell not.

This is good kind of XML - there are binary offsets, or other nonsense in it - conversion scripts handle all that, you can just go ahead and edit.

Right now the most serious practical problems are:

• These XMLs are really huge - startspos.xml is 473MB (75% of that is CAI-related stuff, not even binaries). I should be able to cut it at least in half without losing anything of value, but it will still need a really good computer.
• Conversion takes about 4 minutes each way. 4 minutes is nothing compared to an evening of trying to get things done with ESFEditor, but it should be faster.
• It's all Ruby code that might be hard to get running on Windows. I'll write some detailed instructions at least.

Once these three are done I'll publish the scripts, hopefully in a few days and together with collection of easy to use recipes for making amazing things with ESFs.

If you want early alpha access, ask me on PM or email, but it's still quite messy, and stress testing has been limited.

[The Hedge Knight]

Awesome news!

Great stuff with db editor (shame its not fully compatible with ntw though )

[taw]

Awesome news!

Great stuff with db editor (shame its not fully compatible with ntw though )

Well, I don't even have NTW.

If anyone's is in giving mood, my steam account is here -> http://steamcommunity.com/profiles/76561198021187429/

Unless there were some drastic changes between ETW and NTW,
extending my analysis to work with NTW should take little time once I have access to it.
I cannot make any hard promises, but I see no reason why it shouldn't
be straightforward after everything I've done to ETW already.

(I tend to ignore *TW games altogether for the first year or two,
they're all awful before at least the third patch).

[The Hedge Knight]

Ntw is awesome and has had 5 patches .

Also its possible to make some tables for ntw work using db editor if you stick a blank file names empire.exe (txt) in data and use that as the path... Only wanted to look at models artillery as most i need are covered by pfm.

[taw]

If it's just sticking blank empire.exe, it's not a big deal - for example Steam version of M2TW and Kingdoms is broken and requires similar silliness as well.

Could you quickly check how well does the new DB.xsd work with NTW, if it's not much work?

[Heavy Weapons Guy]

Does this means that we will be able to properly mod ETW like MTW2?

[taw]

Does this means that we will be able to properly mod ETW like MTW2?

That has been my intention all along. As far as I can tell most things in ETW are not so much hardcoded, just very difficult to access without proper tools, and CA is definitely not being friendly.

[Primergy]

I'm looking forward.

Btw...Would it be possible for your converter to convert the information of rigidsplines (which are used to place roads, rivers and borders) to something esf can handle?
Actual if it you change the position it's only a visual effect.

Does this means that we will be able to properly mod ETW like MTW2?

If it would be already possible to handle the esf files much smarter, would be already a huge step forward

[taw]

I'm looking forward.

Btw...Would it be possible for your converter to convert the information of rigidsplines (which are used to place roads, rivers and borders) to something esf can handle?
Actual if it you change the position it's only a visual effect.

Is this the rigid splines converter you're looking for?

regions.esf has massive float arrays region_data/vertices and
many int32 arrays with indexes into it (I think indexes are of vec2s, so you need to double them).

I attached a snippet of xml created out of regions.esf. Does it look like it might contain what's needed?
Doing vector graphics from xml is not really optimal, but I bet regions.esf SVG export tool would only take an evening or two to a dedicated person (import probably more).

[Peteriscoolz]

If this site like to maintain "Good" realations with CA And Sega then why can't they maintain Good realations likewise and realese modding tools?

[husserlTW]

Current ESF architecture makes even the simplest changes really difficult, and you're only allowed one change - there's no esf merging ever (hybrid startpos.esf tricks aside).
(...)
Once data is in XML format, all sort of mass modifications like removing every single fort (db-based no forts mod horribly confuse AI, esf-based no forts mod don't have such problems), flipping factions from minor to major, changing character skills and abilities, giving every minister a mistress, changing towns minor to major, and many more change from long painful effort to trivial find-and-replace-via-xquery, and a brave new world of far more interesting modding becomes wide open

I am not sure what you mean. You can make complex changes, export and import parts of esf, etc., using the Editor.

On the other hand any development in esf editing is mostly welcome though I don't understand why XML method will make it easier. You have to have in mind a couple of things though. Editor saves changes instantly or in a few seconds. When you have to make complex changes is wise to make them in stages testing everyone first before you proceed in next one. The referred converting times are too large and must be reduced significantly, or ot will be not practical at all using converter.

If it is not by default it should be necessary to be able as well the conversion of certain parts which first have been exported by the editor. For example to change a faction from minor to major it is only a few seconds using editor. But to relocate an army it is a complex work and it will be indeed easier if I can have in XML the necessary startpos sub-records (CHARACTERS, ARMY_ARRAYS, CAI_RESOURSES_MOBILE, etc.) available on my desktop. I can see a dynamic combination of an editor and a quick converter which perhaps will make esf editing easier. And I can assume that as smaller the esf is so faster the conversion will be.

including all CAI stuff that hides inside ESF for no apparent reason, and maybe even new campaign map because why the hell not.

There are CAI fields like MANAGERS, BDI, etc that your idea might be very helpful. Though we've managed to advance the research in some of those areas, these endless lists are not practically at all in editing. I expect the same endless lists in XML but on this case will be all data on sight than jumping from a sub folder to another (there are some thousands of them). The same stands for campaign files like poi.esf, pathfinder.esf, etc.

If you want early alpha access, ask me on PM or email, but it's still quite messy, and stress testing has been limited.

I defiantly like to be a part of this effort and have access in the stuff you've got under development. I work with esf files all the time and I can help -at least- in stress testing.

Last I want to underline that EsfEditor 1.5 (still in beta) is a remarkable tool, very fast, uses labels and can edit Napoleon's files as well. If anyone can help Erasmus to complete it it would be good to communicate with him.

[taw]

I am not sure what you mean. You can make complex changes, export and import parts of esf, etc., using the Editor.

This is slow, painful, error-prone, manual work. Once I spent two hours manually removing every fort from game in startpos.esf. Then I found someone else made all factions including protectorates fully playable with working diplomacy, and I wanted that as well. How long would it take to merge both changes?

Exactly as long as making one of them from scratch.

With text-based formats it's just run diff, run patch, finished. Amazingly easy.

How can I even as little as search for all settlements with fortifications in ESF Editor, let alone do search&replace?

When all of it is in nice text, Cmd-F settlement_fortifications, and for every block like:

<record flags='1' name='BUILDING_MANAGER'>
<true />
<BUILDING flags='1' percent='100' building='settlement_fortifications' faction='knights_stjohn' government='gov_absolute_monarchy' />
<false />
</record>

Delete and paste over a block like:

<record flags='1' name='BUILDING_MANAGER'>
<false />
<false />
</record>

Done. Actually I'm not quite sure if interpretation of flags and booleans used by ESFEditor and by my script is quite correct - they seem to have something to do with optional fields - I'll try finding out what's going on later.

On the other hand any development in esf editing is mostly welcome though I don't understand why XML method will make it easier. You have to have in mind a couple of things though. Editor saves changes instantly or in a few seconds. When you have to make complex changes is wise to make them in stages testing everyone first before you proceed in next one. The referred converting times are too large and must be reduced significantly, or ot will be not practical at all using converter.

Editor saves changes quickly only if it file size doesn't change, otherwise (as with deleting fortifications) it is very slow.

Current converting times are 3+4 minutes round trip for startpos.esf, but that's due to weird binary fields I'm serializing and deserializing as hex mostly. regions.esf is 10% size of startpos.esf, but takes 10 for round trip.

Converter can skip parts of xml for super-fast operation, but there's no way to merge such partial xmls yet. I hope to make this unnecessary and make it always fast.

If it is not by default it should be necessary to be able as well the conversion of certain parts which first have been exported by the editor. For example to change a faction from minor to major it is only a few seconds using editor. But to relocate an army it is a complex work and it will be indeed easier if I can have in XML the necessary startpos sub-records (CHARACTERS, ARMY_ARRAYS, CAI_RESOURSES_MOBILE, etc.) available on my desktop. I can see a dynamic combination of an editor and a quick converter which perhaps will make esf editing easier. And I can assume that as smaller the esf is so faster the conversion will be.

There are CAI fields like MANAGERS, BDI, etc that your idea might be very helpful. Though we've managed to advance the research in some of those areas, these endless lists are not practically at all in editing. I expect the same endless lists in XML but on this case will be all data on sight than jumping from a sub folder to another (there are some thousands of them). The same stands for campaign files like poi.esf, pathfinder.esf, etc.

I defiantly like to be a part of this effort and have access in the stuff you've got under development. I work with esf files all the time and I can help -at least- in stress testing.

Last I want to underline that EsfEditor 1.5 (still in beta) is a remarkable tool, very fast, uses labels and can edit Napoleon's files as well. If anyone can help Erasmus to complete it it would be good to communicate with him.

Let's move this part over email. I'll PM you.

[hip63]

Sounds pretty dang awesome!

and I love the attitude: and maybe even new campaign map because why the hell not.

hip63

[taw]

Right now the most serious practical problems are:

• These XMLs are really huge - startspos.xml is 473MB (75% of that is CAI-related stuff, not even binaries). I should be able to cut it at least in half without losing anything of value, but it will still need a really good computer.
• Conversion takes about 4 minutes each way. 4 minutes is nothing compared to an evening of trying to get things done with ESFEditor, but it should be faster.

I got it down to 291M and 2 minutes from esf->xml, and 3m20s xml->esf for whole startpos.esf - on a good laptop.

I cannot think of any serious technical reason why it needs to be worse than 1 minute each way and about 2x overhead (~150 MB). As you only need to convert once before you start a game, 1 minute would really not be a big deal. The only thing it needs is sufficiently many rounds of profiling, improving, and re-profiling.

I could get there in 5 minutes by just throwing out large blobs of binary data of unknown functionality aside, but this is precisely where most interesting modding opportunities are likely to reside.

That mystery flag seems to be version number, just like in db. They're mostly 0s, a lot of 1s, then a few 2s, and even fewer and fewer but it reaches as high as 13. It probably needs to be per-entry not per-file to make new versions of the game work with old save games.


By the way here's a screenshot of svg I quickly made from regions.esf. Is this actual map of the world or not? ETW already has a test area, between Louisianas and Natives for which map says one thing visually but it's different really, and by a few thousand kilometers.

If this one won't work, there are others to check elsewhere.

(low quality picture likely due to unrelated data getting in a way of my regular expressions,
that's why it's important for XML to have high quality schema, not merely to make the data available)

Who's the first volunteer to try enlarging Guyanas?

Spoiler Alert, click show to read: 

Oh, and position technical explanation: all data in regions.esf uses indexes stuffed in integer arrays. Integers [17, 105, 92] all refer to one global vertices table of floats, and it means x0=float34, y0=float35, x1=float210, y1=float211, x2=flota184, y2=float185. (Y is upside down, coordinates normalized to x=-1280..1280 y=-640..640, but obviously you knew that much.

The first person to create a new region (without destroying another) gets an award.

[Heavy Weapons Guy]

I know right people would could try I'll PM him.

[Peteriscoolz]

Lets make a region and show those ers that we mean buisness!

[The Hedge Knight]

:0... I still think regions are a long way off (with tables to be edited and other esfs) but... Nice...

[Peteriscoolz]

The word "impossible" is not in the dictionary

[Nizam89]

Failure is not an option

[taw]

Failure is not an option

Well, I can always release what I got. It already works just fine,
and if you happened to have 50GHz computer with 128 GB memory
you'd be totally comfortable with it.

Volunteers for alpha access welcome, but if you only run Windows,
you're on your own with setting up Ruby dev environment
(it works out of the box on osx and Linux)


While you wait, here's a fresh portion of notes.
The most recent ESFEditor sources got about half of them right.
It still worked as ESF lets you skip over data you don't understand easily.
Everything just makes so much more sense when you know
something is vec3s and something else is 2d byte pair array,
instead of just binary blob, binary blob, binary blob.

ESF types system is much less messy than initially seemed.
Lower nibble is type, upper nibble is container type.

Absolutely certain types are - and notice regularity:
* 0x01 0x41 - boolean; array of booleans
* 0x04 0x44 - int32; array of int32s
* 0x06 0x46 - byte; array of bytes
* 0x08 0x48 - uint32; array of uint32s
* 0x0a 0x4a - float; array of floats
* 0x0c 0x4c - vec2; array of vec2s
* 0x0d 0x4d - vec3; array of vec3s
* 0x0e - utf-16-le string
* 0x0f - ascii string

And record types:
* 0x80 - record [uint16 type tag; uint8 version; uint32 end_ofs; uint32 element_count; arbitrary tagged data]
* 0x81 - array of record [uint16 type tag; uint8 version; uint32 end_ofs; uint32 element_count]
each element is [uint32 elem_end_ofs; arbitrary tagged data]

One very interesting find is:
* 0x42 - array of byte pairs [!!!] - for wind map mostly, (x wind byte, y wind byte) or something like it

A few big unknowns left:

These types are definitely 16 bit, but signedness is uncertain, and what is 0x10?
* 0x00 - [u?]int16
* 0x07 - [u?]int16
* 0x10 - [u?]int16
* 0x47 - array of [u?]int16s

Some records like especially CAMPAIGN_LOCALISATION have extremely peculiar contents.
It seems to be either one non-empty unicode string,
or two empty unicode strings (0x0e 0x00 0x00 0x0e 0x00 0x00).

A lot of records have variable content, often like:
- [true] X or
[false]
Or like:
- [some int type count] [count times X]

But they are not "real" ESF arrays you'd expect (0x81 or 0x4?).

Entire poi.esf is huge blob of those, first approximation:

• u4(entries count)
  
  • u4(serial_id)
  • i4(some type id?)
  • bool
  • i4
  • i4
  • str(name),u4(?)
  • str(name),u4(?) repeated for no reason?
  • flt
    
    • i4(neighbours_count)
    • str(neighbour_name)
    • flt
  • flt
  • u4
  • u4
  • u4
  • bool

It seems that esf has far too many options, so they picked one at random every time.