Page 1 of 17 1234567891011 ... LastLast
Results 1 to 20 of 334

Thread: Perfect ESF<->XML converter and revolution in ESF modding [Released!]

  1. #1
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,303

    Default Perfect ESF<->XML converter and revolution in ESF modding [Released!]

    So now that every single DB table except one is moddable, the next step is ESF.

    Current ESF architecture makes even the simplest changes really difficult, and you're only allowed one change - there's no esf merging ever (hybrid startpos.esf tricks aside).

    I didn't like any of it, so I wrote a pair of scripts that convert ESF to XML, and then XML back to ESF. ESF->XML->ESF round trip is perfect to the very last byte on startpos.esf, regions.esf, and every other esf I tested.

    Once data is in XML format, all sort of mass modifications like removing every single fort (db-based no forts mod horribly confuse AI, esf-based no forts mod don't have such problems), flipping factions from minor to major, changing character skills and abilities, giving every minister a mistress, changing towns minor to major, and many more change from long painful effort to trivial find-and-replace-via-xquery, and a brave new world of far more interesting modding becomes wide open - including all CAI stuff that hides inside ESF for no apparent reason, and maybe even new campaign map because why the hell not.

    This is good kind of XML - there are binary offsets, or other nonsense in it - conversion scripts handle all that, you can just go ahead and edit.

    Right now the most serious practical problems are:

    • These XMLs are really huge - startspos.xml is 473MB (75% of that is CAI-related stuff, not even binaries). I should be able to cut it at least in half without losing anything of value, but it will still need a really good computer.
    • Conversion takes about 4 minutes each way. 4 minutes is nothing compared to an evening of trying to get things done with ESFEditor, but it should be faster.
    • It's all Ruby code that might be hard to get running on Windows. I'll write some detailed instructions at least.

    Once these three are done I'll publish the scripts, hopefully in a few days and together with collection of easy to use recipes for making amazing things with ESFs.

    If you want early alpha access, ask me on PM or email, but it's still quite messy, and stress testing has been limited.
    Last edited by taw; November 06, 2010 at 02:15 PM.

  2. #2
    The Hedge Knight's Avatar Fierce When Cornered
    Artifex

    Join Date
    Oct 2009
    Location
    England
    Posts
    5,875

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    Awesome news!

    Great stuff with db editor (shame its not fully compatible with ntw though )

  3. #3
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,303

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    Quote Originally Posted by 'The Hedge Knight View Post
    Awesome news!

    Great stuff with db editor (shame its not fully compatible with ntw though )
    Well, I don't even have NTW.

    If anyone's is in giving mood, my steam account is here -> http://steamcommunity.com/profiles/76561198021187429/

    Unless there were some drastic changes between ETW and NTW,
    extending my analysis to work with NTW should take little time once I have access to it.
    I cannot make any hard promises, but I see no reason why it shouldn't
    be straightforward after everything I've done to ETW already.

    (I tend to ignore *TW games altogether for the first year or two,
    they're all awful before at least the third patch).

  4. #4
    The Hedge Knight's Avatar Fierce When Cornered
    Artifex

    Join Date
    Oct 2009
    Location
    England
    Posts
    5,875

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    Ntw is awesome and has had 5 patches .

    Also its possible to make some tables for ntw work using db editor if you stick a blank file names empire.exe (txt) in data and use that as the path... Only wanted to look at models artillery as most i need are covered by pfm.

  5. #5
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,303

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    If it's just sticking blank empire.exe, it's not a big deal - for example Steam version of M2TW and Kingdoms is broken and requires similar silliness as well.

    Could you quickly check how well does the new DB.xsd work with NTW, if it's not much work?

  6. #6
    Indefinitely Banned
    Join Date
    Jan 2009
    Location
    Somewhere
    Posts
    12,379

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    Does this means that we will be able to properly mod ETW like MTW2?

  7. #7
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,303

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    Quote Originally Posted by Kaiser Of Crunk View Post
    Does this means that we will be able to properly mod ETW like MTW2?
    That has been my intention all along. As far as I can tell most things in ETW are not so much hardcoded, just very difficult to access without proper tools, and CA is definitely not being friendly.

  8. #8
    Primergy's Avatar Protector of the Union
    Join Date
    Mar 2009
    Location
    Augsburg
    Posts
    2,469

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    I'm looking forward.

    Btw...Would it be possible for your converter to convert the information of rigidsplines (which are used to place roads, rivers and borders) to something esf can handle?
    Actual if it you change the position it's only a visual effect.

    Does this means that we will be able to properly mod ETW like MTW2?
    If it would be already possible to handle the esf files much smarter, would be already a huge step forward
    Last edited by Primergy; September 27, 2010 at 03:34 PM.

  9. #9
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,303

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    Quote Originally Posted by Primergy View Post
    I'm looking forward.

    Btw...Would it be possible for your converter to convert the information of rigidsplines (which are used to place roads, rivers and borders) to something esf can handle?
    Actual if it you change the position it's only a visual effect.
    Is this the rigid splines converter you're looking for?

    regions.esf has massive float arrays region_data/vertices and
    many int32 arrays with indexes into it (I think indexes are of vec2s, so you need to double them).

    I attached a snippet of xml created out of regions.esf. Does it look like it might contain what's needed?
    Doing vector graphics from xml is not really optimal, but I bet regions.esf SVG export tool would only take an evening or two to a dedicated person (import probably more).

  10. #10

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    If this site like to maintain "Good" realations with CA And Sega then why can't they maintain Good realations likewise and realese modding tools?

  11. #11

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    Quote Originally Posted by taw View Post
    Current ESF architecture makes even the simplest changes really difficult, and you're only allowed one change - there's no esf merging ever (hybrid startpos.esf tricks aside).
    (...)
    Once data is in XML format, all sort of mass modifications like removing every single fort (db-based no forts mod horribly confuse AI, esf-based no forts mod don't have such problems), flipping factions from minor to major, changing character skills and abilities, giving every minister a mistress, changing towns minor to major, and many more change from long painful effort to trivial find-and-replace-via-xquery, and a brave new world of far more interesting modding becomes wide open
    I am not sure what you mean. You can make complex changes, export and import parts of esf, etc., using the Editor.

    On the other hand any development in esf editing is mostly welcome though I don't understand why XML method will make it easier. You have to have in mind a couple of things though. Editor saves changes instantly or in a few seconds. When you have to make complex changes is wise to make them in stages testing everyone first before you proceed in next one. The referred converting times are too large and must be reduced significantly, or ot will be not practical at all using converter.

    If it is not by default it should be necessary to be able as well the conversion of certain parts which first have been exported by the editor. For example to change a faction from minor to major it is only a few seconds using editor. But to relocate an army it is a complex work and it will be indeed easier if I can have in XML the necessary startpos sub-records (CHARACTERS, ARMY_ARRAYS, CAI_RESOURSES_MOBILE, etc.) available on my desktop. I can see a dynamic combination of an editor and a quick converter which perhaps will make esf editing easier. And I can assume that as smaller the esf is so faster the conversion will be.

    including all CAI stuff that hides inside ESF for no apparent reason, and maybe even new campaign map because why the hell not.
    There are CAI fields like MANAGERS, BDI, etc that your idea might be very helpful. Though we've managed to advance the research in some of those areas, these endless lists are not practically at all in editing. I expect the same endless lists in XML but on this case will be all data on sight than jumping from a sub folder to another (there are some thousands of them). The same stands for campaign files like poi.esf, pathfinder.esf, etc.

    If you want early alpha access, ask me on PM or email, but it's still quite messy, and stress testing has been limited.
    I defiantly like to be a part of this effort and have access in the stuff you've got under development. I work with esf files all the time and I can help -at least- in stress testing.

    Last I want to underline that EsfEditor 1.5 (still in beta) is a remarkable tool, very fast, uses labels and can edit Napoleon's files as well. If anyone can help Erasmus to complete it it would be good to communicate with him.
    Last edited by husserlTW; September 27, 2010 at 04:50 PM.




  12. #12
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,303

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    Quote Originally Posted by husserlTW View Post
    I am not sure what you mean. You can make complex changes, export and import parts of esf, etc., using the Editor.
    This is slow, painful, error-prone, manual work. Once I spent two hours manually removing every fort from game in startpos.esf. Then I found someone else made all factions including protectorates fully playable with working diplomacy, and I wanted that as well. How long would it take to merge both changes?

    Exactly as long as making one of them from scratch.

    With text-based formats it's just run diff, run patch, finished. Amazingly easy.

    How can I even as little as search for all settlements with fortifications in ESF Editor, let alone do search&replace?

    When all of it is in nice text, Cmd-F settlement_fortifications, and for every block like:

    <record flags='1' name='BUILDING_MANAGER'>
    <true />
    <BUILDING flags='1' percent='100' building='settlement_fortifications' faction='knights_stjohn' government='gov_absolute_monarchy' />
    <false />
    </record>

    Delete and paste over a block like:

    <record flags='1' name='BUILDING_MANAGER'>
    <false />
    <false />
    </record>

    Done. Actually I'm not quite sure if interpretation of flags and booleans used by ESFEditor and by my script is quite correct - they seem to have something to do with optional fields - I'll try finding out what's going on later.

    Quote Originally Posted by husserlTW View Post
    On the other hand any development in esf editing is mostly welcome though I don't understand why XML method will make it easier. You have to have in mind a couple of things though. Editor saves changes instantly or in a few seconds. When you have to make complex changes is wise to make them in stages testing everyone first before you proceed in next one. The referred converting times are too large and must be reduced significantly, or ot will be not practical at all using converter.
    Editor saves changes quickly only if it file size doesn't change, otherwise (as with deleting fortifications) it is very slow.

    Current converting times are 3+4 minutes round trip for startpos.esf, but that's due to weird binary fields I'm serializing and deserializing as hex mostly. regions.esf is 10% size of startpos.esf, but takes 10 for round trip.

    Converter can skip parts of xml for super-fast operation, but there's no way to merge such partial xmls yet. I hope to make this unnecessary and make it always fast.

    Quote Originally Posted by husserlTW View Post
    If it is not by default it should be necessary to be able as well the conversion of certain parts which first have been exported by the editor. For example to change a faction from minor to major it is only a few seconds using editor. But to relocate an army it is a complex work and it will be indeed easier if I can have in XML the necessary startpos sub-records (CHARACTERS, ARMY_ARRAYS, CAI_RESOURSES_MOBILE, etc.) available on my desktop. I can see a dynamic combination of an editor and a quick converter which perhaps will make esf editing easier. And I can assume that as smaller the esf is so faster the conversion will be.

    There are CAI fields like MANAGERS, BDI, etc that your idea might be very helpful. Though we've managed to advance the research in some of those areas, these endless lists are not practically at all in editing. I expect the same endless lists in XML but on this case will be all data on sight than jumping from a sub folder to another (there are some thousands of them). The same stands for campaign files like poi.esf, pathfinder.esf, etc.
    Quote Originally Posted by husserlTW View Post
    I defiantly like to be a part of this effort and have access in the stuff you've got under development. I work with esf files all the time and I can help -at least- in stress testing.

    Last I want to underline that EsfEditor 1.5 (still in beta) is a remarkable tool, very fast, uses labels and can edit Napoleon's files as well. If anyone can help Erasmus to complete it it would be good to communicate with him.
    Let's move this part over email. I'll PM you.

  13. #13
    hip63's Avatar Give me some BASS!
    Patrician Artifex

    Join Date
    Nov 2007
    Location
    groovy galaxy
    Posts
    3,716

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    Sounds pretty dang awesome!

    and I love the attitude: and maybe even new campaign map because why the hell not.

    hip63

  14. #14
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,303

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    Quote Originally Posted by taw View Post
    Right now the most serious practical problems are:

    • These XMLs are really huge - startspos.xml is 473MB (75% of that is CAI-related stuff, not even binaries). I should be able to cut it at least in half without losing anything of value, but it will still need a really good computer.
    • Conversion takes about 4 minutes each way. 4 minutes is nothing compared to an evening of trying to get things done with ESFEditor, but it should be faster.

    I got it down to 291M and 2 minutes from esf->xml, and 3m20s xml->esf for whole startpos.esf - on a good laptop.

    I cannot think of any serious technical reason why it needs to be worse than 1 minute each way and about 2x overhead (~150 MB). As you only need to convert once before you start a game, 1 minute would really not be a big deal. The only thing it needs is sufficiently many rounds of profiling, improving, and re-profiling.

    I could get there in 5 minutes by just throwing out large blobs of binary data of unknown functionality aside, but this is precisely where most interesting modding opportunities are likely to reside.

    That mystery flag seems to be version number, just like in db. They're mostly 0s, a lot of 1s, then a few 2s, and even fewer and fewer but it reaches as high as 13. It probably needs to be per-entry not per-file to make new versions of the game work with old save games.


    By the way here's a screenshot of svg I quickly made from regions.esf. Is this actual map of the world or not? ETW already has a test area, between Louisianas and Natives for which map says one thing visually but it's different really, and by a few thousand kilometers.

    If this one won't work, there are others to check elsewhere.

    (low quality picture likely due to unrelated data getting in a way of my regular expressions,
    that's why it's important for XML to have high quality schema, not merely to make the data available)

    Who's the first volunteer to try enlarging Guyanas?

    Spoiler Alert, click show to read: 




    Oh, and position technical explanation: all data in regions.esf uses indexes stuffed in integer arrays. Integers [17, 105, 92] all refer to one global vertices table of floats, and it means x0=float34, y0=float35, x1=float210, y1=float211, x2=flota184, y2=float185. (Y is upside down, coordinates normalized to x=-1280..1280 y=-640..640, but obviously you knew that much.

    The first person to create a new region (without destroying another) gets an award.

  15. #15
    Indefinitely Banned
    Join Date
    Jan 2009
    Location
    Somewhere
    Posts
    12,379

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    I know right people would could try I'll PM him.

  16. #16

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    Lets make a region and show those ers that we mean buisness!

  17. #17
    The Hedge Knight's Avatar Fierce When Cornered
    Artifex

    Join Date
    Oct 2009
    Location
    England
    Posts
    5,875

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    :0... I still think regions are a long way off (with tables to be edited and other esfs) but... Nice...

  18. #18

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    The word "impossible" is not in the dictionary

  19. #19
    Nizam89's Avatar Ducenarius
    Join Date
    Mar 2009
    Location
    Bora Bora
    Posts
    925

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    Failure is not an option
    "I warn every animal on this farm to keep his eyes very wide open."
    Pig Schwatzwutz
    NASA's biggest blunder was not having Neil Armstrong say, " That's one small step for man,....hey, that looks like gold!"


  20. #20
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,303

    Default Re: Coming soon - Perfect ESF<->XML converter and revolution in ESF modding

    Quote Originally Posted by Nizam89 View Post
    Failure is not an option
    Well, I can always release what I got. It already works just fine,
    and if you happened to have 50GHz computer with 128 GB memory
    you'd be totally comfortable with it.

    Volunteers for alpha access welcome, but if you only run Windows,
    you're on your own with setting up Ruby dev environment
    (it works out of the box on osx and Linux)


    While you wait, here's a fresh portion of notes.
    The most recent ESFEditor sources got about half of them right.
    It still worked as ESF lets you skip over data you don't understand easily.
    Everything just makes so much more sense when you know
    something is vec3s and something else is 2d byte pair array,
    instead of just binary blob, binary blob, binary blob.
    ESF types system is much less messy than initially seemed.
    Lower nibble is type, upper nibble is container type.

    Absolutely certain types are - and notice regularity:
    * 0x01 0x41 - boolean; array of booleans
    * 0x04 0x44 - int32; array of int32s
    * 0x06 0x46 - byte; array of bytes
    * 0x08 0x48 - uint32; array of uint32s
    * 0x0a 0x4a - float; array of floats
    * 0x0c 0x4c - vec2; array of vec2s
    * 0x0d 0x4d - vec3; array of vec3s
    * 0x0e - utf-16-le string
    * 0x0f - ascii string

    And record types:
    * 0x80 - record [uint16 type tag; uint8 version; uint32 end_ofs; uint32 element_count; arbitrary tagged data]
    * 0x81 - array of record [uint16 type tag; uint8 version; uint32 end_ofs; uint32 element_count]
    each element is [uint32 elem_end_ofs; arbitrary tagged data]

    One very interesting find is:
    * 0x42 - array of byte pairs [!!!] - for wind map mostly, (x wind byte, y wind byte) or something like it

    A few big unknowns left:

    These types are definitely 16 bit, but signedness is uncertain, and what is 0x10?
    * 0x00 - [u?]int16
    * 0x07 - [u?]int16
    * 0x10 - [u?]int16
    * 0x47 - array of [u?]int16s

    Some records like especially CAMPAIGN_LOCALISATION have extremely peculiar contents.
    It seems to be either one non-empty unicode string,
    or two empty unicode strings (0x0e 0x00 0x00 0x0e 0x00 0x00).

    A lot of records have variable content, often like:
    - [true] X or
    [false]
    Or like:
    - [some int type count] [count times X]

    But they are not "real" ESF arrays you'd expect (0x81 or 0x4?).

    Entire poi.esf is huge blob of those, first approximation:

    • u4(entries count)
      • u4(serial_id)
      • i4(some type id?)
      • bool
      • i4
      • i4
      • str(name),u4(?)
      • str(name),u4(?) repeated for no reason?
      • flt
        • i4(neighbours_count)
        • str(neighbour_name)
        • flt
      • flt
      • u4
      • u4
      • u4
      • bool


    It seems that esf has far too many options, so they picked one at random every time.
    Last edited by taw; September 29, 2010 at 11:42 AM. Reason: cleaned up formatting

Page 1 of 17 1234567891011 ... LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •