Page 1 of 3 123 LastLast
Results 1 to 20 of 51

Thread: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

  1. #1
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,308

    Default Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    For way too long editing esf was slow and painful process. I remember spending something like two hours just to remove all settlement fortifications.

    Then came esf2xml, and it became a 15 minute text editor search and replace job, after a bit of practice.

    Now comes the next logical step in esf editing - fully automatic transformations.
    XML is just so damn easy to convert with scripts, why even bother with text editors?
    And thanks to husserlTW's gui for esf editing soon you won't even need any command line.

    How they work

    Esf transformation scripts work on xml form, so what you need to do is:

    • convert esf to xml
    • run all scripts you want one after another
    • convert xml to esf back

    Scripts modify directory full of unpacked xmls in place.

    Available scripts

    Available scripts so far (a lot more coming soon):

    Faction editing - most commands accept either faction name or "*" to apply to all factions:
    • make one faction playable (doesn't fix fog of war bug itself, may crash if you try to play emergent or rebel factions etc. - the usual caveats apply)
    • make all vanilla non-emergent campaign minor factions playable
    • make one/all faction major
    • make faction minor
    • change faction's religion
    • change faction's capital region (by name obviously, script handles name to id mapping automatically)
    • change one/all faction's starting treasury
    • give faction one technology (research points is db-dependent, but doesn't seem to matter, so it's just set to status=0, research_points=1000.0), or multiple technologies specified by pattern like military_*
    • give faction all possible technologies (awesome fun for starting as tiny faction)


    Reporting:
    • List all factions
    • List id by faction
    • List ids and cai ids by region
    • List starting (startpos.esf) or current (for save games) treasury by faction
    • Draw regions from regions.esf


    Others:
    • remove all settlement fortifications from whole map - the very thing which made me start esf<->xml converter
    • change turns per year
    • remove current resistance to foreign occupation (doesn't prevent it in for future conquests)
    • reset diplomacy for all factions

    Almost all "level 1" modifications from husserlTW's list have a script ready, except starting date (how is it supposed to work again?) and victory conditions (these have very easy to use xmls in victory_conditions/*, and no obvious scripting interface).

    Plus a few things more complicated than that.

    New scripts, and requests for more scripts eagerly solicited.

    You're not limited to just startpos.esf - any esf can be edited - including savegames!

    I'm sure people who try to open campaign map will find some uses for these scripts as well.

    Usage example

    $ ./esf2xml '/media/c/Program Files (x86)/Steam/steamapps/common/empire total war/data/campaigns/main/startpos.esf' vanilla_startpos
    $ ./scripts/remove_all_fortifications vanilla_startpos
    $ ./xml2esf vanilla_startpos startpos_without_fortifications.esf

    And this used to take a few hours, imagine that!

    How to write your own scripts
    But wait, there's more!

    Unlike esfxml code which is pretty hardcore, esf transformation scripts are ridiculously simple.
    Here's one for making chosen faction major (ignoring some boiletplate):

    class MakeFactionMajor < EsfScript
    def run!(faction_to_change)
    update_faction(faction_to_change) do |faction|
    faction.xpath("yes|no")[2].name = 'yes'
    true
    end
    end
    end


    If you know even a tiny bit of Ruby or Python, or anything vaguely similar,
    and have even heard of xml and xpath, it will be no problem whatsoever
    to write a script like that yourself.

    Not everything can be achieved so easily - but a lot of thing can.

    About the only non-trivial aspect of it is that you need to return true if you modified dom tree, or false otherwise, to avoid unnecessary xml re-saving.

    Also node.name means xml tag name (<foo>); node['foo'] means attribute (foo='bar'); and node.content means inner text (<foo>x</foo>)

    I'll write tutorial for writing such scripts if there's demand for it.

    Requirements

    Scripts require just what esfxml needs - Ruby or JRuby and nokogiri gem (jgem install nokogiri --pre). It runs on pretty much any operating system.

    If you modify xml_split.rb too much, scripts might not be able to find relevant xml files to change. It's better to use it with default xml_split.rb

    All scripts are available together with esfxml from etwng repository.

    Good modding!
    Last edited by taw; February 20, 2011 at 02:16 PM. Reason: diplomacy reset

  2. #2

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    Great ! I might try this out after my exams end

  3. #3
    Okmin's Avatar In vino veritas
    Join Date
    May 2010
    Location
    USA
    Posts
    7,506

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    taw, you are a genius. +rep!

    PS: Doesn't work with husserITW's GUI, right?
    Last edited by Okmin; January 28, 2011 at 08:33 PM.
    IN VINO VERITAS
    IN CERVESIO FELICITAS

    Under the patronage of The Lizard King
    Patron of Narf
    and Starlightman

  4. #4
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,308

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    Quote Originally Posted by Okmin View Post
    taw, you are a genius. +rep!

    PS: Doesn't work with husserITW's GUI, right?
    Not yet. husserlTW liked the idea of scripts a lot, so he will probably add support for them to the GUI tool.

  5. #5
    Okmin's Avatar In vino veritas
    Join Date
    May 2010
    Location
    USA
    Posts
    7,506

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    Thanks
    IN VINO VERITAS
    IN CERVESIO FELICITAS

    Under the patronage of The Lizard King
    Patron of Narf
    and Starlightman

  6. #6

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    +rep
    Taw do you believe that in the future you will be able to do the same to things like add more slots to cities?
    Last edited by ForteS; January 29, 2011 at 07:51 AM.

    UNDER THE MOST HONORABLE PATRONAGE OF: Legio!
    PATRON OF: Wangrin, ♔Sir Digby Chicken Caesar♔, Geronimo2006 and Narf!

  7. #7

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    If we had a coder with VB xml editing knowledge we could have it in windowed easily. With Taw's converter all startpos changes are actually xml editing process. I am reading about it but I am very new to VB and I have to get used the basics first...




  8. #8
    Inevitability won
    Patrician Citizen

    Join Date
    Mar 2010
    Posts
    9,594

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    I'm quite fluent in VB. All be it 6, pre .NET, the first language I think I ever learned. Though I know very little about doing anything with XMLs with it, that said I do know my way around the XML themselves. I'll read up on editting XML nodes in VB, see if I can grasp the concept quickly.

  9. #9
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,308

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    Quote Originally Posted by Fortes View Post
    +rep
    Taw do you believe that in the future you will be able to do the same to things like add more slots to cities?
    Yes, that's part of the plan. It might even happen fairly soon.

    Quote Originally Posted by .Mitch. View Post
    I'm quite fluent in VB. All be it 6, pre .NET, the first language I think I ever learned. Though I know very little about doing anything with XMLs with it, that said I do know my way around the XML themselves. I'll read up on editting XML nodes in VB, see if I can grasp the concept quickly.
    Quote Originally Posted by husserlTW View Post
    If we had a coder with VB xml editing knowledge we could have it in windowed easily. With Taw's converter all startpos changes are actually xml editing process. I am reading about it but I am very new to VB and I have to get used the basics first...
    By the way, all converters use JRuby, so they run on JVM, so if you used some Java-based GUI they could interact with converters a lot better. And I could use them on Mac ;-p. But if VB is much easier, go for it.

  10. #10
    Inevitability won
    Patrician Citizen

    Join Date
    Mar 2010
    Posts
    9,594

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    lol ok guys, In my ESF modding career of NTW, I'v needed to turn every faction in the campaign neutral quite often, so that I could set my own diplomacy .etc

    We've gone from having to change thousands of variables by hand in the ESF editor, to being able to do it in about 100 or so Find and Replace queries in Notepad, and we are now at one click.

    For each faction diplomacy XML, there are about 4300 variables that need to be changed to make that faction completely neutral, there are 40 XMLs, which means overall and for diplomacy only there are about 170,000 variables that used to be changed by hand, now all done in less than a minute. Makes me feel sad almost. (God knows how many there are with ETW, and all those factions)

    So as a test to see if I could do anything usefull, I'v made a campaign neutrality exe.

    Drag one or all of the diplomacy.xml's onto the exe and it will output an identical XML in an "XMLOutput" folder the only difference being that its turned the diplomacy for that faction to neutral with everyone.

    I was going to make it turn every XML file in the same directory as the exe to neutral, but I was rushing, so I just made it drag and drop instead.

    I'll no doubt expand it in future, as the diplomacy is one of the biggest aspects of the ESF editing that needs far more automation.
    That said, I'll no doubt create many more of these that do many various things.

    (It may very well be the case there is an extra byte in there somewhere that isn't meant to be there (like i said I rushed very much just to see if I could do anything). It was the case that after the procress the second xml picked up two bytes at the end of the xml, but I dealt with that. So yeh, completely untested atm, but at a glance over the xml it looks like it works.)

    Oh and thanks to Aanker for creating icon's on whim whenever I ask.
    Last edited by .Mitch.; January 29, 2011 at 03:20 PM.

  11. #11
    Okmin's Avatar In vino veritas
    Join Date
    May 2010
    Location
    USA
    Posts
    7,506

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    +rep to you too
    IN VINO VERITAS
    IN CERVESIO FELICITAS

    Under the patronage of The Lizard King
    Patron of Narf
    and Starlightman

  12. #12
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,308

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    Quote Originally Posted by .Mitch. View Post
    lol ok guys, In my ESF modding career of NTW, I'v needed to turn every faction in the campaign neutral quite often, so that I could set my own diplomacy .etc

    We've gone from having to change thousands of variables by hand in the ESF editor, to being able to do it in about 100 or so Find and Replace queries in Notepad, and we are now at one click.

    For each faction diplomacy XML, there are about 4300 variables that need to be changed to make that faction completely neutral, there are 40 XMLs, which means overall and for diplomacy only there are about 170,000 variables that used to be changed by hand, now all done in less than a minute. Makes me feel sad almost. (God knows how many there are with ETW, and all those factions)

    So as a test to see if I could do anything usefull, I'v made a campaign neutrality exe.

    Drag one or all of the diplomacy.xml's onto the exe and it will output an identical XML in an "XMLOutput" folder the only difference being that its turned the diplomacy for that faction to neutral with everyone.

    I was going to make it turn every XML file in the same directory as the exe to neutral, but I was rushing, so I just made it drag and drop instead.

    I'll no doubt expand it in future, as the diplomacy is one of the biggest aspects of the ESF editing that needs far more automation.
    That said, I'll no doubt create many more of these that do many various things.

    (It may very well be the case there is an extra byte in there somewhere that isn't meant to be there (like i said I rushed very much just to see if I could do anything). It was the case that after the procress the second xml picked up two bytes at the end of the xml, but I dealt with that. So yeh, completely untested atm, but at a glance over the xml it looks like it works.)

    Oh and thanks to Aanker for creating icon's on whim whenever I ask.
    How neutral would you like them to be? Just historical grievances/war/alliance/protectorate/etc.,
    or all the way to turning off trade, religion bonuses and so on?

    Could you send me the source? Small details of XML files produced by esfxml change very frequently,
    so I like being able to update all scripts in one place.
    I think I even renamed attributes for draa from a/b/c/d/e/f to something meaningful a few days ago ;-)

  13. #13

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    I agree with Taw. Some elements like religion, type of government, etc. cannot be neutral. I imagine a script about changing diplomacy more or lees like the way I make those modifications manually: I copy an existing DIPLOMACY_RELATION record similar to the one I want for the new status and I change faction Id (of course I delete the old one). So in a scripting automate way I think there could be a pool of DIPLOMACY_RELATION records in xml stored in a folder for the several status and the proper one would replace the existing D_R of the factions (always in pairs this kind of changes). Taw, have a look on my Patron/Protectorate tutorial (Level 2.8) and grab the attached records (Empire and Napoleon). I think that you can easily make this change scripted.

    When I transfer this in a windowed application, more complex situations can be made e.g. neutral with trade or not, by having a second pool with parts of a D_R record like trade, historical relations, type of religion and government, etc. with the proper scripts. Last even the level of these relations could be adjusted by changing the proper number in D_R sub-records.
    Last edited by husserlTW; January 30, 2011 at 05:30 AM.




  14. #14

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    Hello taw.

    First of all congratulations.

    I was wondering how difficult would result for you to write a script for campaign-pathfinder/grid-nnnn.xml that looks more or less like this in pseudo-code (id is the only argument):

    Code:
    // script #1
    
    if u4_ary(PATHFINDING_GRID/OBSTACLE_BOUNDARY_MANAGER/OBSTACLE_BOUNDARY[n]/0).contains(id+2) then
        
        ary(PATHFINDING_GRID/OBSTACLE_BOUNDARY_MANAGER/OBSTACLE_BOUNDARY).remove_by_pos(n)
    
    // script #2
    
    if u4_ary(PATHFINDING_GRID/OBSTACLE_BASE_GRID_NODE[n]/MANAGED_OBSTACLE_BOUNDARY[i]/last_value_or_first_of_type_u4_ary).contains(id+2) then
    
        id = u(PATHFINDING_GRID/OBSTACLE_BASE_GRID_NODE[n]/MANAGED_OBSTACLE_BOUNDARY[i]/0)
        
        // maybe remove something else with id in bdi_pool? don't know atm.
    
        ary(PATHFINDING_GRID/OBSTACLE_BASE_GRID_NODE[n]/MANAGED_OBSTACLE_BOUNDARY).remove_by_pos(i)
        
    // script #3
    
    if u4_ary(PATHFINDING_GRID/OBSTACLE_LISTS/2).contains(id) then
    
        u4_ary(PATHFINDING_GRID/OBSTACLE_LISTS/2).remove_by_val(id)
        
    if u(PATHFINDING_GRID/OBSTACLE_LISTS/CHARACTER_OBSTACLE[n]/1).equal_to(id) then
    
        ary(PATHFINDING_GRID/OBSTACLE_LISTS/CHARACTER_OBSTACLE).remove_by_pos(n)
    I'm making some experiments for removing a whole army in startpos through your great esfxml tools.
    But I need to cleanup all dependencies from there of course (identifier's 2 lsbits change, that's the reason of id+2).
    If it's not too much time-consuming you would make me a huge favour. Otherwise never mind.
    The alternative is C + a xml library or learning jruby... my free time wouldn't be enough for both I think.

    Thank you in advance and good luck with all your projects.
    risorgimento

  15. #15
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,308

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    A bunch of new scripts are available. By the way esf transformation scripts for startpos.esf can be used
    on save games just as easily. husserlTW wrote about hybrid startpos.esf/savegame technique before,
    it's probably even easier with xmls than with EsfEditors.

    Anyway, only by using list_starting_treasury script I've noticed how damn unfair DMUC is towards minor faction.

    Here's DMUC 6.8 + CB Light 2.0 beta (+ forts removed, no settlement mortars, and a few minor changes,
    but nothing else about campaign). Notice how major factions all managed to spend most of their starting ~37500-ish
    (and they have higher income per turn!!!), while minor factions have no way to spend that kind of money
    no matter how hard they try.

    This is just ridiculously unfair, isn't it?

    Output of ./scripts/list_starting_treasury out/hannover_1703_ams_ok/
    sorted for your convenience

    # Dummy faction, ignore
    : 0

    # Emergent, ignore
    afghanistan: 37500
    greece: 37500
    hessen: 37500
    hungary: 37500
    ireland: 37500
    mamelukes: 37500
    naples_sicily: 37500
    punjab: 37500
    quebec: 37500
    scotland: 37500
    united_states: 37500
    colombia: 37500
    mexico: 37500

    # Major (and major-ish) - all spent their money already
    austria: 4449
    britain: 5232
    denmark: 5584
    france: 3627
    maratha: 6754
    mughal: 4594
    netherlands: 4678
    new_spain: 4225
    ottomans: 2454
    poland_lithuania: 6927
    russia: 6513
    safavids: 7444
    spain: 992
    sweden: 4132
    prussia: 13792
    iroquoi: 10843
    thirteen_colonies: 4193

    # Somewhere in between
    barbary_states: 25625
    cherokee: 25554
    huron: 22154
    papal_states: 23857
    pirates: 26225
    portugal: 20433
    saxony: 23043

    # Minor - have money but cannot do anything with it
    bavaria: 33375
    chechenya_dagestan: 41515
    courland: 42449
    crimean_khanate: 32651
    genoa: 30505
    georgia: 35897
    hannover: 30521
    inuit: 37090
    knights_stjohn: 41860
    louisiana: 30406
    morocco: 37364
    mysore: 35327
    norway: 37500
    piedmont_savoy: 40127
    plains: 32766
    pueblo: 30624
    venice: 30239
    westphalia: 36287
    wurttemberg: 40207

    Was this change of balance intended?

  16. #16

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    taw
    Great to see you still working on this.
    Can you run this script with DMUC 6.9 beta update8?
    http://www.twcenter.net/forums/showthread.php?t=420318

    Off course this was not intended but it is somewhat logical because minor factions with 1-2 provinces have no where to spend the money for development.

  17. #17
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,308

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    Quote Originally Posted by DARTH VADER View Post
    taw
    Great to see you still working on this.
    Can you run this script with DMUC 6.9 beta update8?
    http://www.twcenter.net/forums/showthread.php?t=420318

    Off course this was not intended but it is somewhat logical because minor factions with 1-2 provinces have no where to spend the money for development.
    Could you try it yourself?
    It's just a simple script (esf2xml on savegame, ./script/list_treasury_by_faction that/directory)
    and installing DMUC betas would make it hard to go back to my current campaigns, right?

    It's not really DMUC-specific problem. It's what you get from setting starting treasuries so high
    without providing more ways for spending this money.

    M2TW had similar problems, so I made everything 50% more expensive and built in 1 turn ;-)

    ETW cannot really do that, as you'd very quickly run out of buildings without having necessary technologies,
    or population, or fresh settlements.

    Do we really need to give everyone so much money?

    Or if so, could we give minor factions a lot more recruitment slots?
    I'd love some shenanigans to make minors able to recruit more than majors in same settlement
    (startpos.esf CAMPAIGN_BONUSES might allow just that), but just plain more recruitment slots
    for 2-slot cities would work too.

  18. #18
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,308

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    One small step on a way to getting rid of resistance to foreign occupation - there's now a script to remove currently existing resistance.

    Sadly when you conquer a new province, you'll need to rerun (esf2xml, scripts/remove_resistance_to_occupation, xml2esf) again :-(

    On the other hand this doesn't change esf size (and offsets - just readjusting offsets requires analyzing all parts of esf which we don't care about much),
    so it shouldn't be hard to write a simple script to just edit esf in place... Maybe I'll do it later.

  19. #19
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,308

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    Quote Originally Posted by risorgimento View Post
    Hello taw.

    First of all congratulations.

    I was wondering how difficult would result for you to write a script for campaign-pathfinder/grid-nnnn.xml that looks more or less like this in pseudo-code (id is the only argument):

    Code:
    // script #1
    
    if u4_ary(PATHFINDING_GRID/OBSTACLE_BOUNDARY_MANAGER/OBSTACLE_BOUNDARY[n]/0).contains(id+2) then
        
        ary(PATHFINDING_GRID/OBSTACLE_BOUNDARY_MANAGER/OBSTACLE_BOUNDARY).remove_by_pos(n)
    
    // script #2
    
    if u4_ary(PATHFINDING_GRID/OBSTACLE_BASE_GRID_NODE[n]/MANAGED_OBSTACLE_BOUNDARY[i]/last_value_or_first_of_type_u4_ary).contains(id+2) then
    
        id = u(PATHFINDING_GRID/OBSTACLE_BASE_GRID_NODE[n]/MANAGED_OBSTACLE_BOUNDARY[i]/0)
        
        // maybe remove something else with id in bdi_pool? don't know atm.
    
        ary(PATHFINDING_GRID/OBSTACLE_BASE_GRID_NODE[n]/MANAGED_OBSTACLE_BOUNDARY).remove_by_pos(i)
        
    // script #3
    
    if u4_ary(PATHFINDING_GRID/OBSTACLE_LISTS/2).contains(id) then
    
        u4_ary(PATHFINDING_GRID/OBSTACLE_LISTS/2).remove_by_val(id)
        
    if u(PATHFINDING_GRID/OBSTACLE_LISTS/CHARACTER_OBSTACLE[n]/1).equal_to(id) then
    
        ary(PATHFINDING_GRID/OBSTACLE_LISTS/CHARACTER_OBSTACLE).remove_by_pos(n)
    I'm making some experiments for removing a whole army in startpos through your great esfxml tools.
    But I need to cleanup all dependencies from there of course (identifier's 2 lsbits change, that's the reason of id+2).
    If it's not too much time-consuming you would make me a huge favour. Otherwise never mind.
    The alternative is C + a xml library or learning jruby... my free time wouldn't be enough for both I think.

    Thank you in advance and good luck with all your projects.
    risorgimento
    I'll probably do something like this soon, yes.

  20. #20
    Artifex
    Patrician

    Join Date
    Oct 2008
    Location
    London, UK
    Posts
    1,308

    Default Re: Esf Transformation Scripts - the most awesome thing in esf world since esf2xml

    The most recent script is reset_diplomacy.

    It cleans up diplomacy to almost blank slate - removes wars, alliances, protectorates (correctly afaik), trade routes, historical grievances etc.

    What is left unchanged:

    • relationship penalty due to religion and government (also automatically reapplied due to faction leaders)
    • relationships involving Pirates are not touched, as they're semi-hardcoded

    Current problems:

    • Armies do not get back to original borders - so stopping Maratha-Mughal war keeps their armies stuck forever (well, at least until war restarts)
    • Barbary states should probably stay at war with nearly everyone, just like pirates
    • AI's decision during the first few turns are scripted, reseting diplomacy doesn't really change them. You'll see very little original behaviour just by reseting diplomacy.

    The campaign is different, but not really different enough. If anybody knows how to change scripted wars, I'd so totally love to write a script which would set up random-ish initial diplomacy and random-ish scripted wars - to make every campaign at least somewhat different (and while I'm at it, giving characters random traits and ancillaries shouldn't be that hard). Random campaign on the same map is not quite what we want, but we should make the best use out of what we have.

    Quick question: is CAI_HISTORY (cai_interface/cai_history.xml), and in particular
    CAI_HISTORY_EVENT_INVASION_REQUESTED,
    the script for CAI behaviour for the first few turns or is it somewhere else?

Page 1 of 3 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •