Perfect ESF<->XML converter and revolution in ESF modding [Released!]

[uanime5]

By the way here's a screenshot of svg I quickly made from regions.esf. Is this actual map of the world or not? ETW already has a test area, between Louisianas and Natives for which map says one thing visually but it's different really, and by a few thousand kilometers.

ETW does have a world map, CA just choose to hide it. However by expanding the area of the Indian trade theatre I was able to travel over the Indochina and China regions (though they were still hidden).

[Peteriscoolz]

I'm not sur if it's possible but could you have a quick option for changing numbers with your new PFM for instance when editing unit limits it's a pain setting all of the to 0 could you please make an option where you can just make all lof them 0 without having to constantly scroll down and down...

[ForteS]

WOOOOW, this seems great
Do it quickly please

BTW you are right i'm a esf editor and i thing it very hard to mod, but with this it seems to be much easiest.
THIS IS THE FUTURE, THIS IS THE FUTURE

[super_newbie_pro]

Congratulations !! Thanks for all the communauty ! I think that we will have creazy mod in the futur, with your excellent work thanks very much and good luck for the futur of your programm

[ark2010]

Just browsing through the forums and stumbled on your hard work! Well done to you sir! Keep going, you are an absolute inspiration and every modder and mod player will recognise the achievement that will hopefully be made by you. You could make this half pillar of salt into something very interesting
Also big shout out to all that are helping with inspiration or tests, but in the main good luck and thank you to Taw for your ambition!

[taw]

So about quad-trees and other 2D/3D data.

I don't want to keep them with everything else - nobody will be editing them as xml (or from esf editor) manually anyway.

Just take a look what weird things hide inside startpos.esf:

Spoiler Alert, click show to read: 

Spoiler Alert, click show to read: 

Spoiler Alert, click show to read: 

These are rather uninteresting relative to effort it took (did CA give massive bonuses for whoever could invent largest number of file formats?). They might be related to minor faction fog of war bug perhaps, nothing more.

Hopefully my next find will be better.

[taw]

All right, one huge 500MB XML file was a silly idea. Now I'm generating a bunch of files - some XML, some bitmaps - they can be viewed and edited separately and then converter merges them into bit-by-bit perfect reconstruction of the original ESF.

Here are two such small files - ottomans faction from startpos.esf and mysore region from regions.esf

It would be a huge improvement if all vertex / faction / settlement / anything else ids were linked to target, or at least clearly labeled "something_id", but other than that how does it look already?

Also there's huge amount of graphical data is .esfs that I'd like to put in usable format. I can make script generate SVG easily enough (this XML, that XML, same difference), but reading SVG files modified from some editor is out of a question.

I saved some bitmaps (wind, climate, some campaign map stuff I have no idea what) as .pgm, and they're sort of understandable (and modifications go back to esf on rebuilding), but pgm is grayscale, and ones I found are:

• climate - 1 byte per pixel (win) - every color has separate meaning, it's not intensity (fail)
• wind - two (fail) values of intensity - each byte per pixel (win). Separate pictures for longitudinal and latitudinal wind seem silly, but I have no other ideas.
• campaign maps - 4 bytes per pixel - no idea what they are, or which simple format represents them well.

There are also outlines, faces, bounding boxes, quad bit trees, quad trees with more complicated data on leaves, so many kinds of data, and we often want to display and modify it together! Seems like a lot of work even after data is openly available.

People with a clue about graphics formats, and about running Ruby programs on Windows definitely needed. Especially if you can come up with some solution to graphics data that works for me on Mac as well ;-)

[Condottiere SOG]

Sweet! Keep up the Great Work!

[taw]

All right, it seems geographical programs like Qgis will be capable of dealing with ETW data.

On an off chance, anybody has relevant experience with Qgis or other GIS programs?

[The Hedge Knight]

Cant make 3d model converters too can you?

[taw]

Cant make 3d model converters too can you?

Which formats and which modeling programs are you thinking about now? They didn't seem to make much sense for maps when I looked at a few.

[taw]

It's in testdata.pack

From campaign.xml:

<?xml version="1.0" encoding="UTF-8"?>
<Model xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="campaign.xsd">
<calendar>
<start_year>1700</start_year>
<start_season>summer</start_season>
</calendar>
<World>
<Faction name="britain" starting_treasury="1000">
<Leader surname="Zenger" age="34" name="Samuel">
<Type>general</Type>
<Position x="-54" y="293"/>
<Army>
<Unit type="guard_infantry" soldiers="50"/>
<Unit type="guard_infantry" soldiers="50"/>
<Unit type="guard_infantry" soldiers="10"/>
<Unit type="guard_infantry" soldiers="10"/>
<Unit type="guard_infantry" soldiers="10"/>
<Unit type="guard_infantry" soldiers="10"/>
<Unit type="guard_infantry" soldiers="10"/>
<Unit type="guard_infantry" soldiers="50"/>
</Army>
<Child surname="Zenger" age="1" name="George">
<Type>spy</Type>
<Position x="-35" y="297"></Position>
</Child>
<Child surname="Zenger" age="1" name="Edward">
<Type>general</Type>
<Position x="-45" y="296"></Position>
<Army>
<Unit type="guard_infantry" soldiers="50"/>
<Unit type="guard_infantry" soldiers="50"/>
</Army>
</Child>
</Leader>
</Faction>
<Faction name="united_states" starting_treasury="2000">
<Leader surname="Hudson" age="34" name="Richard">
<Type>general</Type>
<Position x="-49" y="300"/>
<Army>
<Unit type="guard_infantry" soldiers="50"/>
<Unit type="guard_infantry" soldiers="50"/>
...

It even has damn schema right there:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" attributeFormDefault="unqualified">
<xs:element name="Model">
<xs:complexType>
<xs:sequence>
<xs:element name="calendar">
<xs:complexType>
<xs:sequence>
<xs:element name="start_year" type="xs:positiveInteger"/>
<xs:element name="start_season" minOccurs="0">
<xs:annotation>
<xs:documentation>Defaults to summer</xs:documentation>
</xs:annotation>
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:enumeration value="summer"/>
<xs:enumeration value="winter"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element name="rounds_per_year" type="xs:positiveInteger" minOccurs="0">
<xs:annotation>
<xs:documentation>Defaults to 2</xs:documentation>
</xs:annotation>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="World">
<xs:annotation>
<xs:documentation>Setup of the world</xs:documentation>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:element name="Faction" maxOccurs="unbounded">
<xs:complexType>
<xs:sequence>
<xs:element name="Leader" type="Character"/>
</xs:sequence>
<xs:attribute name="name" type="xs:string" use="required"/>
<xs:attribute name="starting_treasury" type="xs:positiveInteger"/>
</xs:complexType>
</xs:element>
...

Now esf has a lot more information, so it's either very early version, or there's something taking these simple xmls and turning them into complex esf data - a lot of esf data like cai and visibility quadtrees is definitely not manually edited.

But to everyone doubting that XML is the way to go - CA uses exactly that!

[Cursisanafou]

How far are you along in creating this tool? Your fresh take on modding ETW is nice... the release of NTW and the time it took to get anywhere with modding prior to that seemed to have killed the modding advancements.. until now.


ESF editor is a very nice tool there's no arguing that, but XML is very nice, and if it has a lot of functions it would be amazing... ESF editor stagnated with a half-completed project a long time ago. (public modding wise anyways)

[taw]

How far are you along in creating this tool? Your fresh take on modding ETW is nice... the release of NTW and the time it took to get anywhere with modding prior to that seemed to have killed the modding advancements.. until now.

ESF editor is a very nice tool there's no arguing that, but XML is very nice, and if it has a lot of functions it would be amazing... ESF editor stagnated with a half-completed project a long time ago. (public modding wise anyways)

Status:

• Perfect export and import of every ESF to low-level XML - 100% done
• Making this XML sufficiently high-level - 80% done
• Gathering map data from multiple sources into something moddable - stalled by problems with GIS and modelling tools, will be a lot faster if I get help with it.
• One-click Windows port with GUI - fffffffuuuuuuuu! I wanted to use JRuby as path of least resistance, and esf2xml part works, but JRuby has no libxml, so I'll need to rewrite some stuff to use another xml library; also I'll need to figure out how to make portable GUIs, I forgot how much of a fail Java GUIs are. This part is not fun
• Right now you can run converter both ways even on Windows if you install all relevant libraries. Or on OSX pretty much right away.

I'm uploading code that's more or less usable to github.

Most likely course of events:

• I'll need a few days without too much regular work/life stuff
• Then I'll need to figure out how the can I get it all running on Windows without too much trouble. As I learned from a game I wrote ages ago, "cross-platform" usually means it works perfectly everywhere except Windows, and you'll be getting support requests from Windows users for the rest of your life.
• Then I'll release low-to-mid-level esf<->xml converter, plus converters for a few other simple formats like farms, db<->tsv, etc. - the code is already there, so I can as well put it in package once ready.

All that needs about one successful day.

Now my requests for help were mostly unanswered, but I'll ask again. If anyone has a clue about any of:

• GIS software
• Other 2D/3D modelling software that can deal with vector maps
• JRuby cross-platform GUIs
• JRuby one-click packaging
• Windows programming in general - to make things like my luadec one click build script work on Windows.

It would be very much appreciated. Things stopping me are mostly not ETW-specific, it's Windows and serious lack of free time.

[Tellos Athenaios]

Some points:
(1) You could design the GUI forms in, say, NetBeans which creates pretty good GUI's in terms of resizing, alignment, so on and so forth. With a minimum of Java code you can expose the necessary code to populate the GUI and extract data from it; then simply import that code using whatever constructs (J)Ruby provides for it.
(2) You could write the JRuby code to generate SAX events in a JAXP style input source. The idea is that you get to/from XML conversion nearly for free. On the to side, the JRuby code could present the ESF format (whatever it is) as if it were an ordinary XML Source, to which the API then lets you apply Transformations (e.g. XSLT stylesheets) which massages this “XML” to be in whatever shape you want it to be, or dump it right to an output stream. On the from side the XML->ESF conversion is simply a SAX ContentHandler or similar which does the reverse of the ESF->XML conversion, in between the high and low level XML formats you can again have any number of “Transformations” as required.

Also, that way you do not need to worry about what specific XML crunching library the users have installed: that is a job for the authors of the Java/JVM package.

[taw]

XML library I was using was native library for (C)Ruby, not Java. I only went for JRuby as it seemed like the least complicated way to have it running on Windows.
I never used JRuby before, and I've never programmed anything for Windows, other than a few bad cross-compiles.

Transformations I need are too complicated for XSLT, and especially for streaming subsets of it.
In my unusually limited free time last weekend, I only managed to gather a bunch of detection and conversion code for a few formats
and have them work with JRuby, but no Jruby ESF<->XML yet. Any of these formats are interesting?
Code on etwng cannot export it to anything useful yet, but format specs are often adequate.

Most modders are not programmers, so I doubt most would know what to do with JAXP source.

[Tellos Athenaios]

XML library I was using was native library for (C)Ruby, not Java. I only went for JRuby as it seemed like the least complicated way to have it running on Windows.
I never used JRuby before, and I've never programmed anything for Windows, other than a few bad cross-compiles.

Well on the (J)Ruby part, neither have I. I do not have Empire Total War, or Napoleon Total War so I cannot really provide in-depth comments here.

Most modders are not programmers, so I doubt most would know what to do with JAXP source.

What I meant with a JAXP Input Source is simply an “object” which presents itself as XML document. JAXP is essentially a framework for massaging, transforming and converting XML in a portable manner on the JVM (and by extension on JRuby). So if the ruby code simply generates the right events (if you are mocking a SAX style input source) you can have the ESF file treated as XML and slot into a “pipeline” of transformations as necessary.

Of course if it must be fully compatible with (C)Ruby then this is not much of an option.

[taw]

Well on the (J)Ruby part, neither have I. I do not have Empire Total War, or Napoleon Total War so I cannot really provide in-depth comments here.

No ETW or NTW? I'm surprised to see you here then.

What I meant with a JAXP Input Source is simply an “object” which presents itself as XML document. JAXP is essentially a framework for massaging, transforming and converting XML in a portable manner on the JVM (and by extension on JRuby). So if the ruby code simply generates the right events (if you are mocking a SAX style input source) you can have the ESF file treated as XML and slot into a “pipeline” of transformations as necessary.

Unfortunately SAX is not quite enough, there's a lot of contextual information needed to make these XML look meaningful.

Where can I read about these transformations, mock sax etc.? I know Java enough to use it when I have to, but I'm not really up to date with everything going on in Java universe.

Also, a minor one - does all this JAXP business handle xinclude transparently out of the box? It's pretty neat feature of nokogiri library when having to deal with massive xml to just cut them into pieces behind the scenes.

Of course if it must be fully compatible with (C)Ruby then this is not much of an option.

It doesn't have to. GUI definitely won't be. Would be neat if converters themselves did.

Right now I'm automatically testing against (C/J)Ruby (1.8/1.9) - all found combinations. Not as bad as I expected.

[alhoon]

ETW does have a world map, CA just choose to hide it. However by expanding the area of the Indian trade theatre I was able to travel over the Indochina and China regions (though they were still hidden).

You traveled to... Indochina and China? Like, be able to see them or you were traveling to fog/blackness where China should be?

[Tellos Athenaios]

No ETW or NTW? I'm surprised to see you here then.

Well I'm the author of the CUF tool, and it so happens that ETW uses CUF files too. There's a thread on that somewhere, so here I was browsing the ETW forums a bit and then I stumble upon “Perfect ESF <-> XML converter and revolution in ESF modding”. Someone promising something Perfect? I had to take a look.

Unfortunately SAX is not quite enough, there's a lot of contextual information needed to make these XML look meaningful.

Where can I read about these transformations, mock sax etc.? I know Java enough to use it when I have to, but I'm not really up to date with everything going on in Java universe.

Well I find this to be a good introduction: http://docstore.mik.ua/orelly/xml/jxml/index.htm especially when coupled with the usual JDK Javadoc (the quickest way is the usual Java + classname Google search)...

As for mocking an XML document using the SAX API: http://www.ling.helsinki.fi/kit/2004...XSLT/Ch05.html (scroll down a bit to “JAXP SAX I/O”, to see how a CSV file can be presented as XML).

Also, a minor one - does all this JAXP business handle xinclude transparently out of the box? It's pretty neat feature of nokogiri library when having to deal with massive xml to just cut them into pieces behind the scenes.

Quick Google search found me this: http://xerces.apache.org/xerces2-j/faq-xinclude.html So you would need to request support for it through the API prior to creating the actual parsers but it should work with modern JAXP/JVM distributions.