Perfect ESF<->XML converter and revolution in ESF modding [Released!]

[husserlTW]

If you have some interesting cuts, send them to me and I'll add them to the list.
Good xml size should be 100kB-ish I guess, current XMLs are too big.

(but then semantic cleanup will get them down to 25% of current size,
so maybe aim at 500kB instead...)

I had not a standard schema, it was depending on my research. The way I work is this: I use this python tool to track an Id, then I take the report and export from startpos the pieces I like and then convert them to xml. If it is big e.g like campaign_pathfinder (about 60.000 lines) i cut them further. But now i can see many potential here. I've already started preparing an XmlSplit schema that will help first receiving xml data in an organized useful way. After that we can proceed to finalize xml sizes and then in this!

[taw]

I had not a standard schema, it was depending on my research. The way I work is this: I use this python tool to track an Id, then I take the report and export from startpos the pieces I like and then convert them to xml. If it is big e.g like campaign_pathfinder (about 60.000 lines) i cut them further. But now i can see many potential here. I've already started preparing an XmlSplit schema that will help first receiving xml data in an organized useful way. After that we can proceed to finalize xml sizes and then in this!

By the way xml splitter should be using first string found in section instead of number for most xmls.
So you'd get things like factions/britain.xml not faction/0012.xml. It's not always meaningful, but works about half of the time. I'll fix code to do that later.

It would also be reasonably simple to modify esf2xml to export partial xmls, and partial binaries (pnm_include / esf_include etc. in addition to xml_include).

Rebuilding them wouldn't be totally trivial, but fairly so, and speedup would be really big.
Bitmap export code is there, but not ported to jruby/nokogiri yet.

[husserlTW]

Here is a XmlSlit proposition according the modifications I am doing more often:



The output folder looks like:



Since making a complex XmlSplit makes convertion more slow (more folders and files to create), how can we:

It would also be reasonably simple to modify esf2xml to export partial xmls, and partial binaries (pnm_include / esf_include etc. in addition to xml_include).

[uanime5]

When I try to convert a startpos.dir into a startpos.esf I get this error.



Anyone know how I can fix it?

[husserlTW]

Did you try another name without (.) in it for your startpos.dir folder?

[taw]

When I try to convert a startpos.dir into a startpos.esf I get this error.



Anyone know how I can fix it?

Well, that's a new one. You're not supposed to see "Too many open files" in Ruby,
it automatically close them on GC. As should JRuby.

It looks like file descriptors are leaking somewhere, hmm...
Any information about your system? It would be easier to fix it if I could reproduce it.

[uanime5]

Did you try another name without (.) in it for your startpos.dir folder?

I tried renaming the folder startpos_dir but that didn't work either.

Well, that's a new one. You're not supposed to see "Too many open files" in Ruby,
it automatically close them on GC. As should JRuby.

It looks like file descriptors are leaking somewhere, hmm...
Any information about your system? It would be easier to fix it if I could reproduce it.

What information do you need?

Also should I try to reinstall jRuby?

[husserlTW]

Untill taw give's the right answer, do the following I did for another kind of problem but just in case:

1. Download again taw's files because he updates them from time to time.

2. Open a cmd and run these commands:

jgem uninstall nokogiri

jgem install nokogiri --pre

[uanime5]

Thanks that worked.

[taw]

Untill taw give's the right answer, do the following I did for another kind of problem but just in case:

1. Download again taw's files because he updates them from time to time.

2. Open a cmd and run these commands:

jgem uninstall nokogiri

jgem install nokogiri --pre

I cannot really give "the right answer" if the question is "why this or that doesn't work on Windows".
I test it on pretty much fresh installation of English Windows 7 on a virtual machine, with the most recent code, and following all my instructions for installing JRuby, libraries etc. to the letter.
If it works like that, that's good enough for me. There's no way I could possibly guarantee anything
working on any Windows without making this my full time job.

If you need support getting it working on OSX 10.5 or newer,
or reasonably recent Ubuntu or other sensible Linux, I'm your man.

I'll leave Windows tech support mostly to people with more clue about Windows.

[taw]

Here is a XmlSlit proposition according the modifications I am doing more often:

I'll added your rules just without numbers, as single set of rules is used for all
esf subtypes right now.

Since making a complex XmlSplit makes convertion more slow (more folders and files to create), how can we:

You cannot yet. It will be supported later.
Splitting shouldn't be noticeably slower unless you split into ridiculously many files.
It might be even slightly faster due to lower memory requirements.
Moving binary files out will definitely speed it up, but that will come later.

[taw]

New code pushed to etwng, now with big speedup of esf2xml. That's current numbers:

~/etwng/esfxml$ time jruby --server ./esf2xml samples/startpos.esf sdir
real 1m24.521s
user 1m19.819s
sys 0m4.246s
~/etwng/esfxml$ time jruby --server ./xml2esf sdir s.esf
real 3m3.122s
user 3m1.186s
sys 0m3.876s

The path forward towards higher speed is mostly by generating less XML - that is
moving towards more compact higher level representations (will speed up only xml2esf),
and by extracting binary data to external files instead of including hex dumps in XML
(will speed up both).

[husserlTW]

I think you had mentioned in the past that you would add some kind of labels to replace the xml symbols so it shows e.g. <faction>britain instead of <s>britain. Can we help in that?

[taw]

I think you had mentioned in the past that you would add some kind of labels to replace the xml symbols so it shows e.g. <faction>britain instead of <s>britain. Can we help in that?

I'm mostly converting such entries to XML attributes if possible, for technical reason.
But figuring out that britain is a faction is not that difficult - the hard part are all the numbers.

My top wish list right now is for someone to document structure of the following in startpos.esf:

• region/*.xml, especially POPULATION_CLASS
• factions/faction-*.xml, especially DIPLOMACY_RELATIONSHIP_ATTITUDES_ARRAY, techs
• victory_conditions/*.xml

Second item on my wish list is some editable formats (pure binary with simple header) for:

• height map of singles
• indexed-color picture for uint8/uint16/uint32 (uint8 has PGM but there's no way to put palette there)
• some very simple outline format - I know I can export to SVG, but I cannot import it back once it gets through any editor. What else can I use?
• By the way I have written quadtree to pgm and svg converter before - but this sucks hard to expand a few kB of quadtree into a few MBs of boring repetitive data of PGM, and SVG I can generate but not really parse. Are there any saner representations?

[MatrixStorm]

Taw, I don't know how to say this but...
You're bloody amazing!

How did you decode the .esf files?
I tried it once with some tools I don't even know anymore and with notepad, but with no results.

Also, + rep

Oh, and I'm counting on you to make the campaign map editable before Chrissman

[taw]

Taw, I don't know how to say this but...
You're bloody amazing!

How did you decode the .esf files?
I tried it once with some tools I don't even know anymore and with notepad, but with no results.

Also, + rep

Oh, and I'm counting on you to make the campaign map editable before Chrissman

Are you asking how to use tools or how I wrote them?

If you want to use it, follow instructions in README.txt.

By the way could someone please create a tutorial with screenshots, and sticky it somewhere
so I could link to it from README.txt? The interface probably won't change much.

As for decoding esf, format was mostly known before, there was EsfEditor before I even started.

[taw]

Code now extracts a lot more binaries from bmd.dat, regions.esf, and startpos.esf

The only problem - most of them are not really PGM (grayscale uint8 bitmaps),
they're float32 / uint32 / uint16 / or whatever else bitmaps.

I also found some jpegs in bmd.dat, they're extracted as is, and should be immediately useful.

Here's up to date full list of supported tags.

BMD_TEXTURES shows how highly irregular tag can be supported.

Now that I think of it, bmd.dat might be the format that benefits most from all this at the moment;
regions.esf and startpos.esf are much larger projects, and it will be quite some time before you
get campaign editor out of it. Did I mention how much free time I tend to have? Not too much.

[alhoon]

regions.esf and startpos.esf are much larger projects, and it will be quite some time before you get campaign editor out of it. Did I mention how much free time I tend to have? Not too much.

Since we're not paying you, we can't complain.
It's not "how much free time I tend to have" but "How much free time I want to spend without payment to help you guys, just because I'm a good and helpful guy. The rest of my free time I'll play games because that's what I like to do."

[taw]

Since we're not paying you, we can't complain.

Well, you write mods for me to play, that works to :-p

[taw]

New code on etwng now has five more tags + better conversion of agent attributes.
All for startpos.esf. Example of new tag:

Before:

<rec type='TRAITS'>
<ary type='TRAIT'>
<rec type='TRAIT'>
<s>C_Minister_Reactionary</s>
<i>1</i>
</rec>
<rec type='TRAIT'>
<s>C_Minister_Entertainer</s>
<i>2</i>
</rec>
<rec type='TRAIT'>
<s>C_Minister_Fiscal_Genius</s>
<i>2</i>
</rec>
</ary>
</rec>


After:

<traits>
C_Minister_Reactionary=1
C_Minister_Entertainer=2
C_Minister_Fiscal_Genius=2
</traits>