Request for help with esfs

[taw]

Hi,

esf2xml is capable of displaying more meaningful xmls, so instead of:

<rec type='DATE'>
<u>1676</u>
<asc>summer</asc>
</rec>
<rec type='DATE'>
<u>0</u>
<asc>summer</asc>
</rec>

You'd see:

<date>summer 1676</date>
<date />

(Did you know summer of year 0 is null date indicator? I bet you didn't.)

And instead of:

<rec type='TRAIT'>
<s>C_Admiral_Good</s>
<i>8</i>
</rec>

You'd see:

<trait name="C_Admiral_Good" level="8" />

And instead of:

<rec type='CAMPAIGN_VICTORY_CONDITIONS'>
<ary type='REGION_KEYS'>
<rec type='REGION_KEYS'>
<s>venice</s>
</rec>
<rec type='REGION_KEYS'>
<s>hungary</s>
</rec>
<rec type='REGION_KEYS'>
<s>austria</s>
</rec>
<rec type='REGION_KEYS'>
<s>rumelia</s>
</rec>
<rec type='REGION_KEYS'>
<s>prussia</s>
</rec>
<rec type='REGION_KEYS'>
<s>west_pommerania</s>
</rec>
<rec type='REGION_KEYS'>
<s>west_prussia</s>
</rec>
</ary>
</rec>

You'd see:

<victory_conditions>
venice
hungary
austria
rumelia
prussia
west_pommerania
west_prussia
</victory_conditions>

This was basically the idea all along.
Low-level XML was only step one.

Anyway - what I need now is for someone to go over all xmls generated and
compile list of such mappings.

I could automatically generate <foo a='1' b='2' c='3' /> but attributes like a=/b=/c= are
fairly worthless, so I won't follow that path.

Volunteers? It shouldn't be that much harder than db table labeling,
and this will help me with opening campaign map more than buying me beer.
Not that I mind beer.

How to contribute

Preferred format is just like above, as list of pairs:

current low level XML

better semantic XML

If mapping is not-trivial, and it will often be, describe what variations
there are, and how they should work, as with <date /> above.
Remember you're ultimately describing it to a computer, so be precise,
and don't expect too much intelligence from the machine.

And remember I want byte perfect round trips, but XML attribute order is not preserved, so <traits C_Admiral_Good="8" C_Whatever="4" /> won't do,
as it is identical as <traits C_Whatever="4" C_Admiral_Good="8" /> on DOM level.

I might change my mind about byte perfect round trips, but the reason would need to be really really good.

[Okmin]

Looking at startpos.xml now

BTW, thanks for all the work you put into this and the campaign map.

EDIT: So basically, you want stuff like this?

<SAVE_GAME_HEADER>
    <faction>britain<faction>
        <leader>ui/portraits/european/cards/king/old/014.tga</leader>
        <turn>1</turn>
        <yr>1700</yr>
        <season>Summer</season>
        <flag>data\ui\flags\britain</flag>
        <maps>
            <xml_include path='maps-0001.xml'/>
            <xml_include path='maps-0002.xml'/>
            <xml_include path='maps-0003.xml'/>
        </maps>
</SAVE_GAME_HEADER>

Instead of this:

<rec type='SAVE_GAME_HEADER'>
 <s>britain</s>
 <s>ui/portraits/european/cards/king/old/014.tga</s>
 <u>1</u>
 <u>1700</u>
 <s>Summer</s>
 <s>data\ui\flags\britain</s>
 <ary type='MAPS'>
  <xml_include path='maps-0001.xml'/>
  <xml_include path='maps-0002.xml'/>
  <xml_include path='maps-0003.xml'/>
 </ary>
</rec>

[taw]

Looking at startpos.xml now

BTW, thanks for all the work you put into this and the campaign map.

EDIT: So basically, you want stuff like this?

<SAVE_GAME_HEADER>
    <faction>britain<faction>
        <leader>ui/portraits/european/cards/king/old/014.tga</leader>
        <turn>1</turn>
        <yr>1700</yr>
        <season>Summer</season>
        <flag>data\ui\flags\britain</flag>
        <maps>
            <xml_include path='maps-0001.xml'/>
            <xml_include path='maps-0002.xml'/>
            <xml_include path='maps-0003.xml'/>
        </maps>
</SAVE_GAME_HEADER>

Instead of this:

<rec type='SAVE_GAME_HEADER'>
 <s>britain</s>
 <s>ui/portraits/european/cards/king/old/014.tga</s>
 <u>1</u>
 <u>1700</u>
 <s>Summer</s>
 <s>data\ui\flags\britain</s>
 <ary type='MAPS'>
  <xml_include path='maps-0001.xml'/>
  <xml_include path='maps-0002.xml'/>
  <xml_include path='maps-0003.xml'/>
 </ary>
</rec>

More or less. Especially if all records of some type follow identical pattern,
so every SAVE_GAME_HEADER can get converted.

There's 1000 or so of such record types.

[taw]

I will upload esfxml with support for a few dozen such record types to show what I mean later.

[Hazbones]

Yeah, I'll need some more record type examples before I could jump in. I am not experienced with this type stuff except for old HTML coding.

[Primergy]

I tend to believe that the .enviroment files are a good source, since those seam to be some sort of semantic xml, or?

[ForteS]

Taw have you been working on this? I've tried to do some myself but there are lots of entries which i don't know what they mean I've changed some anyway so if you want me to send them just tell.

obs: I'm not sure if I've done the changes in the right place so if you could explain us better we would be grateful.

[taw]

Taw have you been working on this? I've tried to do some myself but there are lots of entries which i don't know what they mean I've changed some anyway so if you want me to send them just tell.

obs: I'm not sure if I've done the changes in the right place so if you could explain us better we would be grateful.

Well, send me whatever you've got. Code now supports simple semantic tags,
and even climate map export to PGM bitmap format.

Right now only very very simple tags will work, but it will change the next time I have some time to work on this.

[taw]

Relax guys, you'll see a new release in a few days hopefully ;-)

[The Hedge Knight]

enviroment is just xml i think. I edited quite a few to make my nightmod...

[ForteS]

Relax guys, you'll see a new release in a few days hopefully ;-)

Great Taw

enviroment is just xml i think. I edited quite a few to make my nightmod...

Nightmod?

[Peteriscoolz]

Nightmod?

http://www.twcenter.net/forums/showthread.php?t=386575

[taw]

Well, it's not of much use but esfxml supports <date>summer 1660</date> now.
I'm mostly just trying to figure out sane way of supporting variety of tag styles internally,
it's not of terribly much use to others yet.

[taw]

Hi, I uploaded new code. Supported semantic tags are:

regions.esf
<bounds_block xmin='-581.6795' ymin='253.953' xmax='-567.31494' ymax='266.0332'/>
<climate_map pgm='climate_map-0001.pgm'/>
<connectivity mask='00070000' from='79' to='113'/>

startpos.esf
<agent_ability ability='can_research' level='-1' attribute=''/>
<agent_ability ability='can_duel' level='1' attribute='duelling_pistols'/>
<agent_attribute attribute='management' level='4'/>
<agent_attribute attribute='subterfuge' level='-1'/>
<date>summer 1676</date>

Explanations:

climate_map is saved as actual image.
PGM is trivial format with just very trivial header, and raw binary data following.
AFAIK it's not really intensity levels, it's index-coded - but if you want to make battles in Sweden
be a jungle or something, and French Guyana to be a desert, I think that should be the exported image.

connectivity is example of hex data. There's plenty of them

bounds_block is example of many to many mapping - two Vec2D tags become four attributes.
It can go the other way around with three Byte tags becoming one #RRGGBB color attribute, I'll add one of those later.

agent_ability/agent_attributes/date should support attribute-specific defaults (-1, or "summer 0", or '')
but nothing like that has been coded yet as startpos conversion is so damn slow ;-p

It's just 6 tags for now. Treat them as examples really.
Maybe 20% of all tags will be convertible to one of three supported formats (pure attributes, pure text+regexp, external pgm),
for everything else more code will be necessary.

Go ahead and start submitting documentation for whichever tags you find which can be converted that easily.
If something is not quite that simple but close, submit documentation anyway, I'll be adding more code soon-ish anyway.

For performance reasons, something that's not startpos.esf will probably be a good start ;-)

Internals

If you want to see how conversion works, look at EsfSemantic in esf2xml
and at XmlCallbacks in xml2esf. Thanks to support code it's just a few lines per conversion,
but I don't really expect people to code here - just document things for me, and I'll add the code.

[Okmin]

Could we have an example of some output the converter couldn't handle? Maybe it could help us figure out how simple we can make our suggestions.

[taw]

Could we have an example of some output the converter couldn't handle? Maybe it could help us figure out how simple we can make our suggestions.

Couldn't for architectural reason, or couldn't right now?
Because right now it can do very little, but I'll be adding code anyway.

So:

Anything that gathering data from all over the file is probably too hard.
Yes, this is exactly what campaign map export tools will need to do, but let's not get into that too early.

Right now anything that involves constant number of fields is very simple.

Converting arrays of simple records (<ary><rec><s>foo</s></rec><rec><s>bar</s></rec></ary>)
to simple <something>foo bar</something> is coming right next.

Records with variable number of fields can be handled if:

• interesting fields are in prefix, and/or
• field types are quite regular

Converter for poi.esf's root type is about the most difficult I can imagine writing reasonably quickly.

I'm sure we'll have a few rounds of "actually, there's no way converter can handle that" before we get to the goal of semantic representation of 90% of nodes.

[Okmin]

Couldn't for architectural reasons then, since you're updating the thing a lot.

So basically stuff that looks like the <date> and <agent_...> things you did?

[taw]

Couldn't for architectural reasons then, since you're updating the thing a lot.

So basically stuff that looks like the <date> and <agent_...> things you did?

Things like <date> and <agent_...> work right away, and there's plenty of common tags
that can be handled in this simple format, so let's start there.

I think we can get to 20% coverage before we need anything more complicated.

[ForteS]

Taw shall we convert the arrays our self or send you suggestions and the you do them?

[taw]

Taw shall we convert the arrays our self or send you suggestions and the you do them?

What I want is documentation, even if code doesn't work with it just yet.