Guide to the ETW map

[the emerald ingot]

If there is anything I can do to help, I'd be more than willing to dedicate what free time I have to helping you guys. ^_^ Just let me know.

[PietroMicca]

If there is anything I can do to help, I'd be more than willing to dedicate what free time I have to helping you guys. ^_^ Just let me know.

Have you ever convert an esf file using taw's tools? This the first step. Then read any from the first post in this topic.

[PietroMicca]

passable_part: 0; 00x00
passable_part: 1; 0xx00
passable_part: 4; x0x00
passable_part: 5; xxx00
passable_part: 13; xxx00
passable_part: 32; 00x0x
passable_part: 33; 0xx0x
passable_part: 36; x0x0x
passable_part: 37; xxx0x
passable_part: 45; xxx0x
passable_part: 97; 0xx0x
passable_part: 101; xxx0x
passable_part: 109; xxx0x 
passable_part: 128; 00xx0
passable_part: 129; 0xxx0
passable_part: 132; x0xx0
passable_part: 133; xxxx0
passable_part: 134; x0xx0
passable_part: 135; xxxx0
passable_part: 141; xxxx0
passable_part: 143; xxxx0
passable_part: 160; 00xxx
passable_part: 161; 0xxxx
passable_part: 164; x0xxx
passable_part: 165; xxxxx
passable_part: 165; x0xxx
passable_part: 166; x0xxx
passable_part: 167; 1xxxx
passable_part: 173; xxxxx
passable_part: 175; 1xxxx
passable_part: 176; 00xxx
passable_part: 177; 0xxxx
passable_part: 180; x0xxx
passable_part: 181; xxxxx
passable_part: 182; x0xxx
passable_part: 183; 1xxxx
passable_part: 189; xxxxx
passable_part: 191; xxxxx
passable_part: 225; 0xxxx
passable_part: 229; x1xxx
passable_part: 231; xxxxx
passable_part: 237; xxxxx
passable_part: 239; xxxxx
passable_part: 241; 0xxxx
passable_part: 245; xxxxx
passable_part: 247; xxxxx
passable_part: 253; xxxxx
passable_part: 255; xxxxx

in this list for each passable_part there's the unknown2 pattern. x means not fixed value. But in many cases we can see that 0 is a repeated value. For instance, passable_part: 134; x0xx0. This means that the 2nd and 5th digit are always 0 for passable_part=134. If each hex digit is a feature of boundaries, this could entail that 2nd and 5th feature are not activated for boundaries with passable_part=134.

[the emerald ingot]

got some interesting news; I completely wiped etw off my computer (including all files and savegames) and did a fresh install. I decided not to run it yet. Instead, I took the regions.esf file, converted it to xml, and opened it up; it was completely blank.

To clarify, the file I opened was in data/campaign_maps/global map. If that was the wrong one, or if I'm just being an idiot, please let me know.

[taw]

got some interesting news; I completely wiped etw off my computer (including all files and savegames) and did a fresh install. I decided not to run it yet. Instead, I took the regions.esf file, converted it to xml, and opened it up; it was completely blank.

To clarify, the file I opened was in data/campaign_maps/global map. If that was the wrong one, or if I'm just being an idiot, please let me know.

How did you convert it, and what was the error message?

[the emerald ingot]

ok, nvm, I figured it out. using your tool, in the commmand prompt, I typed (blah blah blah directory) >jruby esf2xml <regions.esf >regions.xml. What I should have typed was (directory info)>jruby esf2xml regions.esf regions.xml. When I did that, it worked. Sorry, that was a total noob trying to figure out stuff he's never done before.

so, now that I've got the file unpacked and editable, what can I do to help?

[PietroMicca]

ok, nvm, I figured it out. using your tool, in the commmand prompt, I typed (blah blah blah directory) >jruby esf2xml <regions.esf >regions.xml. What I should have typed was (directory info)>jruby esf2xml regions.esf regions.xml. When I did that, it worked. Sorry, that was a total noob trying to figure out stuff he's never done before.

so, now that I've got the file unpacked and editable, what can I do to help?

first of all convert all types of esf

you have just converted regions.esf

take a look inside that folder

What can you understand about what you read?

Understand our language, our research, our discoveries

You have to understand how each esf file is linked to the others. Read also the first post of this topic.

Before helping us you get to understand where we are. It will take time. I know this may seem daunting, but this is the only way.

[the emerald ingot]

I've read through this thread in its entirety. I'm not gonna be so arrogant as to say I completely understand everything you guys are talking about, but I think I get the gist of it, and I can always refer to the first post (which is why I've bookmarked this thread). I've got a few ideas I'd like to try, but I don't know where to start, so I thought I'd ask you guys where you want me to start so that I contribute to the research rather than just do my own thing.

I've also unpacked both regions.esf and pathfinding.esf. I can unpack the others as needed.

[PietroMicca]

I've read through this thread in its entirety. I'm not gonna be so arrogant as to say I completely understand everything you guys are talking about, but I think I get the gist of it, and I can always refer to the first post (which is why I've bookmarked this thread). I've got a few ideas I'd like to try, but I don't know where to start, so I thought I'd ask you guys where you want me to start so that I contribute to the research rather than just do my own thing.

I've also unpacked both regions.esf and pathfinding.esf. I can unpack the others as needed.

now the our field of research is unknown2 in pathfinding

[the emerald ingot]

OK guys, I added paths in comments:

<rec type="grid_cells">
<bin6>1f 1f 1f 1f ; 01 01 01 1f</bin6>
<ary type="boundaries">
<boundaries unknown1="0 (00)" unknown2="0 (00000)" path_type="2 (transition area)" path_id="-1 (transition)" vertex_index="19"/><!-- 7 (-566.0,0.0501108169555664); 8 (-568.0,0.0501327514648438); 0; 2 -->
<boundaries unknown1="143 (8f)" unknown2="69648 (11010)" path_type="0 (passable area)" path_id="0 (new_grenada)" vertex_index="24"/><!-- 3; 1; 8 (-568.0,0.0501327514648438); 7 (-566.0,0.0501108169555664) -->
</ary>
</rec>

We need to plot every single hexdigit of unknown2.

If there's too much overlap, we can plot multiple times (map of every path with first hexdigit=0, with first hexdigit=1 etc. for 16*5 maps).

Then we can plot unknown1 as well.

Any volunteers to do the plotting? I don't understand 0,1,2,3 flags.

Drawing script ready.

jruby scripts/draw_boundaries tmp/pathfinding_etw_esf_dir/grid_data-0004.xml out

It creates files like out/hex_xxx0x.svg (everything that has 0 as 4th hexdigit) and out/hex_xxxZx.svg (all hex_xxx*x.svg drawn together and colorcoded). Hopefully I didn't reverse the order of digits or anything like it.
There are 16 arbitrary colors for each hexdigit value.

I just ignored 0/1/2/3 coords while drawing, and treated the rest as path in order.

If you want to research it, edit this line and set proper coordinates for high resolution view:
set_view(-1280.0, -640.0, 1280.0, 640.0, 2560)

At full world resolution it's hard to see much.

Some of these color-coded maps look very sensible, that is they don't look random at all.
*Z* svgs have too much stuff on them to be able to see much at this zoom level.

By the way how sure are you of path_type? (that is lowest digit?) I could make it draw it as well in the same format.
For S2TW at least I'm not certain at all it is right.

Ok, so, I ran that script and got all of the SVG files. Then, I opened ALL of the SVG files this tool mapped out in GIMP as seperate layers, and saved it as an XCF (the standard file that gimp saves your work as). It's almost 27 MB. I've compressed it and attached it to this post.

I hope that this file will help you guys figure out what the different unknown2 hex variables do. I played around with it, and I've found a few interesting things; xxxxx5 are the transition zones for the land bridges, such as between denmark and sweeden, or spain and morocco, and what xxxxx4 draws is right between what is shown on xxxxx5.
xxxxx1 is a perfect outline of the coasts, except the land bridges that xxxxx5 shows up as a solid path on xxxxx1, so maybe it has to do with where troops can walk? Except, of course, there's no barrier in the Sahara or east of Russia...
xxxxx3 perfectly matches rivers.
xxxxx2 matches the outlines for unpassable reagions (such as the barrier in the sahara, around the alps, and east of russia) as well as a thick border matching the coastline. So, maybe this is where troops can't march.

anything with 0 in it or with x in it appear to be a mishmash of more or less everything, so it's impossible for me to discern.

pretty much all of the hex maps with something in the xxXxx (where the capitol x is) seems to correspond to the locations of cities, town, ports, and roads. so this might help us create new cities ect.

then again, xxxxx7 is a perfect match for all places with slots, such as cities, towns, and ports, but no roads...

Let me know if there's anything else I can do!

[the emerald ingot]

I'd edit my previous post, but I haven't posted enough for the button to show up. I just wanted to clarify that I have, so far, only done the hex map for the European theater. If you guys would like, I can run the script on all the other theaters and compile all of those SVG's into a similar format that I did here for Europe.

[PietroMicca]

the emerald ingot take a look at post #1358. You made the discovery of hot water. We know that since more than a month ago. If you read correctly pathfinding, you will find an attribute called path_type. What we need is an explanation for the other hex digits.

[PietroMicca]

There's something that we miss. It's like we are looking data from a completely wrong point fo view. Until we arrived to path_type, all seemed logical and correct. Now we have studied data changing our approach in many ways, but all seems so messy, complicated and illogical.

[uanime5]

There doesn't seem to be any connection between the unknown2 and the number of coordinates in a boundary, the flags present, or slot type. It may be helpful to try to figure out what information it could be providing ETW. In M2TW the map files related to height, ground type, climate, and rivers so it may be something similar.

PietroMicca do you think the unknown2 is made up of 3 numbers or 5 numbers? For example would 81501 be 81, 5, 01 (3 numbers); or 8, 1, 5, 0, 1 (5 numbers).

[PietroMicca]

There doesn't seem to be any connection between the unknown2 and the number of coordinates in a boundary, the flags present, or slot type. It may be helpful to try to figure out what information it could be providing ETW. In M2TW the map files related to height, ground type, climate, and rivers so it may be something similar.

PietroMicca do you think the unknown2 is made up of 3 numbers or 5 numbers? For example would 81501 be 81, 5, 01 (3 numbers); or 8, 1, 5, 0, 1 (5 numbers).

5 numbers more probably. I think so because there isn't a complete scale of value for the 1st, 2nd, 4th and 5th digit. The range of values is 0-c (or 0-12). This excludes the presence of sequential numbers as for vertex_index and path_id. Hex digit in the middle contains a complete scale of values (from 0 to f) instead. This makes possible a connection between 2nd and 3rd digit (and maybe 1st).

In other words, following your example, it is possible 8, 15, 0, 1 or 815, 0, 1. But I believe to be very improbable 81, 5, 01. Anyway I could be wrong.

In the end there could be many other kinds of information in pathfinding.

height? There's already an height_map
ground type and climate? regions.esf already contains this information
rivers? there's already path_type indicating where rivers are

All these things could be repeated, but we don't know if all, partially or none.

[taw]

Correlation matrices say it's 5 integers most likely (post somewhere deep back in this thread).

[PietroMicca]

we should find 5 features in common between boundaries and boundaries_empty, because I remember you that unknown2 has the same meaning for both tags.

[uanime5]

As unknown2 is always 00000 when the path_id is -1 (transition areas and rivers) they're proably something to do with areas land units can walk. They could represent any of the following from the db_tables:

Battle_terrain_set_groupings_tables: 5 types of climate
Battle_weather: levels 0-3
Climate_tables: colour index 0-34
Campaign_ground_types_tables: 11 types

As every part of the unknown2 can be different for different boundaries in the same grid_cell it may be helpful to examine the difference between different path_types. For example as roads are easier to travel on than passable_areas this means if in a grid_cell one part of the unknown2 is always the same for passable_areas but different for roads this is the part that affects how easy it is to travel through grid_cells. The middle hex value seems to fit this role.

[PietroMicca]

these are values used by last hex in unknown2

0
1
2
3
4
5
6
7
8
9
a
c

never used b

why?

anyway there are other strange things



this is a list of colors used by me to depict cells and areas following the last hex digit

Ceylon

Spoiler Alert, click show to read: 

Ivory Coast

Spoiler Alert, click show to read: 

Many slots are a half 0 (black) and a half 2 (red). 1 (blue) is used in the sea and as well in the land. So ground type is excluded.

I will draw some other regions where are used 9, a, c. Maybe we will infer something more.

[the emerald ingot]

Just a thought, but maybe it's feeding info into the battlemap generator? The third hex value almost always corresponds to a feature on the campaign map, such as building or roads. Perhaps what digit 1 is signifying is that there are no significant features that need to be included in the battlemap, such as roads or rivers, in that particular area. Hmm. I'll run some experiments and get back to you.