Guide to the ETW map

[PietroMicca]

Wait, what?

-1 in 32bit hex is 0xFFFFFFFF.

+1 in 32bit hex is 0x00000001.

ok I have a theory buzzing in my head

unknown 8 digits 2:3:3

the first 2 a sort of unknown id
the second 3 signed values multiple of 16 and so the last 3

it would be possible?

[taw]

Added [uint2, int3, uint3] in comment, enjoy.

[PietroMicca]

Added [uint2, int3, uint3] in comment, enjoy.

why you have done it for unknown1?

I just want unknown2 split in 2 part, the first one signed

[taw]

Then I have no idea what you're talking about. There are 24 bits in unknown2.

You want it split into [int12, uint12] ? I've done it in the code, but it's obviously wrong.

The only negative numbers you'll ever see are in S2TW, I don't think there are any in ETW pathfinding.esf ever.

<boundaries unknown1="0 (00)" unknown2="2 (000002 | 0 2)" path_id="-1" vertex_index="19"/>
<boundaries unknown1="143 (8f)" unknown2="1114368 (110100 | 272 256)" path_id="0" vertex_index="24"/>

<boundaries unknown1="0 (00)" unknown2="9116771 (8b1c63 | -1871 3171)" path_id="-1" vertex_index="5"/>
<boundaries unknown1="13 (0d)" unknown2="6252321 (5f6721 | 1526 1825)" path_id="14" vertex_index="11"/>

[PietroMicca]

We have to give up for now.

Maybe we'll be more successful in startpos campaign_pathfinder.

[taw]

path_ids now have comments. No idea if they are correct. What I see doesn't look terribly sensible.


<boundaries unknown1="0 (00)" unknown2="2 (000002 | 0 2)" path_id="-1 (transition)" vertex_index="19"/>
<boundaries unknown1="143 (8f)" unknown2="1114368 (110100 | 272 256)" path_id="0 (sea)" vertex_index="24"/>

<boundaries unknown1="176 (b0)" unknown2="273 (000111 | 0 273)" path_id="49 (new_york, ontario)" vertex_index="1408"/>
<boundaries unknown1="0 (00)" unknown2="2 (000002 | 0 2)" path_id="-1 (transition)" vertex_index="1414"/>
<boundaries unknown1="134 (86)" unknown2="1048833 (100101 | 256 257)" path_id="49 (new_york, ontario)" vertex_index="1427"/>
<boundaries unknown1="109 (6d)" unknown2="1114128 (110010 | 272 16)" path_id="0 (sea)" vertex_index="1431"/>

<boundaries unknown1="176 (b0)" unknown2="273 (000111 | 0 273)" path_id="49 (new_york, ontario)" vertex_index="9413"/>
<boundaries unknown1="0 (00)" unknown2="2 (000002 | 0 2)" path_id="-1 (transition)" vertex_index="9417"/>
<boundaries unknown1="134 (86)" unknown2="1048832 (100100 | 256 256)" path_id="4 (panama, guatemala)" vertex_index="9427"/>
<boundaries unknown1="97 (61)" unknown2="65553 (010011 | 16 17)" path_id="49 (new_york, ontario)" vertex_index="9433"/>

<boundaries unknown1="176 (b0)" unknown2="273 (000111 | 0 273)" path_id="49 (new_york, ontario)" vertex_index="13468"/>
<boundaries unknown1="0 (00)" unknown2="2 (000002 | 0 2)" path_id="-1 (transition)" vertex_index="13473"/>
<boundaries unknown1="13 (0d)" unknown2="1114112 (110000 | 272 0)" path_id="7 (wilderness_mexico, new_mexico)" vertex_index="13484"/>
<boundaries unknown1="97 (61)" unknown2="65553 (010011 | 16 17)" path_id="49 (new_york, ontario)" vertex_index="13490"/>

[PietroMicca]

you have built your array in the wrong way

you have put sea as first element of this array

listed regions, sea and at last what you see after the separator

you have an <u2_ary>, put comments in order of apparition as elements of an array and replace separator comment with sea

[taw]

I don't understand you. Just explain it as an example.

[PietroMicca]

ok an example from India (grid_data-0004)

this is <u2_ary>

<u2_ary>
  1 <!-- unexplorable -->
  2 <!-- ceylon -->
  3 <!-- carnatica -->
  4 <!-- mysore -->
  5 <!-- malabar -->
  6 <!-- hyderabad -->
  7 <!-- bijapur -->
  8 <!-- orissa -->
  9 <!-- ahmadnagar -->
  10 <!-- berar -->
  11 <!-- gujarat -->
  12 <!-- malwa -->
  13 <!-- bengal -->
  14 <!-- rajpootana -->
  15 <!-- sindh -->
  16 <!-- hindustan -->
  17 <!-- baluchistan -->
  18 <!-- punjab -->
  19 <!-- afghanistan -->
  20 <!-- kashmir -->
  21 <!-- wilderness_khiva -->
 
  0 <!-- sea -->
 
  2 3, 4 <!-- carnatica, mysore -->
  2 4, 5 <!-- mysore, malabar -->
  2 3, 6 <!-- carnatica, hyderabad -->
  3 4, 5, 7 <!-- mysore, malabar, bijapur -->
  2 4, 7 <!-- mysore, bijapur -->
  2 4, 6 <!-- mysore, hyderabad -->
  3 3, 4, 6 <!-- carnatica, mysore, hyderabad -->
  2 3, 8 <!-- carnatica, orissa -->
  2 5, 7 <!-- malabar, bijapur -->
  2 6, 8 <!-- hyderabad, orissa -->
  3 3, 6, 8 <!-- carnatica, hyderabad, orissa -->
  2 6, 7 <!-- hyderabad, bijapur -->
  2 7, 9 <!-- bijapur, ahmadnagar -->
  3 6, 7, 9 <!-- hyderabad, bijapur, ahmadnagar -->
  2 6, 9 <!-- hyderabad, ahmadnagar -->
  2 6, 10 <!-- hyderabad, berar -->
  3 6, 8, 10 <!-- hyderabad, orissa, berar -->
  2 8, 10 <!-- orissa, berar -->
  2 9, 11 <!-- ahmadnagar, gujarat -->
  3 6, 9, 12 <!-- hyderabad, ahmadnagar, malwa -->
  2 6, 12 <!-- hyderabad, malwa -->
  2 9, 12 <!-- ahmadnagar, malwa -->
  3 6, 10, 12 <!-- hyderabad, berar, malwa -->
  2 10, 12 <!-- berar, malwa -->
  3 9, 11, 12 <!-- ahmadnagar, gujarat, malwa -->
  2 11, 12 <!-- gujarat, malwa -->
  2 8, 13 <!-- orissa, bengal -->
  2 11, 14 <!-- gujarat, rajpootana -->
  3 11, 12, 14 <!-- gujarat, malwa, rajpootana -->
  2 12, 14 <!-- malwa, rajpootana -->
  2 10, 13 <!-- berar, bengal -->
  2 11, 15 <!-- gujarat, sindh -->
  3 10, 12, 13 <!-- berar, malwa, bengal -->
  2 12, 13 <!-- malwa, bengal -->
  3 11, 14, 15 <!-- gujarat, rajpootana, sindh -->
  2 12, 16 <!-- malwa, hindustan -->
  3 12, 13, 16 <!-- malwa, bengal, hindustan -->
  2 14, 15 <!-- rajpootana, sindh -->
  2 14, 16 <!-- rajpootana, hindustan -->
  3 12, 14, 16 <!-- malwa, rajpootana, hindustan -->
  2 13, 16 <!-- bengal, hindustan -->
  2 15, 17 <!-- sindh, baluchistan -->
  3 14, 15, 18 <!-- rajpootana, sindh, punjab -->
  2 14, 18 <!-- rajpootana, punjab -->
  2 15, 18 <!-- sindh, punjab -->
  3 15, 17, 18 <!-- sindh, baluchistan, punjab -->
  2 17, 19 <!-- baluchistan, afghanistan -->
  2 17, 18 <!-- baluchistan, punjab -->
  3 17, 18, 19 <!-- baluchistan, punjab, afghanistan -->
  3 14, 16, 18 <!-- rajpootana, hindustan, punjab -->
  2 18, 19 <!-- punjab, afghanistan -->
  2 16, 18 <!-- hindustan, punjab -->
  3 16, 18, 20 <!-- hindustan, punjab, kashmir -->
  2 16, 20 <!-- hindustan, kashmir -->
  2 18, 20 <!-- punjab, kashmir -->
  2 19, 20 <!-- afghanistan, kashmir -->
  3 18, 19, 20 <!-- punjab, afghanistan, kashmir -->
  2 19, 21 <!-- afghanistan, wilderness_khiva -->
</u2_ary>

this is the array to call back for comments (first column position of elements in the array)

0	unexplorable
1	ceylon
2	carnatica
3	mysore
4	malabar
5	hyderabad
6	bijapur
7	orissa
8	ahmadnagar
9	berar
10	gujarat
11	malwa
12	bengal
13	rajpootana
14	sindh
15	hindustan
16	baluchistan
17	punjab
18	afghanistan
19	kashmir
20	wilderness_khiva
21	sea
22	carnatica, mysore
23	mysore, malabar
24	carnatica, hyderabad
25	mysore, malabar, bijapur
26	mysore, bijapur
27	mysore, hyderabad
28	carnatica, mysore, hyderabad
29	carnatica, orissa
30	malabar, bijapur
31	hyderabad, orissa
32	carnatica, hyderabad, orissa
33	hyderabad, bijapur
34	bijapur, ahmadnagar
35	hyderabad, bijapur, ahmadnagar
36	hyderabad, ahmadnagar
37	hyderabad, berar
38	hyderabad, orissa, berar
39	orissa, berar
40	ahmadnagar, gujarat
41	hyderabad, ahmadnagar, malwa
42	hyderabad, malwa
43	ahmadnagar, malwa
44	hyderabad, berar, malwa
45	berar, malwa
46	ahmadnagar, gujarat, malwa
47	gujarat, malwa
48	orissa, bengal
49	gujarat, rajpootana
50	gujarat, malwa, rajpootana
51	malwa, rajpootana
52	berar, bengal
53	gujarat, sindh
54	berar, malwa, bengal
55	malwa, bengal
56	gujarat, rajpootana, sindh
57	malwa, hindustan
58	malwa, bengal, hindustan
59	rajpootana, sindh
60	rajpootana, hindustan
61	malwa, rajpootana, hindustan
62	bengal, hindustan
63	sindh, baluchistan
64	rajpootana, sindh, punjab
65	rajpootana, punjab
66	sindh, punjab
67	sindh, baluchistan, punjab
68	baluchistan, afghanistan
69	baluchistan, punjab
70	baluchistan, punjab, afghanistan
71	rajpootana, hindustan, punjab
72	punjab, afghanistan
73	hindustan, punjab
74	hindustan, punjab, kashmir
75	hindustan, kashmir
76	punjab, kashmir
77	afghanistan, kashmir
78	punjab, afghanistan, kashmir
79	afghanistan, wilderness_khiva

highlighted in red position of sea

-1 transition as you have correctly done

[taw]

Try it now.

[PietroMicca]

empty cells

<rec type="grid_cells">
   <bin6>17 1b 21 1f ; 09 07 0d 11</bin6>
   <ary type="boundaries"/>
   <u>4279304465</u>
   <u2>21</u2>

they could follow the same schema

the second integer is coded in 32bit

but it could be because in an empty cell there isn't a vertex_index or there is an empty vertex_index

so the second integer could be just a path_id? or in alternative it could be a path_id + an empty vertex index?

[taw]

empty cells

<rec type="grid_cells">
   <bin6>17 1b 21 1f ; 09 07 0d 11</bin6>
   <ary type="boundaries"/>
   <u>4279304465</u>
   <u2>21</u2>

they could follow the same schema

the second integer is coded in 32bit

but it could be because in an empty cell there isn't a vertex_index or there is an empty vertex_index

so the second integer could be just a path_id? or in alternative it could be a path_id + an empty vertex index?

Try it now.

[PietroMicca]

Try it now.

I have checked my theory and it is correct.

<u2> tag for empty cells is just path_id.

sometimes path_id for empty cells is 1023, this happens when id is undefined

undefined means that cell is shared between 2 regions, or that empty cell is part of an impassable areas or something else

in other words undefined is for cells not belonging to a path

Also the <u> tag has the same structure of what is unknown now. Finally we can say that empty cells schema is equal to non empty. The only difference is lack of vertex_index for an obvious reason.

So now we have a sort of hint to understand the unknown. What features are in common between an empty and a not empty cell?

[uanime5]

Also the <u> tag has the same structure of what is unknown now. Finally we can say that empty cells schema is equal to non empty. The only difference is lack of vertex_index for an obvious reason.

So now we have a sort of hint to understand the unknown. What features are in common between an empty and a not empty cell?

Well since the unknown is used to tell ETW where the sea is located perhaps it's related to the terrain. Could unknown 2 be the ground type such as desert or plain? I think ground type is more likely to be unknown 2 because anything with a path_id="-1" has an unknown 2 value of 2 or 3, while its unknown 1 entry is always 0.

Another idea I had was that the unknowns tell ETW what type of thing was on the map. For example a passable area, impassible area, town, settlement, etc. So each unknown would represent things like passable area 8, road 1, town 4, sea 2. This would explain why the unknowns are repeated so many times.

[Tantalaul]

This program will help you with .xml files?
http://www.ultraedit.com/ultraedit-tour.html

[King William the Conqueror]

you can open .xml with notepad if I am not mistaken

[Taraphir]

My sincere compliments. Great work gentlemen!
I take a bow.

[PietroMicca]

I hope taw will come back soon

decoded last for 4 bits (or last hex digit if you prefer) of unknown2 in pathfinding grid_data

this last digit defines type of path

there are 8 types of path from 0 to 7

0 passable area
1 sea boundary
2 transition area
3 river
4 land bridge area
5 land bridge transition area
6 road
7 slot

some more detail about each category (an example of how it should be new code format before each explanation)

0 passable area

<boundaries unknown1="f7" unknown2="11211" path_type="0 (passable area)" path_id="1 (ceylon)" vertex_index="601"/>

The most used type. Generally comprehensive of the entire regional area, except room left free for buildings, rivers, mountains and roads.

Spoiler Alert, click show to read: 

They typically looks like above. But this image could make you think that roads, slots etc... are already part of type 0. So I post you another image that will show you how it should correctly read.

Spoiler Alert, click show to read: 

Passable area is in red. This type is also used by land-land transition areas.

1 sea boundary

<boundaries unknown1="fd" unknown2="11011" path_type="1 (sea boundary)" path_id="21 (sea)" vertex_index="406"/>

Simply what you read. This type of path defines boundary of sea.

2 transition area

<boundaries unknown1="00" unknown2="00000" path_type="2 (transition area)" path_id="-1 (transition)" vertex_index="413"/>

This type need some more explanation instead. Transition is also a path_id. This path defines sea-land transition areas, but also impassable areas. Type 2 defines that a <boundaries> is used for a sea-land transition areas, mountains and lakes. Not for rivers.

3 river

<boundaries unknown1="00" unknown2="000003" path_type="3 (river)" path_id="-1 (transition)" vertex_index="7451"/>

In fact rivers have a separated type.

4 land bridge area

<boundaries unknown1="65" unknown2="81b014" path_type="4 (land bridge area)" path_id="50 (sea)" vertex_index="22798"/>

an example of land bridge area from Curacao

Spoiler Alert, click show to read: 

Land bridge areas are areas where you can walk also if on the sea.

5 land bridge transition area

<boundaries unknown1="86" unknown2="10440" path_type="5 (land bridge transition area)" path_id="50 (sea)" vertex_index="22793"/>

Spoiler Alert, click show to read: 

Land bridges work as tunnels. Land bridge areas are the tubes and land bridge transition areas are the entrances of tunnels.

6 road

<boundaries unknown1="20" unknown2="00900" path_type="6 (road)" path_id="73 (lower_louisiana, upper_louisiana)" vertex_index="73564"/>

an example from Carnatica

Spoiler Alert, click show to read: 

Type road defines simply roads as you can see.

7 slot

<boundaries unknown1="04" unknown2="20100" path_type="7 (slot)" path_id="24 (carolinas)" vertex_index="74355"/>

Type slot distinguish are occupied by buildings slots. All kind of buildings are included: settlements, town, farms, etc... and also ports. Just a notation for port. A port has a land area and a sea area. Land areas of ports are part of regions, sea areas of ports are part of sea region. So type 7 is also used by sea. Trade nodes are to be considered slots.

[taw]

Congratulations. I'll get back to you after holidays. Keep on the good work.

[uanime5]

I've noticed a pattern among the building slots; one path is for a passable area, while the reverse path is for the slot. I believe this is so units can enter and leave a slot.

For example in the code below from the Americas theatre.

  <rec type="grid_cells">
   <bin6>00 00 00 00 ; 00 00 00 00</bin6>
   <ary type="boundaries">
    <boundaries unknown1="247 (f7)" unknown2="1122576 (112110)" path_id="23" vertex_index="68466"/>
    <boundaries unknown1="13 (0d)" unknown2="2232327 (221007)" path_id="23" vertex_index="68472"/>
   </ary>
  </rec>

The passable area vertex_index="68466" is for the following path in the pathfinding_area.

  5 <!-- vertices count -->
  16001 <!-- -640.0,239.783366203308 -->
  16030 <!-- -640.216618537903,240.0 -->
1
0
2

While building slot vertex_index="68472" is for the reverse path.

  3 <!-- vertices count -->
3
  16030 <!-- -640.216618537903,240.0 -->
  16001 <!-- -640.0,239.783366203308 -->

This allows a unit to enter a slot from this cell and exit a slot into this cell.

Also the unknown 1 seems to be related to the number of vertex_indices that a slot has and how they're connected. So a slot with 8 vertex_indices spread over 4 grid_cells will use a different unknown 1 than a slot with 7 vertex_indices spread over 3 grid_cells. It may also be related to whether the grid_cells for a square, triangle, or a straight line.