ESF File Format notes

[Delphy]

Hi All,

I'm starting to look at the various .esf files, specifically the regions.esf in data\campaign_maps\global_map

Here's my (very!) preliminary notes on the structure:

byte[48] ???;
byte ???; // Always 0E
string unknown; // -1133129049

byte[37] ???;

// Region Repeat - 3 regions (America, India, Europe)
for (int i = 1; i <= 3; i++) {
   byte[40] ???;
   dword numRegionsHere; // Changes for America, India, Etc
   dword ???;
   byte ???; // Always 0E
   
   for (int j = 1; j <= numRegionsHere; j++) {
      string regionName; // new_spain
      byte[14] ???;
   }
}


byte ???; // Always 0E
string ???; // Long number (836795134)
byte[44] ???;

byte ???; // Always 0E
string ???; // Long number (1197997136)
byte[44] ???;

byte ???; // Always 0E
string ???; // Long number (2113354257)
byte[56] ???;

byte ???; // Always 0E
 string ???; // Long number (-1133129049)

This is as far as I've gotten so far... more to come soon!

If anybody else is working on these .esf files, please share notes!

Thanks
Delphy

[just]

Consider using the campaign.xml and campaign.xsd from testdata as a guide to what you should be able to find in a startpos.esf (However the order is not necessarily the same is in the xml). It seems likely that an xml similar to that one is used to generate the esf. The xsd is the specification of what the xml should look like.

[Jams79]

I think someone tried deciphering the stratpos already.

edit: Yep, Seraphim http://www.twcenter.net/forums/showthread.php?t=234722

[Delphy]

Yeah I looked at that but unfortunately I couldn't map anything obvious into the .esf file - that and campaigns.xml contains regions that don't actually exist in regions.esf (like khorasan), make me think that it's not related.

I suspect the top part of the file with the regions in it (the bits I've mapped out so far) is just an index table since each region is noted later on too with more detail.

Jams, yeah thats a good find - unfortunately it seems like it's a different format (like the various db tables)

[just]

I've just taken a look at these files, they seem to be somewhat generic container files, as they can contain widely varying data, startpos for the campaign, savegames, regions, etc.

byte[0-3] : esf type, can be 0x0000ABCD or 0x0000ABCE
These two types have a different layout for what's following.

following offsets are ABCD / ABCE
byte[4-7] / [12-15] : offset of list of strings (short count, repeating { short length, ascii string }), possibly the types of data that can be found in this esf
byte[12-15] / [20-23] : the same offset to the list of strings

[GeorgiaPeanuts]

I found that in regions .esf the file format is listed at the very bottom and from my skim off the whole file it appears valid. I think it is in fact based off XML because of this structure.

root
    theater_and_region_keys
             theater_region_keys
    theaters
             climate_map
             wind_map
             transition_areas (defining regions of the map that allow you to port to other theaters)
             transition_links (defining how the different areas link together)
             black_shroud_outlines (???)
    region_data
             vertices (This seems to be where the actual region geometry on the actual map is defined, the lookups seem to be just for the gui)
                       regions
                       areas
                       faces
             outlines
             connectivity
             settlement_and_slots
                       slot_descriptions
                       roads
                       links
                       railways (Railways :) )
                       canals (I assume impassable rivers unless bridge exists)
    mountain_data
             MOUNTAIN_OUTLINES
                       OUTLINE_BLOCK
                       BOUNDS_BLOCK
    climate_indices
    land_indices (What part of the map is land?)
    sea_indices (What part of the map is sea?)
    query_info
             node? cell? (bottom_left, top_left, bottom_right, top_right)
    ground_types (Mapping the land to certain battle terrains?)
             groundtype_indices_list
             groundtype_index
    map_heights
             MAP_HEIGHT_AREA
    trade_nodes (Probably the nodes in the trade theaters)
    bridges (Probably the yellow arrow bridges and normal bridges)

Also I started to line up the different bits for the theater_and_region_keys section to see if any patterns exist which they do.

Header CEAB 0000 0000 0000 

Region ID	00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 
-----------------------------------------------------------------------

Americas Regions
(Theater Header)00 31 00 0C 00 00 3E C4 00 00 00 00 0C 00 00 AA C3 00 00 E6 43 01 00 81 03 00 00 EA 06 00 00 2A 00 00 00 C4 00 00 00 0E 0D
-----------------------------------------------------------------------
french_guyana 	00 0C E2 05 BA C3 EB 0C 03 42 EC 00 00 00 0E 0C 
dutch_guyana  	00 0C 18 32 C4 C3 74 83 21 42 1A 01 00 00 0E 0F 
trinidad_tobago	00 0C 5C 89 D9 C3 CA 00 94 42 44 01 00 00 0E 0D 
new_andalusia	00 0C E2 A3 ED C3 DA D2 91 42 6A 01 00 00 0E 0B 
new_grenada	00 0C A3 37 04 C4 C4 09 08 42 8C 01 00 00 0E 09 
new_spain	00 0C C3 15 30 C4 C7 16 0A 43 A4 01 00 00 0E 04 
cuba		00 0C B5 51 12 C4 F8 62 22 43 C2 01 00 00 0E 07 
jamaica		00 0C B4 75 09 C4 A7 0C 01 43 E0 01 00 00 0E 07 
bahamas		00 0C A0 E9 08 C4 64 96 2F 43 04 02 00 00 0E 0A 
hispaniola	00 0C 8E DD F8 C3 8D A4 05 43 22 02 00 00 0E 07 
curacao		00 0C 85 EA F4 C3 6E 3B AD 42 52 02 00 00 0E 10 
windward_islands00 0C B9 20 D9 C3 C4 4E D0 42 80 02 00 00 0E 0F 
leeward_islands	00 0C F6 13 DC C3 07 EC FA 42 A4 02 00 00 0E 0A 
new_france	00 0C 2C 21 FD C3 F0 99 A7 43 CA 02 00 00 0E 0B 
new_england	00 0C F8 76 FD C3 7A 84 96 43 F8 02 00 00 0E 0F 
-----------------------------------------------------------------------
European Regions
(Theater Header)00 32 00 0C 00 00 32 C3 00 00 0C 43 0C 00 00 DC 43 00 00 02 44 01 00 81 03 00 00 9D 10 00 00 4D 00 00 00 39 07 00 00 0E 07
-----------------------------------------------------------------------
england		00 0C F6 58 46 3F 05 5C B7 43 55 07 00 00 0E 06 
france		00 0C F8 2E 84 41 92 88 AD 43 6F 07 00 00 0E 05 
spain		00 0C 28 EB D0 C1 C0 C4 8F 43 8F 07 00 00 0E 08
-----------------------------------------------------------------------
Indian Regions
(Theater Header)00 33 00 0C 00 00 DC 43 00 00 A0 41 0C 00 00 2A 44 00 00 8C 43 01 00 81 03 00 00 F4 12 00 00 12 00 00 00 F0 10 00 00 0E 09
-----------------------------------------------------------------------
hyderabad	00 0C 44 83 0B 44 64 FB F6 42 0C 11 00 00 0E 06
ceylon		00 0C EF A9 0F 44 E6 1A 70 42 28 11 00 00 0E 06
mysore		00 0C 4C F3 07 44 B1 35 AD 42 44 11 00 00 0E 06
orissa		00 0C 16 09 18 44 58 24 10 43 5E 11 00 00 0E 05

[GeorgiaPeanuts]

Hmm well some more working around and I found some little error in mine

right before each string you will have the length of the string.

0D - 13 - french_guyana
0C - 12 - dutch_guyana
0F - 15 - trinidad_tobago
0D - 13 - new_andalusia

and so on.

so strings seem to be

{ E0 | (short size) | string | 0C }

[Delphy]

Wow nice catch on the file format at the bottom! That certainly helps a lot. Now we just have to figure out the individual parts.

You are right about the strings, it's definately 0x0E then a standard unicode string (2 bytes string length, with 2 bytes per character), followed by 0x0C. The 0E/0C are new to me - but the middle string format is the same as in the db tables.

That leaves only 12 bytes per region, like so:

Americas Regions
-----------------------------------------------------------------------
french_guyana        E2 05 BA  C3  EB 0C 03  42   EC 00 00 00 (236)
dutch_guyana         18 32 C4  C3  74 83 21  42   1A 01 00 00 (282)
trinidad_tobago      5C 89 D9  C3  CA 00 94  42   44 01 00 00 (324)
new_andalusia        E2 A3 ED  C3  DA D2 91  42   6A 01 00 00 (362)
new_grenada          A3 37 04  C4  C4 09 08  42   8C 01 00 00 (396)
new_spain            C3 15 30  C4  C7 16 0A  43   A4 01 00 00 (420)
cuba                 B5 51 12  C4  F8 62 22  43   C2 01 00 00 (450)
jamaica              B4 75 09  C4  A7 0C 01  43   E0 01 00 00 (480)
bahamas              A0 E9 08  C4  64 96 2F  43   04 02 00 00 (516)
hispaniola           8E DD F8  C3  8D A4 05  43   22 02 00 00 (546)
curacao              85 EA F4  C3  6E 3B AD  42   52 02 00 00 (594)
windward_islands     B9 20 D9  C3  C4 4E D0  42   80 02 00 00 (640)
leeward_islands      F6 13 DC  C3  07 EC FA  42   A4 02 00 00 (676)
new_france           2C 21 FD  C3  F0 99 A7  43   CA 02 00 00 (714)
new_england          F8 76 FD  C3  7A 84 96  43   F8 02 00 00 (760)

Edit - Last 4 bytes of each region data in the theater_and_region_keys section is definately an incremental value... notice how it goes 00 EC... 01 1A... 01 44... etc. The next entry now shown in the list above is also incremented.

[GeorgiaPeanuts]

I looked at the climate_map and it seems to correspond to the climate_tables values; I guess per index they define the climate.

It seems that they also define the land climate all together and then do the sea climate.

Because first comes a big chunk of indices pointing to db entries with lc_

then come a bunch with db entries of sc_ I assume being landclimate_ and seaclimate_

between the land climate map and the sea climate map that I was looking at not sure if they split them per theater some data
(America?)
80 05 00 00 8A E5 00 00 08 25 01 00 00 08 B7 00 00 00 46 8A E5 00 00 (land climates start)(land climates end)
80 05 00 00 14 B7 01 00 08 25 01 00 00 08 B7 00 00 00 46 14 B7 01 00 (sea climates start)(sea climates end)
80 06 00 00 70 C2 01 00 08 30 00 00 00 08 1E 00 00 00 0A C3 C7 B8 3B 42 70 C2 01 00 (wind map????)
(Europe?)
80 05 00 00 A3 56 13 00 08 01 04 00 00 08 63 04 00 00 46 A3 56 13 00 (land climates start)(land climates end)
80 05 00 00 1D E7 24 00 08 01 04 00 00 08 63 04 00 00 46 1D E7 24 00 (sea climates start)(sea climates end)
80 06 00 00 B9 D8 25 00 08 A8 00 00 00 08 B8 00 00 00 0A 5E 13 EF 3B 42 B9 D8 25 00 (wind map????)
(India?)
80 05 00 00 55 4E 3B 00 08 EA 05 00 00 08 A0 03 00 00 46 55 4E 3B 00 (land climates start)(land climates end)
80 05 00 00 AC BE 50 00 08 EA 05 00 00 08 A0 03 00 00 46 AC BE 50 00 (sea climates start)(sea climates end)
80 06 00 00 48 E5 51 00 08 F8 00 00 00 08 98 00 00 00 0A 70 71 EF 3B 42 48 E5 51 00 (wind map????)

(Trade Theaters for rest?)
80 05 00 00 76 97 57 00 08 4A 02 00 00 08 7B 02 00 00 46 76 97 57 00
80 05 00 00 1B 45 5D 00 08 4A 02 00 00 08 7B 02 00 00 46 1B 45 5D 00
80 06 00 00 37 93 5D 00 08 60 00 00 00 08 68 00 00 00 0A 81 CA EE 3B 42 37 93 5D 00

80 05 00 00 E7 3F 5E 00 08 D0 00 00 00 08 D0 00 00 00 46 E7 3F 5E 00
80 05 00 00 FE E8 5E 00 08 D0 00 00 00 08 D0 00 00 00 46 FE E8 5E 00
80 06 00 00 22 F2 5E 00 08 22 00 00 00 08 22 00 00 00 0A 46 FE B7 3B 42 22 F2 5E 00

80 05 00 00 54 C7 5F 00 08 F4 00 00 00 08 DC 00 00 00 46 54 C7 5F 00
80 05 00 00 1B 99 60 00 08 F4 00 00 00 08 DC 00 00 00 46 1B 99 60 00
80 06 00 00 77 A4 60 00 08 28 00 00 00 08 24 00 00 00 0A 1C A0 E0 3B 42 77 A4 60 00

80 05 00 00 7D 7B 61 00 08 F4 00 00 00 08 DC 00 00 00 46 7D 7B 61 00
80 05 00 00 44 4D 62 00 08 F4 00 00 00 08 DC 00 00 00 46 44 4D 62 00
80 06 00 00 A0 58 62 00 08 28 00 00 00 08 24 00 00 00 0A 83 1B EB 3B 42 A0 58 62 00

[just]

The 4 bytes at the start of the record (e.g. right before the 0E before french_guyana) are a pointer to the next record (which then also starts with pointer to the next record). This seems to be valid for every type of record throughout the file.

[just]

The byte before a value indicates the type, e.g.
00 - 2 byte value
01 - 1 byte value, possibly a boolean
04 - 4 byte signed int, sometimes used for vertices, so then multiple shorter values must have been packed into it
06 - 1 byte value
07 - 2 byte unsigned int
08 - 4 byte integer, possibly unsigned
0A - 4 byte float
0C - 8 byte, 2 single precision floats
0E - varying, utf16 bstring
0F - varying, ascii bstring
10 - 2 byte value
41,42,43,44,45,46,47,48,49,4A,4B,4C,4D
All those indicate binary data, followed by a 4 byte integer that is the offset past the data, everything in between is the data. The different numbers maybe indicate different types of data.
80 - sub section, exists only once, followed by 2 byte value which is the index into the list of section names. Followed by one byte of unknown function. Followed by a 4 byte integer of the next bit of data/the first byte after this sub section ends.
81 - sub section, exists 0 or more times. Followed by 2 byte which is the index into the list of section names. Followed by one byte which is always 0, Followed by a pointer to where all instances of this sub section are over. Followed by 4 byte count of instances of this section (can be 0 !) After this the sections start, of course starting with a pointer to the next instance of this section or the next piece of data for the last section.

These are all the value types I have encountered in parsing most of the ESF files that come with the game as well as some save games.

[Scanian]

Good find, just! startpos.esf seems to have the same structure with the pointers to the next record. I always thought those 4 bytes belonged to the record before, but never could make any sense out of them.
That will make trying to parse startpos a whole lot easier, 'cause now I know when the record ends and the next one begins.
+rep!

[just]

I have even more, now you can almost start building an automatic parser:

0x80, 0x81 and probably higher are new sections. Following are two bytes indicating the section name, 0 might be the root, and not mentioned in the sections list. After that is an unknown byte, possibly the amount of entries in the section.

btw, now I'm not completely sure about type 0 and 1 anymore, I found one position where they don't seem to work right.

[just]

Updated with more information

[GeorgiaPeanuts]

Looks like pretty much can parse most the file now and then work out what each parameter is from there

Nice

[GeorgiaPeanuts]

I noticed that the record for each settlement in the theater_and_region_keys points to the settlement data for that region further in the file.

The format of the beginning seems to be that each theater has a pointer to its climate map start. You will see a 0C [XX XX XX XX XX XX XX XX] And I found that these values will point to the beginning of a climate map section. Then the pointer right after that in the climate section seem to then point to somewhere toward the bottom of the file, maybe a lookup for the indices for land/sea or maybe the groundtypes.

Furthermore comparing some other ESF files it seems the file always start with CE AB then 14 bytes then the 80 00 00 00 which define the start of the root node of the file.

Next come 4 bytes then the next node definition which is 81 01 00 00. From this I noticed that the 4 bytes before 80 00 00 00 match the 4 bytes after 80 00 00 00 and the 4 bytes after 81 01 00 00 as well. Both files then have a 07 00 00 so... XX meaning I don't know what it is


Regions.esf
CE AB 00 00 00 00 00 00 XX XX XX XX [4Bytes] 80 00 00 00 [4Bytes] 81 01 00 00 [4Bytes] 07 00 00 00 [*4Bytes] 80 02 00 00 [*4Bytes] "-1133129049" {Climate Map Pointer= 0C [8Bytes] 0C [8Bytes]} 01 01 (true?) | 81 03 00 00 | 69 00 00 00 00 00 00 [**4Bytes] 80 02 00 00 [**4Bytes] "1" {Next Climate Map Pointer = 0C [8Bytes] 0C [8Bytes]} 01 00 (false) | 81 03 00 00 [**4Bytes]

in pathfinding.esf
CE AB 00 00 00 00 00 00 XX XX XX XX [4Bytes] 80 00 00 00 [4Bytes] 81 01 00 00 [4Bytes] 07 00 00 00 [*4Bytes] 81 02 00 00 [Start some data I think] (*4Bytes can be found later in the file a few times.)

in poi.esf
CE AB 00 00 00 00 00 00 XX XX XX XX [4Bytes] 80 00 00 00 [4Bytes] [Start some data I think]

in startpos.esf
CE AB 00 00 00 00 00 00 XX XX XX XX [4Bytes] 80 00 00 00 [4Bytes] 80 01 00 01 | [*4Bytes] | "britain" "ui/portraits/european/cards/king/old/014.tga" | 08 [1] | 08 [1700] | "Summer" | "data\ui\flags\britain" | 81 02 00 00 | [*4Bytes] | 03 00 00 00 | [**4Bytes] | "america" ...

[just]

I'll try to explain it:

In ABCD files the fixed part of the header is 4 bytes, in ABCE files it's 12. After that comes the offset to the list of section names.
At this point the layout starts.
80 -> new section, occurs once
00 00 -> it's the 0th name in the list (start counting from 0) = root
00 -> unknown
4 bytes -> When you reach this point the section ends.
81 -> new section, it can occur several times
01 00 -> it's the 1st name in the list (start counting from 0) = theatres_and_region_keys
00 -> unknown
4 bytes -> The last of this type of section
07 00 00 00 -> this section is repeated 7 times
4 bytes -> where the end of the first section is
80 -> new section, occurs once
02 00 -> 2nd name = theatre
00 -> unknown
4 bytes -> end of this section
0E -> here comes a utf16 string
09 00 -> 9 characters long (18 bytes)
18 bytes -> "628219853"
0C -> here comes a point, consisting of two floats
8 bytes -> -760, 0
etc. etc.

So the file, from what's decoded now looks like
<root>
..<theatres_and_region_keys>
....<theatre>
......<entry type="string" value="628219853" />
......<entry type="point" x=-760 y=0 />
....</theatre>
..</theatres_and_region_keys>
..<theatres_and_region_keys>
..</theatres_and_region_keys>
..<theatres_and_region_keys>
..</theatres_and_region_keys>
..<theatres_and_region_keys>
..</theatres_and_region_keys>
..<theatres_and_region_keys>
..</theatres_and_region_keys>
..<theatres_and_region_keys>
..</theatres_and_region_keys>
..<theatres_and_region_keys>
..</theatres_and_region_keys>
</root>

[GeorgiaPeanuts]

Btw in startpos.esf that unknown 00 is a 01

[just]

Yes, for one time occuring nodes the one unknown byte is different every time throughout the file.

I can now parse all the esf files that come with the game (startpos.esf for the main campaign has over 1 million tags and 4 million values), I'll update the type identifiers now.

[GeorgiaPeanuts]

Btw I was parsing a bit myself and the file seems to be more like this structurally

[A0 DB D3 00] <root> [A0 DB D3 00]
			<theater_and_region_keys> [BB 13 00 00] count = 7 (referring to theaters)
				[69 00 00 00]<theater>[69 00 00 00]
							<string>-1133129049</string>
							<point x=800 y=-80 />
							<point x=920 y=-5 />
							<unknown>true</unknown>
							<region_keys> [69 00 00 00] count=0?
							</region_keys>
					       </theater>
				[EA 06 00 00]<theater>[EA 06 00 00]
							<string>1</string>
							<point x=-760 y=0 /> 
							<point x=-340 y=460 />
							<unkown>false</unknown>
							<region_keys> [EA 06 00 00] count=42 unknown=196
								<region name="french_guyana" x=-372.046 y=32.763 unk_val=236 />
							</region_keys>
					        </theater>
</root>