Page 1 of 3 123 LastLast
Results 1 to 20 of 50

Thread: ESF File Format notes

  1. #1

    Default ESF File Format notes

    Hi All,

    I'm starting to look at the various .esf files, specifically the regions.esf in data\campaign_maps\global_map

    Here's my (very!) preliminary notes on the structure:

    Code:
    byte[48] ???;
    byte ???; // Always 0E
    string unknown; // -1133129049
    
    byte[37] ???;
    
    // Region Repeat - 3 regions (America, India, Europe)
    for (int i = 1; i <= 3; i++) {
       byte[40] ???;
       dword numRegionsHere; // Changes for America, India, Etc
       dword ???;
       byte ???; // Always 0E
       
       for (int j = 1; j <= numRegionsHere; j++) {
          string regionName; // new_spain
          byte[14] ???;
       }
    }
    
    
    byte ???; // Always 0E
    string ???; // Long number (836795134)
    byte[44] ???;
    
    byte ???; // Always 0E
    string ???; // Long number (1197997136)
    byte[44] ???;
    
    byte ???; // Always 0E
    string ???; // Long number (2113354257)
    byte[56] ???;
    
    byte ???; // Always 0E
     string ???; // Long number (-1133129049)
    This is as far as I've gotten so far... more to come soon!

    If anybody else is working on these .esf files, please share notes!

    Thanks
    Delphy

  2. #2

    Default Re: ESF File Format notes

    Consider using the campaign.xml and campaign.xsd from testdata as a guide to what you should be able to find in a startpos.esf (However the order is not necessarily the same is in the xml). It seems likely that an xml similar to that one is used to generate the esf. The xsd is the specification of what the xml should look like.
    Last edited by just; March 20, 2009 at 03:24 AM.

  3. #3

    Default Re: ESF File Format notes

    I think someone tried deciphering the stratpos already.

    edit: Yep, Seraphim http://www.twcenter.net/forums/showthread.php?t=234722

  4. #4

    Default Re: ESF File Format notes

    Yeah I looked at that but unfortunately I couldn't map anything obvious into the .esf file - that and campaigns.xml contains regions that don't actually exist in regions.esf (like khorasan), make me think that it's not related.

    I suspect the top part of the file with the regions in it (the bits I've mapped out so far) is just an index table since each region is noted later on too with more detail.

    Jams, yeah thats a good find - unfortunately it seems like it's a different format (like the various db tables)
    Last edited by Delphy; March 18, 2009 at 04:50 PM.

  5. #5

    Default Re: ESF File Format notes

    I've just taken a look at these files, they seem to be somewhat generic container files, as they can contain widely varying data, startpos for the campaign, savegames, regions, etc.

    byte[0-3] : esf type, can be 0x0000ABCD or 0x0000ABCE
    These two types have a different layout for what's following.

    following offsets are ABCD / ABCE
    byte[4-7] / [12-15] : offset of list of strings (short count, repeating { short length, ascii string }), possibly the types of data that can be found in this esf
    byte[12-15] / [20-23] : the same offset to the list of strings

  6. #6

    Default Re: ESF File Format notes

    I found that in regions .esf the file format is listed at the very bottom and from my skim off the whole file it appears valid. I think it is in fact based off XML because of this structure.

    Code:
    root
        theater_and_region_keys
                 theater_region_keys
        theaters
                 climate_map
                 wind_map
                 transition_areas (defining regions of the map that allow you to port to other theaters)
                 transition_links (defining how the different areas link together)
                 black_shroud_outlines (???)
        region_data
                 vertices (This seems to be where the actual region geometry on the actual map is defined, the lookups seem to be just for the gui)
                           regions
                           areas
                           faces
                 outlines
                 connectivity
                 settlement_and_slots
                           slot_descriptions
                           roads
                           links
                           railways (Railways :) )
                           canals (I assume impassable rivers unless bridge exists)
        mountain_data
                 MOUNTAIN_OUTLINES
                           OUTLINE_BLOCK
                           BOUNDS_BLOCK
        climate_indices
        land_indices (What part of the map is land?)
        sea_indices (What part of the map is sea?)
        query_info
                 node? cell? (bottom_left, top_left, bottom_right, top_right)
        ground_types (Mapping the land to certain battle terrains?)
                 groundtype_indices_list
                 groundtype_index
        map_heights
                 MAP_HEIGHT_AREA
        trade_nodes (Probably the nodes in the trade theaters)
        bridges (Probably the yellow arrow bridges and normal bridges)
    Also I started to line up the different bits for the theater_and_region_keys section to see if any patterns exist which they do.

    Code:
    Header CEAB 0000 0000 0000 
    
    Region ID	00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 
    -----------------------------------------------------------------------
    
    Americas Regions
    (Theater Header)00 31 00 0C 00 00 3E C4 00 00 00 00 0C 00 00 AA C3 00 00 E6 43 01 00 81 03 00 00 EA 06 00 00 2A 00 00 00 C4 00 00 00 0E 0D
    -----------------------------------------------------------------------
    french_guyana 	00 0C E2 05 BA C3 EB 0C 03 42 EC 00 00 00 0E 0C 
    dutch_guyana  	00 0C 18 32 C4 C3 74 83 21 42 1A 01 00 00 0E 0F 
    trinidad_tobago	00 0C 5C 89 D9 C3 CA 00 94 42 44 01 00 00 0E 0D 
    new_andalusia	00 0C E2 A3 ED C3 DA D2 91 42 6A 01 00 00 0E 0B 
    new_grenada	00 0C A3 37 04 C4 C4 09 08 42 8C 01 00 00 0E 09 
    new_spain	00 0C C3 15 30 C4 C7 16 0A 43 A4 01 00 00 0E 04 
    cuba		00 0C B5 51 12 C4 F8 62 22 43 C2 01 00 00 0E 07 
    jamaica		00 0C B4 75 09 C4 A7 0C 01 43 E0 01 00 00 0E 07 
    bahamas		00 0C A0 E9 08 C4 64 96 2F 43 04 02 00 00 0E 0A 
    hispaniola	00 0C 8E DD F8 C3 8D A4 05 43 22 02 00 00 0E 07 
    curacao		00 0C 85 EA F4 C3 6E 3B AD 42 52 02 00 00 0E 10 
    windward_islands00 0C B9 20 D9 C3 C4 4E D0 42 80 02 00 00 0E 0F 
    leeward_islands	00 0C F6 13 DC C3 07 EC FA 42 A4 02 00 00 0E 0A 
    new_france	00 0C 2C 21 FD C3 F0 99 A7 43 CA 02 00 00 0E 0B 
    new_england	00 0C F8 76 FD C3 7A 84 96 43 F8 02 00 00 0E 0F 
    -----------------------------------------------------------------------
    European Regions
    (Theater Header)00 32 00 0C 00 00 32 C3 00 00 0C 43 0C 00 00 DC 43 00 00 02 44 01 00 81 03 00 00 9D 10 00 00 4D 00 00 00 39 07 00 00 0E 07
    -----------------------------------------------------------------------
    england		00 0C F6 58 46 3F 05 5C B7 43 55 07 00 00 0E 06 
    france		00 0C F8 2E 84 41 92 88 AD 43 6F 07 00 00 0E 05 
    spain		00 0C 28 EB D0 C1 C0 C4 8F 43 8F 07 00 00 0E 08
    -----------------------------------------------------------------------
    Indian Regions
    (Theater Header)00 33 00 0C 00 00 DC 43 00 00 A0 41 0C 00 00 2A 44 00 00 8C 43 01 00 81 03 00 00 F4 12 00 00 12 00 00 00 F0 10 00 00 0E 09
    -----------------------------------------------------------------------
    hyderabad	00 0C 44 83 0B 44 64 FB F6 42 0C 11 00 00 0E 06
    ceylon		00 0C EF A9 0F 44 E6 1A 70 42 28 11 00 00 0E 06
    mysore		00 0C 4C F3 07 44 B1 35 AD 42 44 11 00 00 0E 06
    orissa		00 0C 16 09 18 44 58 24 10 43 5E 11 00 00 0E 05
    Last edited by GeorgiaPeanuts; March 18, 2009 at 07:00 PM.

  7. #7

    Default Re: ESF File Format notes

    Hmm well some more working around and I found some little error in mine

    right before each string you will have the length of the string.

    0D - 13 - french_guyana
    0C - 12 - dutch_guyana
    0F - 15 - trinidad_tobago
    0D - 13 - new_andalusia

    and so on.

    so strings seem to be

    { E0 | (short size) | string | 0C }
    Last edited by GeorgiaPeanuts; March 18, 2009 at 08:44 PM.

  8. #8

    Default Re: ESF File Format notes

    Wow nice catch on the file format at the bottom! That certainly helps a lot. Now we just have to figure out the individual parts.

    You are right about the strings, it's definately 0x0E then a standard unicode string (2 bytes string length, with 2 bytes per character), followed by 0x0C. The 0E/0C are new to me - but the middle string format is the same as in the db tables.

    That leaves only 12 bytes per region, like so:

    Code:
    Americas Regions
    -----------------------------------------------------------------------
    french_guyana        E2 05 BA  C3  EB 0C 03  42   EC 00 00 00 (236)
    dutch_guyana         18 32 C4  C3  74 83 21  42   1A 01 00 00 (282)
    trinidad_tobago      5C 89 D9  C3  CA 00 94  42   44 01 00 00 (324)
    new_andalusia        E2 A3 ED  C3  DA D2 91  42   6A 01 00 00 (362)
    new_grenada          A3 37 04  C4  C4 09 08  42   8C 01 00 00 (396)
    new_spain            C3 15 30  C4  C7 16 0A  43   A4 01 00 00 (420)
    cuba                 B5 51 12  C4  F8 62 22  43   C2 01 00 00 (450)
    jamaica              B4 75 09  C4  A7 0C 01  43   E0 01 00 00 (480)
    bahamas              A0 E9 08  C4  64 96 2F  43   04 02 00 00 (516)
    hispaniola           8E DD F8  C3  8D A4 05  43   22 02 00 00 (546)
    curacao              85 EA F4  C3  6E 3B AD  42   52 02 00 00 (594)
    windward_islands     B9 20 D9  C3  C4 4E D0  42   80 02 00 00 (640)
    leeward_islands      F6 13 DC  C3  07 EC FA  42   A4 02 00 00 (676)
    new_france           2C 21 FD  C3  F0 99 A7  43   CA 02 00 00 (714)
    new_england          F8 76 FD  C3  7A 84 96  43   F8 02 00 00 (760)
    Edit - Last 4 bytes of each region data in the theater_and_region_keys section is definately an incremental value... notice how it goes 00 EC... 01 1A... 01 44... etc. The next entry now shown in the list above is also incremented.
    Last edited by Delphy; March 18, 2009 at 09:19 PM.

  9. #9

    Default Re: ESF File Format notes

    I looked at the climate_map and it seems to correspond to the climate_tables values; I guess per index they define the climate.

    It seems that they also define the land climate all together and then do the sea climate.

    Because first comes a big chunk of indices pointing to db entries with lc_

    then come a bunch with db entries of sc_ I assume being landclimate_ and seaclimate_

    between the land climate map and the sea climate map that I was looking at not sure if they split them per theater some data
    (America?)
    80 05 00 00 8A E5 00 00 08 25 01 00 00 08 B7 00 00 00 46 8A E5 00 00 (land climates start)(land climates end)
    80 05 00 00 14 B7 01 00 08 25 01 00 00 08 B7 00 00 00 46 14 B7 01 00 (sea climates start)(sea climates end)
    80 06 00 00 70 C2 01 00 08 30 00 00 00 08 1E 00 00 00 0A C3 C7 B8 3B 42 70 C2 01 00 (wind map????)
    (Europe?)
    80 05 00 00 A3 56 13 00 08 01 04 00 00 08 63 04 00 00 46 A3 56 13 00 (land climates start)(land climates end)
    80 05 00 00 1D E7 24 00 08 01 04 00 00 08 63 04 00 00 46 1D E7 24 00 (sea climates start)(sea climates end)
    80 06 00 00 B9 D8 25 00 08 A8 00 00 00 08 B8 00 00 00 0A 5E 13 EF 3B 42 B9 D8 25 00 (wind map????)
    (India?)
    80 05 00 00 55 4E 3B 00 08 EA 05 00 00 08 A0 03 00 00 46 55 4E 3B 00 (land climates start)(land climates end)
    80 05 00 00 AC BE 50 00 08 EA 05 00 00 08 A0 03 00 00 46 AC BE 50 00 (sea climates start)(sea climates end)
    80 06 00 00 48 E5 51 00 08 F8 00 00 00 08 98 00 00 00 0A 70 71 EF 3B 42 48 E5 51 00 (wind map????)

    (Trade Theaters for rest?)
    80 05 00 00 76 97 57 00 08 4A 02 00 00 08 7B 02 00 00 46 76 97 57 00
    80 05 00 00 1B 45 5D 00 08 4A 02 00 00 08 7B 02 00 00 46 1B 45 5D 00
    80 06 00 00 37 93 5D 00 08 60 00 00 00 08 68 00 00 00 0A 81 CA EE 3B 42 37 93 5D 00

    80 05 00 00 E7 3F 5E 00 08 D0 00 00 00 08 D0 00 00 00 46 E7 3F 5E 00
    80 05 00 00 FE E8 5E 00 08 D0 00 00 00 08 D0 00 00 00 46 FE E8 5E 00
    80 06 00 00 22 F2 5E 00 08 22 00 00 00 08 22 00 00 00 0A 46 FE B7 3B 42 22 F2 5E 00

    80 05 00 00 54 C7 5F 00 08 F4 00 00 00 08 DC 00 00 00 46 54 C7 5F 00
    80 05 00 00 1B 99 60 00 08 F4 00 00 00 08 DC 00 00 00 46 1B 99 60 00
    80 06 00 00 77 A4 60 00 08 28 00 00 00 08 24 00 00 00 0A 1C A0 E0 3B 42 77 A4 60 00

    80 05 00 00 7D 7B 61 00 08 F4 00 00 00 08 DC 00 00 00 46 7D 7B 61 00
    80 05 00 00 44 4D 62 00 08 F4 00 00 00 08 DC 00 00 00 46 44 4D 62 00
    80 06 00 00 A0 58 62 00 08 28 00 00 00 08 24 00 00 00 0A 83 1B EB 3B 42 A0 58 62 00
    Last edited by GeorgiaPeanuts; March 18, 2009 at 09:55 PM.

  10. #10

    Default Re: ESF File Format notes

    The 4 bytes at the start of the record (e.g. right before the 0E before french_guyana) are a pointer to the next record (which then also starts with pointer to the next record). This seems to be valid for every type of record throughout the file.

  11. #11

    Default Re: ESF File Format notes

    The byte before a value indicates the type, e.g.
    00 - 2 byte value
    01 - 1 byte value, possibly a boolean
    04 - 4 byte signed int, sometimes used for vertices, so then multiple shorter values must have been packed into it
    06 - 1 byte value
    07 - 2 byte unsigned int
    08 - 4 byte integer, possibly unsigned
    0A - 4 byte float
    0C - 8 byte, 2 single precision floats
    0E - varying, utf16 bstring
    0F - varying, ascii bstring
    10 - 2 byte value
    41,42,43,44,45,46,47,48,49,4A,4B,4C,4D
    All those indicate binary data, followed by a 4 byte integer that is the offset past the data, everything in between is the data. The different numbers maybe indicate different types of data.
    80 - sub section, exists only once, followed by 2 byte value which is the index into the list of section names. Followed by one byte of unknown function. Followed by a 4 byte integer of the next bit of data/the first byte after this sub section ends.
    81 - sub section, exists 0 or more times. Followed by 2 byte which is the index into the list of section names. Followed by one byte which is always 0, Followed by a pointer to where all instances of this sub section are over. Followed by 4 byte count of instances of this section (can be 0 !) After this the sections start, of course starting with a pointer to the next instance of this section or the next piece of data for the last section.

    These are all the value types I have encountered in parsing most of the ESF files that come with the game as well as some save games.
    Last edited by just; March 21, 2009 at 03:41 PM.

  12. #12

    Default Re: ESF File Format notes

    Good find, just! startpos.esf seems to have the same structure with the pointers to the next record. I always thought those 4 bytes belonged to the record before, but never could make any sense out of them.
    That will make trying to parse startpos a whole lot easier, 'cause now I know when the record ends and the next one begins.
    +rep!

  13. #13

    Default Re: ESF File Format notes

    I have even more, now you can almost start building an automatic parser:

    0x80, 0x81 and probably higher are new sections. Following are two bytes indicating the section name, 0 might be the root, and not mentioned in the sections list. After that is an unknown byte, possibly the amount of entries in the section.

    btw, now I'm not completely sure about type 0 and 1 anymore, I found one position where they don't seem to work right.
    Last edited by just; March 20, 2009 at 05:12 AM.

  14. #14

    Default Re: ESF File Format notes

    Updated with more information

  15. #15

    Default Re: ESF File Format notes

    Looks like pretty much can parse most the file now and then work out what each parameter is from there

    Nice

  16. #16

    Default Re: ESF File Format notes

    I noticed that the record for each settlement in the theater_and_region_keys points to the settlement data for that region further in the file.

    The format of the beginning seems to be that each theater has a pointer to its climate map start. You will see a 0C [XX XX XX XX XX XX XX XX] And I found that these values will point to the beginning of a climate map section. Then the pointer right after that in the climate section seem to then point to somewhere toward the bottom of the file, maybe a lookup for the indices for land/sea or maybe the groundtypes.

    Furthermore comparing some other ESF files it seems the file always start with CE AB then 14 bytes then the 80 00 00 00 which define the start of the root node of the file.

    Next come 4 bytes then the next node definition which is 81 01 00 00. From this I noticed that the 4 bytes before 80 00 00 00 match the 4 bytes after 80 00 00 00 and the 4 bytes after 81 01 00 00 as well. Both files then have a 07 00 00 so... XX meaning I don't know what it is


    Regions.esf
    CE AB 00 00 00 00 00 00 XX XX XX XX [4Bytes] 80 00 00 00 [4Bytes] 81 01 00 00 [4Bytes] 07 00 00 00 [*4Bytes] 80 02 00 00 [*4Bytes] "-1133129049" {Climate Map Pointer= 0C [8Bytes] 0C [8Bytes]} 01 01 (true?) | 81 03 00 00 | 69 00 00 00 00 00 00 [**4Bytes] 80 02 00 00 [**4Bytes] "1" {Next Climate Map Pointer = 0C [8Bytes] 0C [8Bytes]} 01 00 (false) | 81 03 00 00 [**4Bytes]

    in pathfinding.esf
    CE AB 00 00 00 00 00 00 XX XX XX XX [4Bytes] 80 00 00 00 [4Bytes] 81 01 00 00 [4Bytes] 07 00 00 00 [*4Bytes] 81 02 00 00 [Start some data I think] (*4Bytes can be found later in the file a few times.)

    in poi.esf
    CE AB 00 00 00 00 00 00 XX XX XX XX [4Bytes] 80 00 00 00 [4Bytes] [Start some data I think]

    in startpos.esf
    CE AB 00 00 00 00 00 00 XX XX XX XX [4Bytes] 80 00 00 00 [4Bytes] 80 01 00 01 | [*4Bytes] | "britain" "ui/portraits/european/cards/king/old/014.tga" | 08 [1] | 08 [1700] | "Summer" | "data\ui\flags\britain" | 81 02 00 00 | [*4Bytes] | 03 00 00 00 | [**4Bytes] | "america" ...
    Last edited by GeorgiaPeanuts; March 21, 2009 at 12:39 AM.

  17. #17

    Default Re: ESF File Format notes

    I'll try to explain it:

    In ABCD files the fixed part of the header is 4 bytes, in ABCE files it's 12. After that comes the offset to the list of section names.
    At this point the layout starts.
    80 -> new section, occurs once
    00 00 -> it's the 0th name in the list (start counting from 0) = root
    00 -> unknown
    4 bytes -> When you reach this point the section ends.
    81 -> new section, it can occur several times
    01 00 -> it's the 1st name in the list (start counting from 0) = theatres_and_region_keys
    00 -> unknown
    4 bytes -> The last of this type of section
    07 00 00 00 -> this section is repeated 7 times
    4 bytes -> where the end of the first section is
    80 -> new section, occurs once
    02 00 -> 2nd name = theatre
    00 -> unknown
    4 bytes -> end of this section
    0E -> here comes a utf16 string
    09 00 -> 9 characters long (18 bytes)
    18 bytes -> "628219853"
    0C -> here comes a point, consisting of two floats
    8 bytes -> -760, 0
    etc. etc.

    So the file, from what's decoded now looks like
    <root>
    ..<theatres_and_region_keys>
    ....<theatre>
    ......<entry type="string" value="628219853" />
    ......<entry type="point" x=-760 y=0 />
    ....</theatre>
    ..</theatres_and_region_keys>
    ..<theatres_and_region_keys>
    ..</theatres_and_region_keys>
    ..<theatres_and_region_keys>
    ..</theatres_and_region_keys>
    ..<theatres_and_region_keys>
    ..</theatres_and_region_keys>
    ..<theatres_and_region_keys>
    ..</theatres_and_region_keys>
    ..<theatres_and_region_keys>
    ..</theatres_and_region_keys>
    ..<theatres_and_region_keys>
    ..</theatres_and_region_keys>
    </root>
    Last edited by just; March 21, 2009 at 02:49 AM.

  18. #18

    Default Re: ESF File Format notes

    Btw in startpos.esf that unknown 00 is a 01

  19. #19

    Default Re: ESF File Format notes

    Yes, for one time occuring nodes the one unknown byte is different every time throughout the file.

    I can now parse all the esf files that come with the game (startpos.esf for the main campaign has over 1 million tags and 4 million values), I'll update the type identifiers now.

  20. #20

    Default Re: ESF File Format notes

    Btw I was parsing a bit myself and the file seems to be more like this structurally

    Code:
    [A0 DB D3 00] <root> [A0 DB D3 00]
    			<theater_and_region_keys> [BB 13 00 00] count = 7 (referring to theaters)
    				[69 00 00 00]<theater>[69 00 00 00]
    							<string>-1133129049</string>
    							<point x=800 y=-80 />
    							<point x=920 y=-5 />
    							<unknown>true</unknown>
    							<region_keys> [69 00 00 00] count=0?
    							</region_keys>
    					       </theater>
    				[EA 06 00 00]<theater>[EA 06 00 00]
    							<string>1</string>
    							<point x=-760 y=0 /> 
    							<point x=-340 y=460 />
    							<unkown>false</unknown>
    							<region_keys> [EA 06 00 00] count=42 unknown=196
    								<region name="french_guyana" x=-372.046 y=32.763 unk_val=236 />
    							</region_keys>
    					        </theater>
    </root>

Page 1 of 3 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •