I've been trying to figure out how the ETW maps works but I'm finding it difficult to progress further, so I decided to post everything I've found out to see if someone else can figure out the rest of this.
In the regions.esf I noticed that many of the coordinates showing where various regions were located on the ETW's map contained negative coordinates. As ETW cannot have regions that are not on the map I realised that the zero point must be in the centre of the map, and that the negative coordinates represented left or down from the centre, while the positive coordinates represented right or up from the centre. This is demonstrated on the map below (made from the campaign_detail_mask.dds).
Spoiler Alert, click show to read:
While I don't know why CA choose to use this method I believe that it may be related to latitude and longitude, as their respective zero points are near to ETW's zero point.
Thought the exact area of the map is unknown as the Pacific Ocean region covers an area 2560 by 1131, it is highly likely that the map is 2560 by 1280. To test whether this was correct I enlarged the 2048 x 1024 campaign_detail_mask.dds to 2560 by 1280, resized the lookup.tga files to be the size of the theatres as listed in the regions.esf, and put on them on the enlarged campaign_detail_mask.dds at the coordinates listed in the regions.esf. As shown on the map below shows these theatres fit these coordinates perfectly.
Spoiler Alert, click show to read:
Each region is made up of one of more areas, which are used to tell ETW which islands belongs to which region and when two regions overlap they tell ETW which cities belong to which region. After adding several areas to the above map I found that the areas had a large degree of overlap, which would prevent them having the precise borders that can be observed in ETW. So there must be another entry which tells ETW what shape the borders should be.
The shape of the borders may be related the connectivity in the regions.esf or the 'land.rigid_mesh'.
While checking the regions mentioned in the regions.esf I wondered whether CA had made a world map but had hidden some of the unplayable regions. To test this I edited the coordinates of the India theatre so that I could travel beyond the wooden border. I found that to the right of the Indonesia region there was a Chinese Empire region and a South Chinese Sea. This means that ETW's map contains may additional regions that are not visible without modding.
The main findings were that some of the numbers in the poi.esf were coordinates for the locations of settlements, ports, towns, and forts for a 2684354560 by 1342177280 map. So the purpose of the Poi.esf is to tell ETW where everything is on the map and how far away it is. This is probably so the AI can make better decisions and won't wander around aimlessly.
UPDATE: 22nd May 2010
alpaca has posted the following:
Originally Posted by alpaca
Anyhow, the borders in the game are defined using rigid splines. Each region defines its own borders as a sequence of points, which are saved as .rigid_spline files and stored in models.pack/rigidmodels/campaignborders. The same is true for roads and rivers. Not sure about bridges, they use a different method.
The height map is in the main pack.
So this means that we'll need to make new .rigid_spline files to add new regions.
UPDATE: 7th November 2010
I've done further testing on how ETW determine where each settlement is located and what needs to be edited. In the regions.esf for each city, town, port, resource there are coordinates that tell ETW where the strat model is located on the map. There are also 8 coordinates that tell ETW the area of the interactive area (not visible using esfeditor 1.45). Below is an explanation of these coordinates.
Code:
Coordinates for Baghdad in Mesopotamia: 314.25 ; 238.9004
8 Coordinates for Baghdad's octagon: [313.8965 ; 239.8903 / 314.6035 ; 239.8903 / 315.24 ; 239.2539 / 315.24 ; 238.5468 / 314.6035 ; 237.9104 / 313.8965 ; 237.9104 / 313.26 ; 238.5468 / 313.26 ; 239.2539 ]
Difference between the original coordinates and each octagon entry:
-0.3535 ; +0.9899
+0.3535 ; +0.9899
+0.99 ; +0.3535
+0.99 ; -0.3536
+0.3535 ; -0.99
-0.3535 ; -0.99
-0.99 ; -0.3536
-0.99 ; +0.3535
There's also the CAI interactive which I believe tells the AI where everything is located. I'm still investigating whether this needs to be updated.
UPDATE: 31st December 2010
I've figured out what each of the CAI interface/CAI world entries are for.
Spoiler Alert, click show to read:
All CAI entries contain BDI entries so different factions can determine how much they want each region. Each of these entries determine the following:
CAI_world_theatre: theatres.
CAI_world_factions: factions.
CAI_world_regions: all regions (including the unexplorable ones), their locations, which faction controls them, and what they contain.
CAI_region_boundaries: the length of the border neighbouring regions share.
CAI_world_settlements: locations of settlements (capital of region), which theatre they're in, which region they're in, what area they're in, which faction owns them, whether they're garrisoned, and their building_slots.
CAI_world_building_slots: number identifying which building in the Region Manager they are, and the CAI_world_settlements or CAI_world_region_slots they're paired with.
CAI_world_region_slots: locations of region_slots (anything in a region that isn't the settlement), which theatre they're in, which region they're in, what area they're in, which faction owns them, whether they're garrisoned, and their building_slots.
CAI_world_trading_posts: where the trade resources are located. What they trade is controlled by the campaign_trade_manager/theatres.
CAI_world_resource_mobiles: locations of ships and armies, which theatre they're in, which region they're in, what area they're in, and which faction owns them.
CAI_world_units: tell ETW which resource_mobiles corresponds to which character.
CAI_world_character: which faction control this character.
CAI_world_forts: locations of forts, which theatre they're in, which region they're in, what area they're in, which faction owns them, and whether they're garrisoned.
CAI_world_technology_trees: the technology each factions possesses.
CAI_world_governorship: how many governors each factions possesses (they need 1 per theatre).
CAI_world_transition_areas: where the transition areas are located.
CAI_world_HLCIS: locations of areas, which theatre they're in, which region they're in.
CAI_world_sea_grid_cells: whether a sea grid cell contains a ship.
If you delete the BDI_pool entries in the CAI_world entries ETW will make new BDI_pool entries. So if you want to add new towns/ports/resources or increase the number of buildings in a settlement you don't need to update the BDI entries, you can simply delete them.
The black barrier and borders are caused by the stpd and stpdi files in the supertexture.pack. This file also where the sea is located (purely cosmetic) but not the rivers, mountains, or trees. Unless these files can be edited it will be impossible to make new maps for ETW, though the existing one can still be edited if someone can figure out how to edit them.
I'm a little late posting this but all the coordinates for the regions, region boundaries, and rivers are in the regions.esf. Taw was able to show this by using a converter to turn all the coordinates into an image.
Regarding mapping until we figure out how the pathfinding.esf works, especially the grid_data, we won't be able to add any new towns, ports, resources, settlements, or regions to the map. Though we can add more buildings to existing settlements.
Ulsterman has found that you can use the pathfinding information of existing buildings for new buildings. This removes the problem of units being unable to leave new buildings but means that all units can only enter and exit the new buildings using the old building. http://www.twcenter.net/forums/showt...65#post9819365
UPDATE 6th May 2013
I have used the official tools to move a farm in TWS2. I have posted all the changes between the vanilla version of the pathfinding and my modified version here: http://www.twcenter.net/forums/showt...1#post12820027
Summary
1) ETW's full map 2684354560 by 1342177280. This map is -1342177280 to 1342177280 along the X axis (left to right) and -671088640 to 671088640 along the Y axis (bottom to top).
2) ETW's has a smaller map that's 2560 by 1280. This map is -1280 to 1280 along the X axis (left to right) and -640 to 640 along the Y axis (bottom to top).
3) All the coordinates on the smaller map are the full map's coordinates divided by 1048576 (2^20). The maximum length of a regions.esf coordinate is 7 digits (392.3914, 40.37837, etc). This means 1 pixel on the smaller map represent over 1 million pixels on the larger map.
4) ETW's map covers the whole world.
5) ETW's map is divided into 7 theatres, the 3 playable theatres are divided into several regions, each region is divided into 1 or more areas.
6) To prevent people seeing some unplayable regions CA has done the following:
6A) Restricted the the player so they can only move the cursor over a few parts of the world (the 7 theatres).
6B) Covered the unused regions with a black barrier to hide them. The black barrier and the theatre borders are created by the stpd and stpdi files in the supertexture.pack
7) Regions.esf determines where all the regions are; the region borders; whether a region is land or sea; which theatre the region belongs to; where all the rivers are located; how many areas each region has; where all the ports, towns, settlements, and resources are; and possibly which battle maps are used.
8) Startpos.esf determines what level all the buildings are at and their maximum level. The BDI_pool entries control how much an AI faction wants to do anything.
9) Pathfinding.esf tells ETW where units can walk, where ships can sail, and where all the settlements/towns/ports/resources are on the campaign map.
10) Poi.esf tells the AI how to get to different regions.
11) Sea_grids.esf tells ETW which regions each port can trade with and which seas it's connected to.
12) The heightmap.tga in the main.pack determines the height of the land on the campaign map. It has no effect on the sea.
13) The coastline_group and coastline_group.cs2 rigid_mesh files tell ETW how the waves should hit the coastlines.
14) The rigid_spline files tell ETW the shape of each region (their borders). However they are purely cosmetic.
15) ETW is constantly updating the BDI_pool entries, so anything added to ETW does not require new BDI_pool entries.
16) The stpd and stpi files form a giant image that is painted onto the ETW world.
Last edited by uanime5; May 18, 2013 at 04:11 PM.
Reason: Additional research
Hopefully this will lead to some further advances in the modding of ETW
+Rep
"Aye Sir?" - Stronghold
EVGA GTX 295
i7 920
6 GB DDR3 @1600
HAF 932 Full Gaming Tower Case
1.5 TB HD W/ 7200 rpm
I never beg for ReP... I just ask for it
----------------------------------
nice information uanime5, I'd give you rep but I already did years ago for your conquest mod
does this mean that we may be able to add meshes to these expanded regions into the game?
I wish CA would release documentation on how to do this..
Has anyone looked over all of the NTW patch packs and other .pack files since the last patch 3 update?
I have never seen this before (not that I was looking for it) but there is a rigidmesh.pack file in the main NTW data folder that has rigid_splines for all of the regions. I think CA might be slipping things in (unlocking) without telling anyone which was why I was looking. I saw the files for the uniteditor in there as well. Opening up the meshes it appears to not be complete yet.
There are also all of the rigid splines for roads, rivers, coastlines, etc. These files can be opened in Cinema or other 3d programs (which I don't currently have).
Maybe this pack has been there all along but thought I'd point it out.
I'll have to check that out. I suspect that understanding this mesh file will be an important part of understanding how the map works and how to improve it.
UPDATE: the first post has been updated with additional information about the size of the full map ( 2684354560 by 1342177280) and what the poi.esf does.
There is also a fogmask.dds file in the campaign map folders (one for each campaign). Is this the fog that prevents you from seeing the non-playable regions or is it an actual fog "weather" effect on the campaign map?
Did Steam download the Warpath campaign to everyone's ETW installation and then only those that pay for the campaign can have it unlocked?
If so, a good reference source to study would be the changes between the ETW America map and the Warpath America map. It could lead to a gold mine of information how to add a region. If we knew exactly which tables and files were used to incorporate the Warpath changes that will pretty much spell out what we need to be researching to add regions to a map.
I did not buy the warpath campaign so I cannot check for the before and after tables/files. Can someone else pick up on this that has purchased the Warpath campaign?
There is also a fogmask.dds file in the campaign map folders (one for each campaign). Is this the fog that prevents you from seeing the non-playable regions or is it an actual fog "weather" effect on the campaign map?
I'm not sure. I tried editing the map_visibility file but it didn't let me see any more of the map.
Originally Posted by Hazbones
I also have a question and observation:
Did Steam download the Warpath campaign to everyone's ETW installation and then only those that pay for the campaign can have it unlocked?
If so, a good reference source to study would be the changes between the ETW America map and the Warpath America map. It could lead to a gold mine of information how to add a region. If we knew exactly which tables and files were used to incorporate the Warpath changes that will pretty much spell out what we need to be researching to add regions to a map.
I did not buy the warpath campaign so I cannot check for the before and after tables/files. Can someone else pick up on this that has purchased the Warpath campaign?
Warpath was downloaded onto everyone's ETW installation, though only people who purchase it can play it.
Did Warpath add any additional regions? If it did then if we compare the Warpath files to the episode 1 files this will make any changes obvious because both are set only in the US.
Yes the Warpath campaign added regions to the US map. I did not purchase Warpath so cannot compare the maps. If you can compile a list of all the files that were used to update for the Warpath addition you'll have all the files identified that need to be modded to add regions.
Just checked the warpath files and you were right, it does contain more regions. Below are the maps of America from the vanilla ETW (left) and Warpath (right) which show how the two campaigns differ.
After checking the regions.esf I have learned the following.
These regions were renamed (probably so the regions would have different names on the map). The left name is original ETW name, the right name is the Warpath name.
For unknown reasons the northwest territories was moved to the bottom of the list of files in the Theatres_and_region_keys. Some of the 'Region_data/regions' entries are in a different place as well.
The Theatres_and_region_keys entries all have the same coordinates but some of the regions' Region_data/regions coordinates are different.
The only difference between the ETW and Warpath versions seems to be that a different 'campaign_maps' and 'campaigns' folders are used. So only the following need to be edited to add new regions.
I would guess that if we wanted to create our own regions we will have to mod folder our changes as CA did with the Warpath campaign and map.
This also means that the borders are defined in one of those esf files which you already assumed.
I guess one could try fooling around with the coordinates in the regions esf and see if tweaking them moves the region border or not?
How do the db tables play into this whole esf setup?
I will see if I can find correlations between the db tables and the esf files...
Last edited by Hazbones; May 09, 2010 at 07:27 PM.
I would guess that if we wanted to create our own regions we will have to mod folder our changes as CA did with the Warpath campaign and map.
There's an entry in the stapos.esf that tell ETW which map to use so making a new map folder isn't difficult.
This also means that the borders are defined in one of those esf files which you already assumed.
I guess one could try fooling around with the coordinates in the regions esf and see if tweaking them moves the region border or not?
In the regions.esf for each area there is an Outline folder with several Connectivity entries. I think that borders are controlled by the these entries but I don't understand how they work.
How do the db tables play into this whole esf setup?
I will see if I can find correlations between the db tables and the esf files...
I'm having trouble reading the db tables because PackFileManager will not longer display them and tells me to download DBEditor, and DBEditor can only display packs 4 and 5. Not sure how to fix this.
Use PFM from Napoleon thread we've developed further. There are many relations between startpos and pack but not much about our research, except some obvious lists like regions, settlements, ports, etc.
In the regions.esf for each area there is an Outline folder with several Connectivity entries. I think that borders are controlled by the these entries but I don't understand how they work.
I saw those connectivity entries as well but I was zipping through them and cannot recall what info they show? Do you know what they are displaying?
Does anyone have a 3D program that will open the spline files? Really any of them can open a spline I just don't have one... I really want to see what those splines look like and if they give coordinates.
I saw those connectivity entries as well but I was zipping through them and cannot recall what info they show? Do you know what they are displaying?
Does anyone have a 3D program that will open the spline files? Really any of them can open a spline I just don't have one... I really want to see what those splines look like and if they give coordinates.
Have you tried the rigid_mesh converter? It let me look at the land.rigid_mesh, though I can't edit anything.