Guide to the ETW map

[uanime5]

Well, I used Matlab to open the metadata and saved it in a .m format. Can also save it in an excel file or, God forbid, a txt. the procedures were explained in this post:
http://www.twcenter.net/forums/showt...5#post10968855

EDIT: I have the values in the God forbidding .txt format if anyone is interested, they can be found here:
https://rapidshare.com/files/3949386309/metadata.txt

Just looked at the meta data. It contains 1,048,576 entries; 99.9% of them are 0 or 6. 1,048,576 entries is sufficient to form a 1024 by 1024 grid. This means it could be related to the world map, which is 2560 by 1280.

Below are all the numbers that are included; the number in brackets is how often they occur and the percentage in square brackets is how common these entries are.

84 (1) [0.000095%]
77 (1) [0.000095%]
73 (1) [0.000095%]
71 (1) [0.000095%]
8 (1) [0.000095%]
6 (210,808) [20%]
4 (1) [0.000095%]
1 (5) [0.00048%]
0 (837,755) [79.9%]

So if the metadata corresponds to a map of the whole world and 0 corresponds to an empty cell then nearly 80% of the map is blank.

mAIOR can you post some metadata from NTW for comparison?

[mAIOR]

This is NTW. I misplaced my ETW dvd and I don't have it in this account of Steam. I think I had another account... I can post the data from the Egypt campaign to compare it. Also, I didn't have time to look at the data again. Hopefully next week I'll do it. The metadata is usually connected to the map by giving relevant information about the images. An example of metadata for a georreferenced DEM generated by LIDAR is something like this:

Spoiler Alert, click show to read: 

ncols 191
nrows 149
xllcorner 457760.000000
yllcorner 4298094.000000
cellsize 2.000000
NaN -9999

It's easy to understand. the first two values are the number of rows and columns, then, the lower left pixel coordinates in UTM meters then the pixel size and then the value corresponding to lack of information (what I assume and you did too means empty places).

Tomorrow, I'll give an example of more complete metadata.



Cheers...

[Taraphir]

Please keep going gentlemen. You have my respect. I´m certain you will be triumphant soon!

[PietroMicca]

Please keep going gentlemen. You have my respect. I´m certain you will be triumphant soon!

Research is not a linear path, but rather a steeplechase. Now we need something makes us jump to the next step.

[Taraphir]

Research is not a linear path, but rather a steeplechase. Now we need something makes us jump to the next step.

Very well put PietroMicca. Unfortunately the only thing I can do is to advert to this thread
to attract not only attention but also some help in regard to the campaign map mystery.

[Nizam89]

Research is not a linear path, but rather a steeplechase. Now we need something makes us jump to the next step.

They told us that this game is the easiest modable tw game of all times once upon a time ....
a long time ago...

[PietroMicca]

They told us that this game is the easiest modable tw game of all times once upon a time ....
a long time ago...

The worst thing for me is that this game has a huge potential exploited only for 30-40%

[Taraphir]

The worst thing for me is that this game has a huge potential exploited only for 30-40%

You are absolutely right about Empire Total War´s potential PietroMicca. I think SEGA is very much afraid of the modding community to explore this potential too fast.Could also be related to the fact that Empire shares the Warscape Engine with Napoleon and Shogun 2. Modding research achievements for one of them has great effects for the other titles in regard to modification two.

[alhoon]

They told us that this game is the easiest modable tw game of all times once upon a time ....
a long time ago...

I think they told us it's the most modable which it is, not the easiest to mod.

[Slaxx Hatmen]

I think they told us it's the most modable which it is, not the easiest to mod.

"Most modable" and "easiest to mod" are pretty much mutual terms in this instance. What they meant by that statement(If a CA employee really said that at all) is that we could do a lot more with this game's code than with the previous TW titles and that is certainly not the case at all. CriagTW said about as much in official comment threads.(Apparently according to his sources the Warscape campaign map is even a workload for the CA staff themselves!)

I admire and commend these guys plugging away in this thread to try their best at cracking the map. I always check by here to see what progress has been made.

I'd really like to play Empire, Nappy, and Shogun2! However the lack of moddability of such key aspects of the game such as the map are keeping me from those purchases.

On topic:So what's the status on adding new regions fellas? Are we any closer to it than the last screenshot I saw of Cyprus?

[PietroMicca]

"Most modable" and "easiest to mod" are pretty much mutual terms in this instance. What they meant by that statement(If a CA employee really said that at all) is that we could do a lot more with this game's code than with the previous TW titles and that is certainly not the case at all. CriagTW said about as much in official comment threads.(Apparently according to his sources the Warscape campaign map is even a workload for the CA staff themselves!)

I admire and commend these guys plugging away in this thread to try their best at cracking the map. I always check by here to see what progress has been made.

I'd really like to play Empire, Nappy, and Shogun2! However the lack of moddability of such key aspects of the game such as the map are keeping me from those purchases.

On topic:So what's the status on adding new regions fellas? Are we any closer to it than the last screenshot I saw of Cyprus?

All or nothing. This is the real problem of esf files. What we know is much more than we can show. But without a complete knowledge (100%) is impossible make a working mod on campaign map. We can add a town or a slot. You will see on the map, but you will not able to enter or exit this building. Maybe I can add a region, but you will not able to move or pass through this land. And so on. All esf files are interwoven. In each file we find some reference which is located in another file. Lately I have discovered that pathfinding ID1 and ID2 are called back in startpos campaign-pathfinder. My research was adding-a-unit oriented, but I have to stop because of this obstacle. The most frustrating thing is that research is just the first phase. The second is create modding tools as Husserl is doing.

[Muizer]

Hello Pietro,

Forgive my ignorance. I've not looked at ETW mapping until maybe a week ago. I had a glance at the OP. I saw some of the data combines geographic coordinates and attributes. Would it be possible to reformat the data contained in these esfs into vector maps and related databases that can be handled by a GIS?

[PietroMicca]

If you are ignorant I am more.

The only thing that seems to refer to a GIS system is quadtree structure of data. Apart from that, any type of direct conversion need a tool to convert back data in esf format. This situation is even worse. We don't know how to convert into vector map. Unluckily there is no magical tool that allow us to read or to convert data in a comprehensible format. Anyway there's an important structure of database in this game. There is a sea_grid file that manages information about ports and sea regions. This file contains references about trade_routes contained in another esf file. There's a POI (Points Of Interest) file that I'm analyzing at this moment. It contains information about where are capitals, bridges, transition points etc.... As you can see information is scattered throughout different files and it's difficult to collect.

[alhoon]

before adding regions, have you tried to make a small theatre or something out of pure ocean? So that only ships would be able to go there.
Perhaps that would be easier to do and work with than a fullscale place that has land, sea, regions, cities, towns etc.

[PietroMicca]

before adding regions, have you tried to make a small theatre or something out of pure ocean? So that only ships would be able to go there.
Perhaps that would be easier to do and work with than a fullscale place that has land, sea, regions, cities, towns etc.

ok this means

add a new theatre in db tables
add a new theatre in regions.esf
add new transition areas in regions.esf
add a new theatre in pathfinding (even if empty or false like east_indies)
add a new theatre in sea_grids.esf (and in sea_grids there are sea_grid_connections: I have never understood how are calculated)
and maybe add something I don't know in poi.esf, trade_routes.esf and startpos.esf

and this is an example of how hard is add even something it seems simple to add.

[uanime5]

ok this means

add a new theatre in db tables
add a new theatre in regions.esf
add new transition areas in regions.esf
add a new theatre in pathfinding (even if empty or false like east_indies)
add a new theatre in sea_grids.esf (and in sea_grids there are sea_grid_connections: I have never understood how are calculated)
and maybe add something I don't know in poi.esf, trade_routes.esf and startpos.esf

and this is an example of how hard is add even something it seems simple to add.

You'll have to add it to the World and CAI_world in the startpos. The CAI_world tells the AI where everything is and has to be updated for any change you make to the map.

[PietroMicca]

You'll have to add it to the World and CAI_world in the startpos. The CAI_world tells the AI where everything is and has to be updated for any change you make to the map.

and probably we are forgetting something else

[uanime5]

and probably we are forgetting something else

Such as the DB tables in the main.pack which contains information about everything in the campaign and even some things which were cut at the last minute (mainly factions).

[alhoon]

Riiight.
So the answer is: "Perhaps after cracking a couple more things and doing a week's worth of work, banging our heads each time we forget something; obviously if we have figured the procedure correct"

However, that is a far more possitive answer than "nope".

[PietroMicca]

Spoiler Alert, click show to read: 

This is a picture of sea_grids in India. I have add area ID for each quad.

There's a correlation between those areas called sea_grid_connections. Each connection has a float value. But I cannot understand how is calculated.

96 97 0
96 98 0
96 99 0
96 100 68,83662
96 101 158,64238
96 102 158,64238
96 103 158,64238
96 104 68,83662
97 98 0
97 99 0
97 100 101,38612
97 101 98,78085
97 102 98,78085
97 103 98,78085
97 104 101,38612
98 99 0
98 100 173,37563
98 101 51,003685
98 102 51,003685
98 103 51,003685
98 104 173,37563
99 100 217,25636
99 101 69,12314
99 102 69,12314
99 103 69,12314
99 104 217,25636
100 101 272,01038
100 102 272,01038
100 103 272,01038
100 104 0
101 102 0
101 103 0
101 104 162,42484
102 103 0
102 104 266,19022
103 104 329,888

I have always thought that this value could be distance between areas. This was confirmed by some example. Distance between 96 and 97 is 0, but why connection between 96 and 100 is 68,83662?

Any idea?