Guide to the ETW map

[taw]

> Due to the difference in file type between ETW and TWS2 regions.esf and startpos.esf files I wasn't able to convert them to xml files so it will be more difficult to see the differences between these files.

Does it crash or just produces something silly? At least startpos.esf should convert more or less properly.If it crashes, could you send me the files and I'll take a look and make them not crash?

[uanime5]

I tried making a version of the pathfinding.esf that just altered the parts where the farm was located and didn't change all the unknown2 information. TWS2 wouldn't load so either I made a mistake or you do need to modify the unknown 2 information.

Does it crash or just produces something silly? At least startpos.esf should convert more or less properly.If it crashes, could you send me the files and I'll take a look and make them not crash?

1) The regions.esf it unpacks part of the file then crashes when it gets to "region_data-0001.xml".

It also produces this error message:

C:\jruby-1.6.7.2\esfxml>jruby --server ./esf2xml regions.esf
RuntimeError: Unknown type 0x1e at 3708796

2) The startpos.esf it unpacks part of the file but cannot unpack the compressed part. So you end up with a "compressed_data.esf.xz".

It also produces this error message:

Warning: Semantic conversion failures (low level conversion performed instead)
(this is OK if converting ESFs for game newer than Empire):
* MAPS: (1 records, 1 failures, 0 quiet failures)

3) The pathfinding.esf will unpack correctly though the <u4_ary> is a large block of text, rather than showing which entry belongs to which coordinates.

[taw]

I tried making a version of the pathfinding.esf that just altered the parts where the farm was located and didn't change all the unknown2 information. TWS2 wouldn't load so either I made a mistake or you do need to modify the unknown 2 information.



1) The regions.esf it unpacks part of the file then crashes when it gets to "region_data-0001.xml".

It also produces this error message:

C:\jruby-1.6.7.2\esfxml>jruby --server ./esf2xml regions.esf
RuntimeError: Unknown type 0x1e at 3708796

2) The startpos.esf it unpacks part of the file but cannot unpack the compressed part. So you end up with a "compressed_data.esf.xz".

It also produces this error message:

Warning: Semantic conversion failures (low level conversion performed instead)
(this is OK if converting ESFs for game newer than Empire):
* MAPS: (1 records, 1 failures, 0 quiet failures)

3) The pathfinding.esf will unpack correctly though the <u4_ary> is a large block of text, rather than showing which entry belongs to which coordinates.

Email me the failing files (or upload them somewhere). I don't have much free time these days, but I'll try to fix at least crashes and compression failures.

0x1e is not one of known data types, so they must have added it recently, unless it's actually a bug elsewhere causing reading frame error.

I probably won't have time for the third one anytime soon.

[uanime5]

CA changed how the esf files worked after releasing TWS2, so that's probably why the unpacker is having problems.

I've attached the compressed data from the startpos.

Regarding point 3 your unpacker can already turn the <uary4> in the pathfinding_areas from a block of numbers, into something that contains coordinates. I'm not sure why it isn't doing this in TWS2.

[taw]

I tried esf2xml on your compressed_data.esf.xz, on S2's startpos.esf, regions.esf, pathfinding.esf - and they all work.

So I can't reproduce your problem, so I can't fix it.

You'll need to send me entire failing .esf files. I have Shogun 2, but none of the DLCs for it.

[uanime5]

In the original version of TWS2 the esf files were the same as the ETW ones, however in a later patch they replaced them with the new version. So to find these files you need to either look through every patch pack or create an esf file using the official tools.

I've uploaded the latest version of these esf files.

[taw]

In the original version of TWS2 the esf files were the same as the ETW ones, however in a later patch they replaced them with the new version. So to find these files you need to either look through every patch pack or create an esf file using the official tools.

I've uploaded the latest version of these esf files.

I'll take a look at them them.

[taw]

I fixed regions.esf crash.

regions.esf and pathfinding.esf have the same problem with semantic analysis of packed u4_ary - the code for dealing with arrays is pretty messy for performance reasons. I'll figure something out when I have a bit of free time.

startpos.esf you sent unpacks perfectly including compressed data. What exactly is the problem with it?

[uanime5]

startpos.esf you sent unpacks perfectly including compressed data. What exactly is the problem with it?

The main problem is that when I unpack the startpos is that I end up with a compressed_data.esf.xz file and I have no idea how to unpack it. Not sure why we're getting different results.

[taw]

Well, that's not a surprise since esf2xml converts ESF file, and doesn't directly deal with ESF-within-lzma-within-ESF situation.

Use unpack_nested_esf/pack_nested_esf for that.

I got annotations working with new style arrays too, I'll upload the code to github once I check it on all esfs.

[taw]

I uploaded the new code. Unfortunately it seems 10-20% slower, but it annotates new Shogun 2 files correctly.

[Puppy Monster]

Hey pioneers,

I just found a way to put vertices from ntw's regions.esf/mountains.xml into 3dmax as a polygon. I assume I can do the same for all other vertices. From this, can I more or less build the .cs2 files needed to remake the map for assembly kit use?
(btw, I m also assuming I would be able to do the same with etw)

[Puppy Monster]

Sorry for double post.
I have here the .cs2 files from raw_data for stw2 exported into Collada xml format (.dae) from 3dmax.
I assume there will be some data loss in the process (3d shapes i think come out a bit weird) but the basics are all there.
My hypothesis is that imputing the relevant data into these files (using them as templates) from ntw/etew esfxml files would provide the basis for us to rebuild the maps using the assembly kit.
Sorry for double post.

[uanime5]

Decided to import these .dae files into 3D Max to see how they looked. Here's how the regions, settlements/town/ports/farms/trade nodes, mountains, and bridges cs2 files look.

Spoiler Alert, click show to read: 

Here's how the original CS2 versions looked.

Spoiler Alert, click show to read: 

So it's clear that converting CS2 files into dae files creates something that's a similar shape to the original. Therefore if it's possible to make ETW/NTW's esf files into dae files then this will allow modders to easily make a version of the ETW/NTW maps that can be edited using the official tools.


However I feel I should point out a few problems with the dae files.

1) It adds "_pivot" to the names of all the settlements, meaning every settlement will have to be renamed or you won't be able to convert them.

2) The dae files completely destroy the borders, coasts, land trade, rivers, roads, and trade route CS2 files. So either you'll have to added this information in manually or these files will have to be converted in another way.

Here is how this cs2 files look in the dae version.

Spoiler Alert, click show to read: 

and here is show they originally looked.

Spoiler Alert, click show to read: 

So while this method works with shapes it doesn't work with a string of vertices.

3) None of the material are in the material editor, which means that none of the polygons in the regions.dae have been assigned to a region. While this isn't a major problem it may be time consuming to assign all the polygons to regions.


So we need a way to create dae files that doesn't result in all the vertices or materials being lost.

[alhoon]

Uanime5, weren't you able to create a new town (Corn resource) from scratch anyway?

[uanime5]

Uanime5, weren't you able to create a new town (Corn resource) from scratch anyway?

While I was to add a new town I could only get the pathfinding to work by replacing an impassible area with a new town. Using the official tools I can add towns wherever I want and can even split regions.

http://www.twcenter.net/forums/showt...ion&p=12845189

So the official tools make mapping a lot easier.

[Fridericus Secundus]

Very interesting. Just for understnading: the above linked post is for Shogun 2 - is ETW now moddable in the same way or is that a future task?

[uanime5]

Very interesting. Just for understnading: the above linked post is for Shogun 2 - is ETW now moddable in the same way or is that a future task?

TWS2 is moddable because CA provided use with all the files needed to mod it (Tweak entries and CS2 files). ETW isn't moddable this way because we don't have these files. So while it should be possible to make an ETW map using these tool it will be a lot of work.

[Puppy Monster]

heyall,
just an update to the .dae idea. I have been unable to find common verticies between the exported .cs2 files and the esfxmls from stw2 (feel free to try and find some yourselves). However, using .ase export format, the verticies are correct (also, its a nice format). However, while 3dmax is able to export to .ase, there is no east way for it to import the file.
Therefore, I am now trying the following:
Seeing if the apparent change in verts has an effect on the map creation process (also checking to see whether the loss of data (the lines inbetween the roads, trade routes verts etc) affect map creation)
Try to find either a way to import .ase files into 3dmax, or file an alternative format.

[vonVince]

So - pardon my ignorance (and lack of time! I simply don't have the time right now to read through all these pages) - but has someone actually succeeded in expanding the map of Empire (and possibly Napoleon) with a mod?