Unit creation methods (ETW, NTW) and their implementation

[Sirlion]

Hello. Since I see that many people look into this matter and there is no clear tutorial out there, I've decided to put together this small guide that is extremely simple and illustrates the pros and cons of ETW and NTW methods of unit creation. One thing to note, that I didnt think was possible before, is that both methods can be used at the same time, thus, creating hybrid units is indeed possible. But I'll go deeper into that matter in a moment.
NOTE: this guide assumes that you have successfully installed and used PFM (Pack File Manager) before. Use 1.5.7 for Napoleon and any other version (the most recent is perfect) for Empire.




COMMON GROUNDS


Needed by both methods of unit creation, you must compile the "units" table under "units_tables" in the database tables. It is really easy and does not require any particular explanation as every line is conveniently labelled for you by PFM. 1.5.7 does not have the correct labels, however writing down what the most recent version of PFM tells you and comparing the same lines you should have no problems at all.
Another thing needed by both methods is the UI icon located in the "ui\units\icons" and the info picture located in "ui\units\info". That's all there is to it really, pretty straightforward. Remember that both must match the name of your unit plus the "_icon" or "_info" suffix. I dont think I need to tell you what those are for.




UNIT CREATION METHOD 1 - Empire


Okay, things get... well, surprisingly easy to be honest. ETW's method of unit creation is similar to MTW2 and does not require any additional program beside Milkshape3D and \ or an external modelling program like Blender that can convert and import back .variant_weighted_mesh files, which is the format that Empire uses to compress the model data. The easiest route is picking Milkshape3D version 1.8.4, and work on your model there, converting it to ms3d format readable by the program with KnightErrand's very useful tools. With ETWrigid_modelConverter_v1_0_beta you can convert to and from ms3d and .variant_weighted_mesh without the need for Ultimate Unwrap 3D, which is the only other program that is capable of converting all of ETW's formats.
The disadvantages of UU3D, in regards to units, are two:


1- it costs money
2- converting ETW unit models require you to revert all vertices and unit parts from left to right (I have no idea why this happens)


So you see, although it is very useful for UV mapping and general model conversion, UU3D has its drawbacks. The way I do it, I stick with the combo KnightErrand's tools + Milkshape3D which is by far the quickest.
I'm not going to teach you how to create UV maps or model a proper human being, that is beyond the scope of this tutorial. However, I'll tell you the main advantage of this method: you can add pieces of equipment like bags, backpacks, hats and plumes, all sorts of things, on the model itself. How do you do that? Well, you must create the mesh that you wish to add, which will be cathegorized by Milkshape3D as a "group". Then, you must give this group a specific name, and then attach it to a bone so that ETW knows how to animate it.
The game will recognize any additional mesh attached to a bone that is not the main body if it has the following suffixes after the name of the group:


_body01, _body02, _body03 ...
_legs01, _legs02, _legs03 ...
_arms01, _arms02, _arms03 ...
_cuffs01, _cuffs02, _cuffs03 ...
_head01, _head02, _head03 ...
_hat01, _hat02, _hat03 ...


Again, pretty straightforward. Note that you can add anything to the groups described, and the game will still treat them like variations no matter the shape or the function of the meshes.
Please note: due to a bug in ETW's coding, the game will only recognize certain number of variations. For example, _head01 and _head02 are ok, but _head03, although recognized, will be ignored and discarded. You should test which number of variations are actually possible, my guess is that legs and arms have 3 variations, heads 2, hat only 1, and cuffs 3. Why this hasnt been fixed in years, is beyond me.


Milkshape3D's group names



At any rate, once you have your mesh attached to a skeleton, you can import it back to ETW's format. Place your mesh in "unitmodels" and the textures associated to it in "unitmodels\textures". There are 3 files required for the textures to work in game:


nameofunit_diffuse - DDS format, DXT1
nameofunit_gloss_map - DDS format, DXT3 (Photoshop + NVidia plugin only)
nameofunit_normal - DDS format, DXT5, opacity level 160


To create a normal map file, you can use CrazyBump or the Nvidia plug in for normal maps that is freely available for Photoshop. The gloss_map texture, while it appears blank in other programs, it is actually capable of defining the type of shader used for parts of the body and metallic equipment while using Photoshop with the NVidia plugin, which is the only viable method to modify it.
The normal map stores the "relief" and folds that clothes have. The greater the opacity level, the more enhanced the effect, that is why a good opacity level should not be above 160.


Once you have meshes and textures set in place, you can import the "meat" of the unit into the game.
Doing so requires the use of three files: "units_stats_land", "warscape_animated", and "warscape_animated_lod" tables.



warscape_animated_lod



warscape_animated




in "units_stats_land" you define, well, the stats of the unit, which abilities it has, if it can throw grenades and go into square formation, the spacing of the single soldiers etc. Everything has been labelled for you by PFM so no worries there.
Under the "officer", "standard_bearer", "musician" and "soldier_model" labels, you must input the desired name of your unit. Each of them can be the same model or any other specifically designed for the function assigned to it. The names of the models just mentioned have to be created inside warscape_animated and warscape_animated_lod, otherwise the game wont know what to look for. In warscape_animated you'll define the textures used by each model, to do so, you must specify "animation" under the "category" label, then, add the desired name of your model under "key" and the name of the diffuse texture - without any suffix - under "texture_filename_base". The game will automatically recognize the other textures with the same name (gloss and normal).
Once you've done that, go into warscape_animated_lod and define four LODs (levels of detail) models to assign to to your model. LODs are important for performance reasons, as every LOD is loaded by the game each time the camera gets away from the unit up to a certain distance. This is needed in every case even if you have only a single model, you must define it four times in that case, no matter what you do, or the game will crash. The numbers under "key" can be overwhelming but they are just, well, random numbers. As long as you dont input any that have been used previously, you can use whatever you want. I suggest using a logical order regardless.
Once you have added four LODs, and associated them with your model under the "animated" label (the game will search for the name you have defined under warscape_animated), you're good to go.
in case you want to create an officer, standard bearer or musician, you should also define it by adding the name of the model to the "battle_personalities" table, in which these special units are defined. Going by examples provided by the game makes things very easy here.


So there you have it. ETW model creation seems complex at first, but I assure you, it is the easiest you can possibly get.
The texture size for the diffuse is 1024 x 1024, exactly as MTW2, which can be restrictive for some small details, but very tidy and neat if you know how to make your UVs. There's more than enough space to make a decent model in one comprehensive pack.





UNIT CREATION METHOD 2 - Napoleon


For Napoleon things get fairly complicated. In an effort to make the system easier and potentially more efficient, CA created a file called "Atlas"; such a file is unique for each faction, rather than based on a single unit, and encompasses multiple variations of model parts. I said model "parts", not "entire" models. This is due to the new nature of the VPM format (.variant_part_mesh), created for NTW, which replaces the .variant_weighted_mesh. This method assumes that the modeller has created multiple parts of a model: gloves, hands, heads, hats, plumes, belts, torsos, you name it, every component of a unit, and has attached them to a skeleton ready for animation, then, separated them all into different folders. In other words, each VPM is a piece that, when attached with other VPMs, can form a complete unit; this is achieved with the help of an additional file called .unit_variant. The unit_variant file defines just the type of 3d mesh loaded and its associated texture in relation to the model part. Now, take a deep breath, relax, and try to follow this (IMHO absurdly complicated) method for assembling a unit.


Instead of the unitmodels\ folder, we need to look for the variantmodels\ folder. Inside, there are "unitparts", where meshes and associated textures reside, and "units", where atlases and .unit_variants are.

Atlases come with a diffuse, a gloss, and a normal map, just like with Empire, however, they also have a new texture called colour_map, which defines a specific colour for the clothing that is shared between multiple factions. In Empire, this was defined inside the "factions" table, where a simple RGB scheme was provided, and all one had to do was to paint the outer layer of a gloss_map texture with a specific shade of blue that would define the place where the colour had to be replaced. In NTW however that is no longer the case: there are three colours now that define multiple layers of changing colours for the uniforms, and all are defined in the "uniform_to_faction_colours_tables" table.
Once all four textures have been created - now 4096 x 4096 as maximum size -, one has to choose which portion of the whole texture will be part of a specific VPM UV. To define a specific texture inside these huge single-faction pictures, atlases come in handy. Opening one with PFM 1.5.7 reveals all sorts of coordinates - squares - and the location of each sub-texture; each of them can be accessed via the "unitparts" folder and looked at.

.unit_variants on the other hand are slightly more complicated. These come in four variations, soldier, bearer, musician and officer. This is self explanatory, they basically replace the need for warscape_lod definitions and battle_personalities definitions.
Once the game loads a "soldier" unit_variant type, for example, you could write whatever you want inside the units_stats_land soldier model field, the game will simply ignore it in favour of the unit_variant counterpart (that is why all of NTWs models inside the units_stats_land table appear to be "euroline"). However, and this is the interesting part, if you remove any of the unit_variants for a unit, the game will load that specific model from the units_stats_land table just like with ETW, thus, enabling you to freely use the old ETW method in a hybrid manner - or totally replacing NTWs method. This is what I did with Risorgimento because it is the more efficient in my opinion.
In any case, unit_variants define, again, the mesh and texture of a single part of a model, which you can customize at your leisure by simply replacing the various parts from the "unitparts" folder and their relative textures.
Keep in mind that the textures defined inside the atlas refer to the corresponding huge texture in the same folder as the atlases of the same name, NOT the ones inside "unitparts". This has never been explained, however I personally believe that it was a leftover from the unit editor that had to be part of NTW, then moved to Shogun2.


So as you can see, both methods have their drawbacks and strong points, one thing is certain though, pointing at efficiency and rapidity of execution is key for success especially with modding, as not only time, but passion and commitment are often fleeting, therefore it is important that "unprofessional" teams get to know their tools well and are able to choose the most easy and accessible paths towards their goal before even starting projects that often take years to complete with failing coordination.