Warscape Modding Primer

[daniu]

Warscape Modding Primer

This is a short introduction into Warscape modding. This should be valid for all Warscape files, i.e. ETW, NTW, TW:S2 and TW:R2.
I'll start with some fundamental concepts of the engine; that is knowledge the modders gained soon after the release of Empire, but I've rarely seen it explained again so my guess is most people's understanding of it is at best fuzzy.

Beyond the foundations, I'll lose some words on the db files because that's about all I have any experience in.



Warscape's virtual file system

Like every piece of software, WS games are divided into the actual software - i.e. executable code - and data. If you browse to your game installation directory (somewhere along steam/steamapps/common/), you'll see Rome2.exe file; this is the executable; no modding takes place there. What you can also see is the data/ subfolder, which is where the magic happens.
Open it and you will see a number of .pack files - these contain the actual data. To open them, you'll need the PackFileManager; assuming you have downloaded it, use it to open any .pack file.
What you see on the left is what looks like a folder structure in the Windows Explorer - because that's what it is: it creates virtual subfolders of the installation's data/ directory.
Each pack contains a subset of the complete game configuration data, like movie, sound, music, and finally data; you can tell by the file name most of the time.

Additionally, each pack has a certain type: Boot, Release, Patch, Mod, Movie. This is used by the game to determine the order in which they are loaded: namely in exactly that order, first Boot, last Movie. CA's packs are of type Boot, Release, Patch and Movie.
For files that appear in two .pack files, the one that is last loaded is the one that will be used by the game. This is how CA patches their games: they replace data from the original pack by uploading one containing the same file in a Patch .pack which will overwrite the old one in the Release pack.

If you want to replace data from the vanilla game (like UI icons), you do that in exactly the same way: create a pack loaded after the Patch files containing the files overwriting the old ones.

Now, if you want to add data to the game (for instance, new units) what you can do is just create a pack containing the file you want to add, and the game will load it.
You do have to follow certain rules or the game won't know what to do with it; a good guess is to create the file in a folder within the pack that already exists. Make sure to not use the name of a file that is already contained in one of the existing packs or you will overwrite it.

Example

Browsing data_rome2.pack, you find entries in ui/units/icons/

In your mod pack, you would also create that directory structure and then put your icons there.
Say you don't like the Pretorians', you can replace the existing one by adding your png as rom_praetorians.png; if you created a new unit and want it to have a unit icon of its own, you might add it as my_little_pony.png.

Your own virtual files

So in general, to make managing your mod easier, you also want to work on a virtual file structure. PFM allows you to do that by using the File->MyMod feature.
When you select “New”, the program will ask you for a root directory of your new mod; this is where it saves your pack and stores (and retrieves from) all your working data.



To continue the example above:

Example

You create a new MyMod “Icons” in c:\Me\Mods\Icons.

You can then open data_rome2.pack (it asks you for a pack to open when creating the new pack), select the rom_praetorians.png, and then Files->Export.

You will find that your mod folder now contains the whole directory structure ui\units\icons containing the exported png you can now edit. Here, you also can now create your my_little_pony.png icons.

Once you're finished with the icons, you want to create your own pack file with File->New (or open one you already created; you have to close rome_data2.pack is the point), then select Files->Import->Add and choose your pngs which will add them to your pack you can now save.
You can always open the Windows Explorer at your Mod directory by selecting MyMod->Open Data Path.

Installing your Mod and more about pack types

To install your mod, you'll still have to copy your pack into the game's data/ directory (Game->Open Data Directory will give you a Windows Explorer there). If you followed the steps above, you will not see the new icon even if you do copy the file and start the game.

The reason for that is that when creating a new pack with PFM, it is of type “Mod”. These will not be loaded automatically by the game, but need an entry in a file called user.script.txt which is located below your user directory (depending on Windows version; you can open an Explorer there via Game->Open Script Directory). This script file is needed by the game to determine the order in which Mod packs are loaded.

To register your mod, you need to create or open “user.script.txt” and enter a line
mod MyIcons.pack;

then save it with Unicode encoding. The last point is very important or the game will not read the file! To save in Unicode, you can select the “Advanced” option in Notepad and then select the encoding.

Alternatively, you can also use My Mods->Install MyIcons which will both copy the pack and add the required entry to the script file.

Also, to play the game with a certain set of mods you can use the ModManager by Mitch which will list all Mod-type packs in your data directory and lets you select the ones you want to run; it will then create the appropriate entries.

Then, there exists the possibility to make your mod of type Movie; these mods are always loaded by the game and don't require a script file entry. I view this as somewhat as the Dark Side because it's easier and faster, but will later on create problems with data dependencies and pack file load order.

There are new pack types for Sound and Music files too, but those are new with Rome 2 and I don't have any experience with them; I'd assume they're loaded like Movie files. If someone figures it out, let me know.

A word of warning: technically, you can also edit the CA packs directly, despite the stern warning from PFM that you shouldn't.
This is a very bad idea for a reason you should be able to figure out by yourself from what I've already written.
As a hint, think about what will happen when CA next patches their game.


On DB files

As promised, here some details on the db files; for Rome 2, these can be found in data_rome2.pack.

The DB files you'll find in the CA packs like

db/<table_name>_tables/<table_name>

If you select one of those files, a table view of the contained data opens on the right side; those table contain most if not all of the configuration data for the game.
However, it is not entirely clear for all tables how they work and interact with each other; some of the modders have experience with many of them, but there are few tutorials around to explain them in detail.
If you're looking for one, it's probably a good idea to search in the modding subforums for all the warscape games because the structure stayed more or less similar.

BTW, the data is decoded from binary data contained in the pack; the fields (i.e. columns) that are displayed and their type are configured in so-called “schemas”. These configurations are user-generated and not guaranteed to be correct; if you find a mistake, let me know.
Also, these are updated from time to time to account for identified issues that were fixed. Check DB Descriptions->Search for Update from time to time; you can also choose to have PFM do this regularly upon startup in the Options menu.

For DB tables, there are additional rules to adding and overwriting data from vanilla files. Most tables contain fields considered “primary keys”, meaning that those are uniquely identifying an entry. If you add a row with the same primary key as a vanilla entry to a db table, it will overwrite the vanilla one.

Example

The table _kv_experience_bonuses in db/_kv_experience_bonuses_tables/_kv_experience_bonuses contains an entry (first_to_enter_enemy_fort=200) . If you were to add a file db/_kv_experience_bonuses_tables/_my_bonuses, containing the entry (first_to_enter_enemy_fort=500).

On a file level, it will be added side-by-side as described above; but the value of the database entry with the primary key first_to_enter_enemy_fort from the original file will be replaced by the content of the new file, i.e. 500.

The common workflow to edit DB data is to first export the files you want to edit, then create your file, then import them into your file – pretty much as described in the second example above. After you have changed the values you want, you can then use the PFM feature “Postprocess->Minimize DB Files” which will remove all entries that have not been changed from the original files. You can also export a table to a .TSV file and edit it in a Spreadsheet program like OpenOffice or MS Excel.

There also are restrictions to what values can be contained in some fields, because they are bound to data in another table. Some of those are know in the schemas, and you can make use of that by selecting the checkbox “Use Combobox Cells” when you have a db table open. This will replace the editor for text in the fields by drop-down comboboxes containing only the allowed data which prevents you from typos and gives you a better overview of the relations between tables.

Example

If you open the table agent_to_agent_attributes, you can see a column “attribute” which is a reference into the agent_attributes table's field “key”; the column “agent” references agents:key.

If you select “Use Combobox Cells”, you'll see the drop-down controls for those columns appearing.

If you right-click the header of a table with a reference, you can see the name of the referenced table in the context menu (Remove reference to <X>).

That's All Folks...

So there you have your Warscape Modding Primer, hope it gives you a bit of orientation on Warscape modding.
I'm sorry I can't answer any questions about “How do I mod X” because my knowledge of what most tables do is pretty superficial; I've stumbled over a lot in the tables but I've not tried out a lot whether they do what I suspect.

I'd love to see a lot more people who know what they're doing put up tutorials, but all of our time's limited I guess.

Have fun modding



EDIT: Here's a really good TL;DR step-by-step howto by crzyrndm:

Step 1: Add a new personal mod in PFM
File -> MyMods -> New. Enter the name and location that you want to store it.
-- When you have a mod selected under MyMods, extracted files are automatically added to the specified mod pack

Step 2: Import required files into your mod
The next step after naming and locating your mod, is to choose what it contains. Open the relevant official pack (more than likely: data_rome2.pack) and PFM will pop out a new window for you to choose the files from. Double click on the files you want to include in your mod, and hit OK when you are done.

Step 3: Make your changes

Step 4: Install your mod
PFM has a great feature where it can manage the user.script.txt entries and add your mod.pack to the Rome2 data folder for you.
File -> MyMods -> Install ____ (depending on which of your mods you have selected)