ETW Runtime Memory Stucture Documention

[.Mitch.]

This thread will be a place for myself and T.C. to document our progress on our ETW/NTW memory hook and documentation on the memory locations and structure of the running processes memory, as well as discuss its theory and implementation. If you have anything to input, please do.The thread may look bleak, but progress is actually quite astounding, theoretical implementation of what we can already do/know could mean quite astounding things. We will no doubt update everyone at a later point.

Byte locations relative to units base address

76 - Unit Size
80 - Full Unit Size
ETW:100/NTW:108 - Move Points
ETW:112 - Unit Experience
NTW:264 - Length of units custom name


ETW:232 - 235: Unit stats pointer. (Numbers at this pointer are little-endian)

+172: Four byte integer, Unit Accuracy

+176: Four byte integer, Unit Reloading

+192: Four byte integer, Unit Charge Bonus

+236: Boolean, Resistant to morale shocks

+236: Four byte integer, Unit Ammunition

BETA POINTER:

I've confirmed another pointer at the Primary Address. This one seems to relate to units_tables, and we've named it Beta pointer for now. It's structure is a little harder to understand than the Alpha pointer (unit_stats_land). The Beta pointer begins at byte 72 in the primary pointer. As usual, four byte little-endian.

There doesn't seem to be a header in these files, which is understandable as there is no difference between a naval entry and a land entry in units_tables.

From the beginning of the memory at the Beta Address:

Byte 36 - Four byte int, recruit time
Byte 40 - Four byte int, unit upkeep cost
Byte 72 - four byte int, unit limit

There's also a lot more pointers here, probably to the string based data you see in units_tables in PFM.

Want to help?

Spoiler Alert, click show to read: 

Documentation Tutorial

Things you'll need:

• Winhex / any decent hex editor
• Cheat Engine
• A grasp of hexadecimal numbers and the difference between data types (bools, ints, shorts, longs, strings)

Begin by creating a folder called "Logs" in your ETW data directory. Then replace the scripting.lua for the grand campaign (data/campaigns/main) with the one attached to this post. This scripting has a feature which will document the address of any unit, character or settlement you click on in the campaign map. The address will be documented in a text file in the Logs directory which you just created.

Begin by starting up Empire, preferably with no mods, and go straight into a campaign.
Then click on something and get it's address - for the moment we are focusing on units, so make sure that the log entry for "Entity" says "Pointer<UNIT>" as in the next picture (red is correct, blue is incorrect).



Note that clicking on an army always reveals the address of the unit/character in command of the army, so if you are having trouble getting a unit pointer remove any characters from the army (eg generals)

You then need to open Empire's memory in Winhex. See the picture below for the "Open RAM" button.



Once you have opened your system RAM, find Empire's process (it will have the name Empire, you'll have to scroll down a bit) and open the section called "Primary memory".

Now you need to navigate to the location in memory listed by the pointer. Do this by clicking on "Position" -> "Go To Offset", as shown in the next picture.



Make sure you select relative to beginning, not current offset.

You will then be brought to the position in memory which the pointer listed. As you will see, Primary Unit Pointers always start with the bytes 68 45 4A 01, or as seen in the interpreter "hEJ ". They end with 00 00 00 8C, if there is another Primary pointer following them (IDK what they end in if there isn't another Primary pointer following them). So now, highlight the bytes representing the Primary pointer (you should see where another PP starts by looking for "hEJ ". Then right click, select "Edit" -> "Copy Block" -> "Into New File"

You can see below a highlighted Primary unit pointer being right clicked - note the first four bytes and last four bytes.

Once you have copied the Primary pointer into a new file, you can compare it with other Primary pointers quite easily. It also makes it easy to work your way through the bytes and find the values we have identified (list kept in first post of this thread).

What you then need to do, is open Cheat Engine. Follow the tutorial presented the first time you run it, though as far as the step 3 (that's all you need that is relevant to this task).

Once you know how, you can then scan for values - use the "Memory Scan Options" to scan between the start and end of a specific Primary pointer. If you begin with an "unknown initial value scan", then proceed onto an "unchanged value" scan, you should get a good list of results. Using this method, change any values that you think represent in-game stats, and check in game to see if the change has registered. If you verify an unknown value, make a post here and let us know! If you see the OP you will also find out where the Alpha Address/Pointer is located (see my last post on the previous page for info on this). We don't know much about the data stored at the Alpha Address, but we do know some of the interesting stats (such as ammo) are stored there.

[T.C.]

Reserved.

[T.C.]

Byte 112 confirmed as unit experience.

If anyone is interested in helping us with this I can write up a tutorial. It's quite interesting and rewarding, and very simple to do.

[Bolkonsky]

What are you doing, using a memory scanner, searching for the number, changing it, and searching the first search's result to see what changed?

[.Mitch.]

Not really.

Topcats LUA outputs the base address of a units location in memory. For example: 0x040ffbef0. So memory scanning isn't necessary.

We then read the following bytes after the base address to give us the position of certain attributes of the unit, for example 76 bytes after a units base address is the units size.

I've been using my own program that reads the following 1000 bytes after a base address and cross references similarities to the unit. Topcat has been using cheatengine to read the memory in hex and more tediously comparing them manually.

Now I know that 76 bytes after a base address is a units size, I can do things like match a units location on the campaign map to a colourmap of europe or w/e and based on its location (if its in snow/desert) we can effect the unit. So basically from what we know it would be extremely easy to implement NTW's attrition feature into ETW.

That's just one example of a million possibilities. We can do exactly this for cities, etc, etc.

And this is two days into things... lol

[Bolkonsky]

Yeah I'm a huge fan of this stuff, though I've only ever made simple trainers (infinite lives, etc.) I'll be following your progress, this is some nice stuff. The real challenge is going to be creating functions that modders can reference.

[T.C.]

Yeah I'm a huge fan of this stuff, though I've only ever made simple trainers (infinite lives, etc.) I'll be following your progress, this is some nice stuff. The real challenge is going to be creating functions that modders can reference.

That's going to be easy on our behalf; the problem for other modders is acquiring the memory pointers, which takes a bit of scripting knowledge to do. In fact I'd say I'm one of very few poeple who knows how to access these pointers, at least I haven't seen of any other active modders utilising them for scripting. I'll have to get writing tutorials again

If you want to help out with this Bolk we can get you off the ground quite easily.

[.Mitch.]

The real challenge is going to be creating functions that modders can reference.

I've already got the concept in my head of how we'll make this available to others.

The main application of this will be implementation of extra game mechanics and so without sounding greedy anything 'amazing' will likely be done natively in the executable (sort of like a second game engine) instead of via any secondary functionality.

I don't know if LUA is capable of working with an executable (infact I know it is.. I just have no experience of LUA implementation into C/C++/C#), if I do then I'll likely make an API of sorts to interface with it and call the functions available to the executable. Via the scripting.lua of the game you can do literally just about anything you want, you just can't make any effects into the game. So for instance you can find out alot of information, make a million if statements have a really l33t system in place, but you cant make it effect the campaign. That was our whole idea here really, to make the possibilites of what LUA can do actually do something.

I know TC has programming experience so I'll likely just make my source code available to him at some point so he can implement his own functionality without having to bother with the shite I've gone through.

Nomatter what though people will need LUA experience (at the very least) to make use of this, I'll likely make TC write some LUA functions that output text command lines to a file, my program will read the file and take the commands from the stack and implement them. Sort of a very very crap API.

If I implement a modding API into warscape can I haz opifex pl0x?

[General Brewster]

Well T.C if it's easy to understand and learn I might be able to help you out.

[Magicman2051]

If I implement a modding API into warscape can I haz opifex pl0x?

Based on this work alone you're both in a prime position.

[T.C.]

Ok so a little theory, most of which will probably only be relevant to Mitch and I until I write more background tutorials.

Note that I'm not used to working with data structures this low level, apologies for any incorrect terminology.

The memory chunks which unit pointers point at have the following format:

First four bytes [0 - 3] are some sort of header; they seem to take the format 68 45 4A 01.

The next four bytes is [4 - 7] are an index of sorts - possibly linked into the "unitXX" ID that is assigned to unit card components, though I doubt it.

Next four unknown, however the next four after that [12 - 15] seem to always take the form D8 45 4A 01 (notice the bytes 45 4A 01, they repeat frequently throughout the memory sector)

Apart from that we know very little, except that each unit pointer ends in 00 00 00 8C when there is another pointer following it.

More to come.

[T.C.]

Well T.C if it's easy to understand and learn I might be able to help you out.

Any experience with binary? makes no difference either way

[T.C.]

I've just confirmed the location in memory of unit ammo, and a chunk of other static stats. Will write documentation up on this later today.

In other terms, we will soon have supply lines that affect unit's ammo on the campaign

[risorgimento]

Hi guys.

In the past I tracked down a few NTW's structure layouts in memory and their pointer offsets, especially concerning battle and ai.

A couple simple examples:

       /* Alliance structure (?? bytes). */
        struct Alliance
        {
            struct {
                char _Pad00[8];
                BattleModel *m_BattleModel;
                char _Pad0C[4];
            } *m_Unk00;
            DWORD m_Unk04;
            DWORD m_Unk08;
            DWORD m_AIPlayer;
            int m_ID;
            ArmyArray m_ArmyList; // 0x1C (-8)
            char _Pad28[16];
            LandUnitArray m_VisibleEnemyUnitList;
            char _Pad4C[52];
        };

        /* Army structure (0x24C bytes). */
        // Army +0x128/0x12C is a m_UnitGroupList
        struct Army
        {
            DWORD m_VTable;
            int m_Id;
            DWORD m_Cache;
            char _Pad0C[204];
            Alliance *m_Alliance; // 0xD8
            char _PadDC[8];
            LandUnitArray m_UnitList; // 0xEC (-8)
            char _PadF8[340];
        };

But I think we need a DLL code-injector first to make any decent usage of them.
I'm waiting for a Craig's statement on this "burning issue" because it's legally very borderline (although ethically more questionable, as it's just memory patching and no copyrighted material is shared). Crossing fingers.
Bye.

[General Brewster]

Any experience with binary? makes no difference either way

Nope total noob

[T.C.]

Hi guys.

In the past I tracked down a few NTW's structure layouts in memory and their pointer offsets, especially concerning battle and ai.

A couple simple examples:

       /* Alliance structure (?? bytes). */
        struct Alliance
        {
            struct {
                char _Pad00[8];
                BattleModel *m_BattleModel;
                char _Pad0C[4];
            } *m_Unk00;
            DWORD m_Unk04;
            DWORD m_Unk08;
            DWORD m_AIPlayer;
            int m_ID;
            ArmyArray m_ArmyList; // 0x1C (-8)
            char _Pad28[16];
            LandUnitArray m_VisibleEnemyUnitList;
            char _Pad4C[52];
        };

        /* Army structure (0x24C bytes). */
        // Army +0x128/0x12C is a m_UnitGroupList
        struct Army
        {
            DWORD m_VTable;
            int m_Id;
            DWORD m_Cache;
            char _Pad0C[204];
            Alliance *m_Alliance; // 0xD8
            char _PadDC[8];
            LandUnitArray m_UnitList; // 0xEC (-8)
            char _PadF8[340];
        };

But I think we need a DLL code-injector first to make any decent usage of them.
I'm waiting for a Craig's statement on this "burning issue" because it's legally very borderline (although ethically more questionable, as it's just memory patching and no copyrighted material is shared). Crossing fingers.
Bye.

Nice info! How did you come across this?

Nope total noob

Better wait till I write a tutorial on the subject then. In the mean time acquaint yourself with Hexadecimal numbers, data types (ints, bools, floats, shorts, strings) and the difference between bits and bytes.

[General Brewster]

Sure.

[Hasefrexx]

If I understand how it works, I'll give it a shot. Will likely need the tuto first though.

[T.C.]

Ok, here's a complete run down of what I know about the memory structure used by ETW, followed by a tutorial on documentation.

Empire's Lua API provides a lot of functions, the most interesting of which are in the library named CampaignUI. An example function is InitialiseUnitDetails(), which returns almost all of the known details about the unit queried. The unit is queried by means of a unit pointer - a pointer to an address in memory. An example use follows:

CampaignUI.InitialiseUnitDetails(0x01f4bb388)

Note that the above code would never work, as the addresses are dynamic and change quite a lot. We can use other CampaignUI functions to find unit pointers, I'll write a tutorial on that at a later stage.

So Mitch and I began by actually looking up this address in Empire's memory using a hex editor. The name for the sector at which these addresses point shall be the "Primary Address" from here on, to avoid confusion. At the Primary Address (PA) there is a distinctive pattern in memory - 248 bytes of memory starting with the header 65 48 4A 01 and ending with 00 00 00 8C (in 99% of cases). Obviously this 248 bytes isn't near enough space to store all of the info on a unit, so from that we correctly concluded that these 248 bytes were a mixture of some unit data and more memory pointers. The unit data stored at the PA is mostly dynamic data - data which can, and does, change frequently during the course of play. So far we have identified at the PA:

• Current Unit Size
• Unit size after replenishment (ie the unit at full strength)
• Action points (has no in game effect, but verified as Action Points by means of Lua logging)
• Unit experience
• Length of unit's name string

We have also successfully identified a crucial sub-pointer in the PA. The memory at which this sub pointer points to shall from here on be referred to as the "Alpha Address". Stored at the AA are things such as unit accuracy, reloading skill, charge bonus, ammunition and flags indicating abilities (the only confirmed flag so far is "Resistant to morale shocks"). The AA contains segments of data from unit_stats_land, including what we have identified as unused columns but crucially contains no string data. It's possible that there are more sub pointers here.

So what do we need to do now? Primarily, find out what data is pointers, and where these pointers point, both in the PA and the AA. There has to be pointers here that lead us to data such as projectiles, associated 2d and 3d artwork, text based data etc.


Here is also my latest documentation:
At the PA:

Bytes 232 - 235: Pointer to the AA.

At the AA:

Byte 172: Four byte integer, Unit Accuracy
Byte 176: Four byte integer, Unit Reloading
Byte 192: Four byte integer, Unit Charge Bonus
Byte 236: Boolean, Resistant to morale shocks
Byte 236: Four byte integer, Unit Ammunition

All numbers are also little-endian.

Tutorial coming...

[MrClean]

Thank you TC. This is where I want to focus my modding efforts after the models portions (i'm a very moddest moddler at that). Given the information you have provided, does this mean we can only modify the information you have provided or are there more details that we have yet to unravel? I'm assuming much more, but you know what that means. Obviously I will be one of the many who will take up the torch once the tutorials have been written...