ETW Runtime Memory Stucture Documention

[.Mitch.]

This thread will be a place for myself and T.C. to document our progress on our ETW/NTW memory hook and documentation on the memory locations and structure of the running processes memory, as well as discuss its theory and implementation. If you have anything to input, please do.The thread may look bleak, but progress is actually quite astounding, theoretical implementation of what we can already do/know could mean quite astounding things. We will no doubt update everyone at a later point.

Byte locations relative to units base address

76 - Unit Size
80 - Full Unit Size
ETW:100/NTW:108 - Move Points
ETW:112 - Unit Experience
NTW:264 - Length of units custom name


ETW:232 - 235: Unit stats pointer. (Numbers at this pointer are little-endian)

+172: Four byte integer, Unit Accuracy

+176: Four byte integer, Unit Reloading

+192: Four byte integer, Unit Charge Bonus

+236: Boolean, Resistant to morale shocks

+236: Four byte integer, Unit Ammunition

BETA POINTER:

I've confirmed another pointer at the Primary Address. This one seems to relate to units_tables, and we've named it Beta pointer for now. It's structure is a little harder to understand than the Alpha pointer (unit_stats_land). The Beta pointer begins at byte 72 in the primary pointer. As usual, four byte little-endian.

There doesn't seem to be a header in these files, which is understandable as there is no difference between a naval entry and a land entry in units_tables.

From the beginning of the memory at the Beta Address:

Byte 36 - Four byte int, recruit time
Byte 40 - Four byte int, unit upkeep cost
Byte 72 - four byte int, unit limit

There's also a lot more pointers here, probably to the string based data you see in units_tables in PFM.

Want to help?

Spoiler Alert, click show to read: 

Documentation Tutorial

Things you'll need:

• Winhex / any decent hex editor
• Cheat Engine
• A grasp of hexadecimal numbers and the difference between data types (bools, ints, shorts, longs, strings)

Begin by creating a folder called "Logs" in your ETW data directory. Then replace the scripting.lua for the grand campaign (data/campaigns/main) with the one attached to this post. This scripting has a feature which will document the address of any unit, character or settlement you click on in the campaign map. The address will be documented in a text file in the Logs directory which you just created.

Begin by starting up Empire, preferably with no mods, and go straight into a campaign.
Then click on something and get it's address - for the moment we are focusing on units, so make sure that the log entry for "Entity" says "Pointer<UNIT>" as in the next picture (red is correct, blue is incorrect).



Note that clicking on an army always reveals the address of the unit/character in command of the army, so if you are having trouble getting a unit pointer remove any characters from the army (eg generals)

You then need to open Empire's memory in Winhex. See the picture below for the "Open RAM" button.



Once you have opened your system RAM, find Empire's process (it will have the name Empire, you'll have to scroll down a bit) and open the section called "Primary memory".

Now you need to navigate to the location in memory listed by the pointer. Do this by clicking on "Position" -> "Go To Offset", as shown in the next picture.



Make sure you select relative to beginning, not current offset.

You will then be brought to the position in memory which the pointer listed. As you will see, Primary Unit Pointers always start with the bytes 68 45 4A 01, or as seen in the interpreter "hEJ ". They end with 00 00 00 8C, if there is another Primary pointer following them (IDK what they end in if there isn't another Primary pointer following them). So now, highlight the bytes representing the Primary pointer (you should see where another PP starts by looking for "hEJ ". Then right click, select "Edit" -> "Copy Block" -> "Into New File"

You can see below a highlighted Primary unit pointer being right clicked - note the first four bytes and last four bytes.

Once you have copied the Primary pointer into a new file, you can compare it with other Primary pointers quite easily. It also makes it easy to work your way through the bytes and find the values we have identified (list kept in first post of this thread).

What you then need to do, is open Cheat Engine. Follow the tutorial presented the first time you run it, though as far as the step 3 (that's all you need that is relevant to this task).

Once you know how, you can then scan for values - use the "Memory Scan Options" to scan between the start and end of a specific Primary pointer. If you begin with an "unknown initial value scan", then proceed onto an "unchanged value" scan, you should get a good list of results. Using this method, change any values that you think represent in-game stats, and check in game to see if the change has registered. If you verify an unknown value, make a post here and let us know! If you see the OP you will also find out where the Alpha Address/Pointer is located (see my last post on the previous page for info on this). We don't know much about the data stored at the Alpha Address, but we do know some of the interesting stats (such as ammo) are stored there.

[T.C.]

Reserved.

[T.C.]

Byte 112 confirmed as unit experience.

If anyone is interested in helping us with this I can write up a tutorial. It's quite interesting and rewarding, and very simple to do.

[Bolkonsky]

What are you doing, using a memory scanner, searching for the number, changing it, and searching the first search's result to see what changed?

[.Mitch.]

Not really.

Topcats LUA outputs the base address of a units location in memory. For example: 0x040ffbef0. So memory scanning isn't necessary.

We then read the following bytes after the base address to give us the position of certain attributes of the unit, for example 76 bytes after a units base address is the units size.

I've been using my own program that reads the following 1000 bytes after a base address and cross references similarities to the unit. Topcat has been using cheatengine to read the memory in hex and more tediously comparing them manually.

Now I know that 76 bytes after a base address is a units size, I can do things like match a units location on the campaign map to a colourmap of europe or w/e and based on its location (if its in snow/desert) we can effect the unit. So basically from what we know it would be extremely easy to implement NTW's attrition feature into ETW.

That's just one example of a million possibilities. We can do exactly this for cities, etc, etc.

And this is two days into things... lol

[PietroMicca]

Not really.

Topcats LUA outputs the base address of a units location in memory. For example: 0x040ffbef0. So memory scanning isn't necessary.

We then read the following bytes after the base address to give us the position of certain attributes of the unit, for example 76 bytes after a units base address is the units size.

I've been using my own program that reads the following 1000 bytes after a base address and cross references similarities to the unit. Topcat has been using cheatengine to read the memory in hex and more tediously comparing them manually.

Now I know that 76 bytes after a base address is a units size, I can do things like match a units location on the campaign map to a colourmap of europe or w/e and based on its location (if its in snow/desert) we can effect the unit. So basically from what we know it would be extremely easy to implement NTW's attrition feature into ETW.

That's just one example of a million possibilities. We can do exactly this for cities, etc, etc.

And this is two days into things... lol

Is it really possible?

[T.C.]

Is it really possible?

Given what we've achieved so far I can confidently say it's possible to design an attrition system like Napoleon's, and also a replenishment system like Napoleons. Expect both of them in the coming months

[PietroMicca]

Given what we've achieved so far I can confidently say it's possible to design an attrition system like Napoleon's, and also a replenishment system like Napoleons. Expect both of them in the coming months

It would be great. In my dream campaign map there's a supply chain. But I also think that this system should be linked to groundtypes in campaign map. So our researches should go in parallel.

[T.C.]

It would be great. In my dream campaign map there's a supply chain. But I also think that this system should be linked to groundtypes in campaign map. So our researches should go in parallel.

What's your ideas relating to ground types?

I've confirmed another pointer at the Primary Address. This one seems to relate to units_tables, and we've named it Beta pointer for now. It's structure is a little harder to understand than the Alpha pointer (unit_stats_land). The Beta pointer begins at byte 72 in the primary pointer. As usual, four byte little-endian.

There doesn't seem to be a header in these files, which is understandable as there is no difference between a naval entry and a land entry in units_tables.

From the beginning of the memory at the Beta Address:

Byte 36 - Four byte int, recruit time
Byte 40 - Four byte int, unit upkeep cost
Byte 72 - four byte int, unit limit

There's also a lot more pointers here, probably to the string based data you see in units_tables in PFM.

[Bolkonsky]

Yeah I'm a huge fan of this stuff, though I've only ever made simple trainers (infinite lives, etc.) I'll be following your progress, this is some nice stuff. The real challenge is going to be creating functions that modders can reference.

[T.C.]

Yeah I'm a huge fan of this stuff, though I've only ever made simple trainers (infinite lives, etc.) I'll be following your progress, this is some nice stuff. The real challenge is going to be creating functions that modders can reference.

That's going to be easy on our behalf; the problem for other modders is acquiring the memory pointers, which takes a bit of scripting knowledge to do. In fact I'd say I'm one of very few poeple who knows how to access these pointers, at least I haven't seen of any other active modders utilising them for scripting. I'll have to get writing tutorials again

If you want to help out with this Bolk we can get you off the ground quite easily.

[.Mitch.]

The real challenge is going to be creating functions that modders can reference.

I've already got the concept in my head of how we'll make this available to others.

The main application of this will be implementation of extra game mechanics and so without sounding greedy anything 'amazing' will likely be done natively in the executable (sort of like a second game engine) instead of via any secondary functionality.

I don't know if LUA is capable of working with an executable (infact I know it is.. I just have no experience of LUA implementation into C/C++/C#), if I do then I'll likely make an API of sorts to interface with it and call the functions available to the executable. Via the scripting.lua of the game you can do literally just about anything you want, you just can't make any effects into the game. So for instance you can find out alot of information, make a million if statements have a really l33t system in place, but you cant make it effect the campaign. That was our whole idea here really, to make the possibilites of what LUA can do actually do something.

I know TC has programming experience so I'll likely just make my source code available to him at some point so he can implement his own functionality without having to bother with the shite I've gone through.

Nomatter what though people will need LUA experience (at the very least) to make use of this, I'll likely make TC write some LUA functions that output text command lines to a file, my program will read the file and take the commands from the stack and implement them. Sort of a very very crap API.

If I implement a modding API into warscape can I haz opifex pl0x?

[Magicman2051]

If I implement a modding API into warscape can I haz opifex pl0x?

Based on this work alone you're both in a prime position.

[T.C.]

Well T.C if it's easy to understand and learn I might be able to help you out.

Any experience with binary? makes no difference either way

[General Brewster]

Any experience with binary? makes no difference either way

Nope total noob

[General Brewster]

Well T.C if it's easy to understand and learn I might be able to help you out.

[T.C.]

Ok so a little theory, most of which will probably only be relevant to Mitch and I until I write more background tutorials.

Note that I'm not used to working with data structures this low level, apologies for any incorrect terminology.

The memory chunks which unit pointers point at have the following format:

First four bytes [0 - 3] are some sort of header; they seem to take the format 68 45 4A 01.

The next four bytes is [4 - 7] are an index of sorts - possibly linked into the "unitXX" ID that is assigned to unit card components, though I doubt it.

Next four unknown, however the next four after that [12 - 15] seem to always take the form D8 45 4A 01 (notice the bytes 45 4A 01, they repeat frequently throughout the memory sector)

Apart from that we know very little, except that each unit pointer ends in 00 00 00 8C when there is another pointer following it.

More to come.

[T.C.]

I've just confirmed the location in memory of unit ammo, and a chunk of other static stats. Will write documentation up on this later today.

In other terms, we will soon have supply lines that affect unit's ammo on the campaign

[risorgimento]

Hi guys.

In the past I tracked down a few NTW's structure layouts in memory and their pointer offsets, especially concerning battle and ai.

A couple simple examples:

       /* Alliance structure (?? bytes). */
        struct Alliance
        {
            struct {
                char _Pad00[8];
                BattleModel *m_BattleModel;
                char _Pad0C[4];
            } *m_Unk00;
            DWORD m_Unk04;
            DWORD m_Unk08;
            DWORD m_AIPlayer;
            int m_ID;
            ArmyArray m_ArmyList; // 0x1C (-8)
            char _Pad28[16];
            LandUnitArray m_VisibleEnemyUnitList;
            char _Pad4C[52];
        };

        /* Army structure (0x24C bytes). */
        // Army +0x128/0x12C is a m_UnitGroupList
        struct Army
        {
            DWORD m_VTable;
            int m_Id;
            DWORD m_Cache;
            char _Pad0C[204];
            Alliance *m_Alliance; // 0xD8
            char _PadDC[8];
            LandUnitArray m_UnitList; // 0xEC (-8)
            char _PadF8[340];
        };

But I think we need a DLL code-injector first to make any decent usage of them.
I'm waiting for a Craig's statement on this "burning issue" because it's legally very borderline (although ethically more questionable, as it's just memory patching and no copyrighted material is shared). Crossing fingers.
Bye.

[General Brewster]

Sure.