ETW Runtime Memory Stucture Documention

[Hasefrexx]

If I understand how it works, I'll give it a shot. Will likely need the tuto first though.

[T.C.]

Ok, here's a complete run down of what I know about the memory structure used by ETW, followed by a tutorial on documentation.

Empire's Lua API provides a lot of functions, the most interesting of which are in the library named CampaignUI. An example function is InitialiseUnitDetails(), which returns almost all of the known details about the unit queried. The unit is queried by means of a unit pointer - a pointer to an address in memory. An example use follows:

CampaignUI.InitialiseUnitDetails(0x01f4bb388)

Note that the above code would never work, as the addresses are dynamic and change quite a lot. We can use other CampaignUI functions to find unit pointers, I'll write a tutorial on that at a later stage.

So Mitch and I began by actually looking up this address in Empire's memory using a hex editor. The name for the sector at which these addresses point shall be the "Primary Address" from here on, to avoid confusion. At the Primary Address (PA) there is a distinctive pattern in memory - 248 bytes of memory starting with the header 65 48 4A 01 and ending with 00 00 00 8C (in 99% of cases). Obviously this 248 bytes isn't near enough space to store all of the info on a unit, so from that we correctly concluded that these 248 bytes were a mixture of some unit data and more memory pointers. The unit data stored at the PA is mostly dynamic data - data which can, and does, change frequently during the course of play. So far we have identified at the PA:

• Current Unit Size
• Unit size after replenishment (ie the unit at full strength)
• Action points (has no in game effect, but verified as Action Points by means of Lua logging)
• Unit experience
• Length of unit's name string

We have also successfully identified a crucial sub-pointer in the PA. The memory at which this sub pointer points to shall from here on be referred to as the "Alpha Address". Stored at the AA are things such as unit accuracy, reloading skill, charge bonus, ammunition and flags indicating abilities (the only confirmed flag so far is "Resistant to morale shocks"). The AA contains segments of data from unit_stats_land, including what we have identified as unused columns but crucially contains no string data. It's possible that there are more sub pointers here.

So what do we need to do now? Primarily, find out what data is pointers, and where these pointers point, both in the PA and the AA. There has to be pointers here that lead us to data such as projectiles, associated 2d and 3d artwork, text based data etc.


Here is also my latest documentation:
At the PA:

Bytes 232 - 235: Pointer to the AA.

At the AA:

Byte 172: Four byte integer, Unit Accuracy
Byte 176: Four byte integer, Unit Reloading
Byte 192: Four byte integer, Unit Charge Bonus
Byte 236: Boolean, Resistant to morale shocks
Byte 236: Four byte integer, Unit Ammunition

All numbers are also little-endian.

Tutorial coming...

[terryman]

Ok, here's a complete run down of what I know about the memory structure used by ETW, followed by a tutorial on documentation.

Empire's Lua API provides a lot of functions, the most interesting of which are in the library named CampaignUI. An example function is InitialiseUnitDetails(), which returns almost all of the known details about the unit queried. The unit is queried by means of a unit pointer - a pointer to an address in memory. An example use follows:

CampaignUI.InitialiseUnitDetails(0x01f4bb388)

Note that the above code would never work, as the addresses are dynamic and change quite a lot. We can use other CampaignUI functions to find unit pointers, I'll write a tutorial on that at a later stage.

So Mitch and I began by actually looking up this address in Empire's memory using a hex editor. The name for the sector at which these addresses point shall be the "Primary Address" from here on, to avoid confusion. At the Primary Address (PA) there is a distinctive pattern in memory - 248 bytes of memory starting with the header 65 48 4A 01 and ending with 00 00 00 8C (in 99% of cases). Obviously this 248 bytes isn't near enough space to store all of the info on a unit, so from that we correctly concluded that these 248 bytes were a mixture of some unit data and more memory pointers. The unit data stored at the PA is mostly dynamic data - data which can, and does, change frequently during the course of play. So far we have identified at the PA:

• Current Unit Size
• Unit size after replenishment (ie the unit at full strength)
• Action points (has no in game effect, but verified as Action Points by means of Lua logging)
• Unit experience
• Length of unit's name string

We have also successfully identified a crucial sub-pointer in the PA. The memory at which this sub pointer points to shall from here on be referred to as the "Alpha Address". Stored at the AA are things such as unit accuracy, reloading skill, charge bonus, ammunition and flags indicating abilities (the only confirmed flag so far is "Resistant to morale shocks"). The AA contains segments of data from unit_stats_land, including what we have identified as unused columns but crucially contains no string data. It's possible that there are more sub pointers here.

So what do we need to do now? Primarily, find out what data is pointers, and where these pointers point, both in the PA and the AA. There has to be pointers here that lead us to data such as projectiles, associated 2d and 3d artwork, text based data etc.


Here is also my latest documentation:
At the PA:

Bytes 232 - 235: Pointer to the AA.

At the AA:

Byte 172: Four byte integer, Unit Accuracy
Byte 176: Four byte integer, Unit Reloading
Byte 192: Four byte integer, Unit Charge Bonus
Byte 236: Boolean, Resistant to morale shocks
Byte 236: Four byte integer, Unit Ammunition

All numbers are also little-endian.

Tutorial coming...

Did you ever write a tutorial on how to grab pointers from CampaignUI? Also is it possible to grab Alpha Address from CampaignUI, or retrieving it from WALI?

[MrClean]

Thank you TC. This is where I want to focus my modding efforts after the models portions (i'm a very moddest moddler at that). Given the information you have provided, does this mean we can only modify the information you have provided or are there more details that we have yet to unravel? I'm assuming much more, but you know what that means. Obviously I will be one of the many who will take up the torch once the tutorials have been written...

[.Mitch.]

Given the information you have provided, does this mean we can only modify the information you have provided or are there more details that we have yet to unravel? I'm assuming much more, but you know what that means.

Quite so, once we find a piece of information, for example a units size, it is ours to do with what we wish. The task right now is finding as much of this information as possible, then the sky is the limit.

[risorgimento]

Another essential data type you will encounter in memory is, following my project's naming convention, NTW::Utils::Array<T>.
It's 12 bytes long: size_t max_size, size_t size, T *data.
Most of times you'll find it in Array<T*> form (= array of object pointers), but wide-char strings (Array<wchar_t>) are also very common.

Unit's 16-bit orientations were called, at least in battles, TABANGLEs. I think to remember the conversion was just matter of multiplying by a factor (0.0054931641) and performing one type casting.

Bye.

[terryman]

Using Cheat Engine and figuring out some here.

The only values that stick after being changed after exiting the game and reloading the save are the values right off the base address, like the number of men in the unit. All other values based off secondary pointers, likd the AA or BA pointers, change back to their defaults if the game is exited and the save is reloaded. Apparently the save only keeps track of a few values, like replenishment value and unit number value, traits, history, etc... and experience.

If you change values in the secondary pointers from the primary base address, then the values will stick only if you reload the save, but you cannot exit the game. For some reason the memory stays the same and it will remember the last amounts from the last load if the game was not exited.

In otherwords, if the game is closed and reopened. ETW reloads the values from the Unit_Stats_Land table. And the values that are not offseted from the unit base address, but are offseted from some other pointer address like the Alpha or Beta, go back to their default values.

Hope that makes sense.

Anyway, how do I grab a pointer address, if I know where the pointer location is, using WALI? I'd like to return values using WALI.

[T.C.]

Documentation Tutorial

Things you'll need:

• Winhex / any decent hex editor
• Cheat Engine
• A grasp of hexadecimal numbers and the difference between data types (bools, ints, shorts, longs, strings)

Begin by creating a folder called "Logs" in your ETW data directory. Then replace the scripting.lua for the grand campaign (data/campaigns/main) with the one attached to this post. This scripting has a feature which will document the address of any unit, character or settlement you click on in the campaign map. The address will be documented in a text file in the Logs directory which you just created.

Begin by starting up Empire, preferably with no mods, and go straight into a campaign.
Then click on something and get it's address - for the moment we are focusing on units, so make sure that the log entry for "Entity" says "Pointer<UNIT>" as in the next picture (red is correct, blue is incorrect).



Note that clicking on an army always reveals the address of the unit/character in command of the army, so if you are having trouble getting a unit pointer remove any characters from the army (eg generals)

You then need to open Empire's memory in Winhex. See the picture below for the "Open RAM" button.



Once you have opened your system RAM, find Empire's process (it will have the name Empire, you'll have to scroll down a bit) and open the section called "Primary memory".

Now you need to navigate to the location in memory listed by the pointer. Do this by clicking on "Position" -> "Go To Offset", as shown in the next picture.



Make sure you select relative to beginning, not current offset.

You will then be brought to the position in memory which the pointer listed. As you will see, Primary Unit Pointers always start with the bytes 68 45 4A 01, or as seen in the interpreter "hEJ ". They end with 00 00 00 8C, if there is another Primary pointer following them (IDK what they end in if there isn't another Primary pointer following them). So now, highlight the bytes representing the Primary pointer (you should see where another PP starts by looking for "hEJ ". Then right click, select "Edit" -> "Copy Block" -> "Into New File"

You can see below a highlighted Primary unit pointer being right clicked - note the first four bytes and last four bytes.

Once you have copied the Primary pointer into a new file, you can compare it with other Primary pointers quite easily. It also makes it easy to work your way through the bytes and find the values we have identified (list kept in first post of this thread).

What you then need to do, is open Cheat Engine. Follow the tutorial presented the first time you run it, though as far as the step 3 (that's all you need that is relevant to this task).

Once you know how, you can then scan for values - use the "Memory Scan Options" to scan between the start and end of a specific Primary pointer. If you begin with an "unknown initial value scan", then proceed onto an "unchanged value" scan, you should get a good list of results. Using this method, change any values that you think represent in-game stats, and check in game to see if the change has registered. If you verify an unknown value, make a post here and let us know! If you see the OP you will also find out where the Alpha Address/Pointer is located (see my last post on the previous page for info on this). We don't know much about the data stored at the Alpha Address, but we do know some of the interesting stats (such as ammo) are stored there.

[.Mitch.]

It would be faaaar more effort than it's worth to use any of Warscapes data such as 'ground types' when implementing our own gameplay mechanics, we could much easier link it to our own set of maps or data.

[Hasefrexx]

Ok, I'll see if I can find anything of interest. Is data necessarily coded by 4 bytes? Are there values to avoid?

[Bolkonsky]

Is data necessarily coded by 4 bytes?

It depends. When changing numbers, most likely. Integers, with a few rare exceptions, are 4 bytes. There's a chance you'll run into other data types, like a float, but it's unlikely.

[.Mitch.]

Ok, I'll see if I can find anything of interest. Is data necessarily coded by 4 bytes? Are there values to avoid?

So far most of the values we've found are hled as single bytes, a byte (8 bits) can hold a value 0-255 which is usualy a value big enough for most unit stats needs.
All of the pointers (locations to more data) you will find will be 4 byte integers.

[T.C.]

It depends. When changing numbers, most likely. Integers, with a few rare exceptions, are 4 bytes. There's a chance you'll run into other data types, like a float, but it's unlikely.

Empire uses quite a lot of floats. If interested, you could check a variable's type by opening one of the DBFileTypes text files associated with PFM, or a schema for DBE and looking at the relevant variable.

So far most of the values we've found are hled as single bytes, a byte (8 bits) can hold a value 0-255 which is usualy a value big enough for most unit stats needs.
All of the pointers (locations to more data) you will find will be 4 byte integers.

While that's true, that's because the values are so small only a single byte is needed (imagine how many unit related stats are smaller than 255). I'm very confident that we're not dealing with any single byte numbers. So @ anyone looking to help with this - my advice would be to keep looking at four byte patterns.

[Bolkonsky]

Empire uses quite a lot of floats. If interested, you could check a variable's type by opening one of the DBFileTypes text files associated with PFM, or a schema for DBE and looking at the relevant variable.

For what you guys are changing you shouldn't be seeing to many. When you get into rendering and operations, yeah, but just simple numbers like unit size, will all be stored in integers.

[.Mitch.]

For what you guys are changing you shouldn't be seeing to many. When you get into rendering and operations, yeah, but just simple numbers like unit size, will all be stored in integers.

I've already said unit stats are stored as single bytes.. Try to keep up.

[T.C.]

For what you guys are changing you shouldn't be seeing to many. When you get into rendering and operations, yeah, but just simple numbers like unit size, will all be stored in integers.

Things like co-ordinates will be floats (I still haven't found them). Also some unit stats, like formation spacing, or damage values are floats. But by and large we've came upon integers.

[just]

Also all the models use lots of float (either 2 byte (half) or 4 byte (single)) values, that I verified from game memory. It is doubtful 2 byte floats would be used anywhere else in the game.

The ESF save game files are serialized objects, so there might/should be some similarity between the memory representation and the ESF.

[PietroMicca]

Also all the models use lots of float (either 2 byte (half) or 4 byte (single)) values, that I verified from game memory. It is doubtful 2 byte floats would be used anywhere else in the game.

The ESF save game files are serialized objects, so there might/should be some similarity between the memory representation and the ESF.

so these should be types used

• 00 - This is not a valid type code
• 01 - 8-bit, boolean (0 false, 1 true)
• 02 - 8-bit signed integer
• 03 - 16-bit signed integer
• 04 - 32-bit signed int
• 05 - 64-bit signed integer
• 06 - 8-bit unsigned int
• 07 - 16-bit unsigned integer
• 08 - 32-bit unsigned int
• 09 - 64-bit something 64-bit unsigned integer
• 0a - 32-bit single precision float
• 0b - 64-bit double precision float
• 0c - pair of 32-bit single precision floats (x,y coordinates)
• 0d - triple of 32-bit single precision floats (x,y,z coordinates)
• 0e - UTF-16 string (encoded directly in old ESF formats, or index to lookup table in S2TW's new format)
• 0f - ASCII string (encoded directly in old ESF formats, or index to lookup table in S2TW's new format)
• 10 - 16-bit angle (1 = 2π/65536)
• 40..4f - array types directly corresponding to 00..0f scalar types
• 80 - object with multiple fields
• 81 - array of objects of the same type

[ASSASSIN1110]

Ultimately what could be achieved with this method? give me a few examples...would this mean we could edit the campaign map?

[Primergy]

Attrition System, proper Supply Lines and something else you can think of^^