ETW Runtime Memory Stucture Documention

[.Mitch.]

Given the information you have provided, does this mean we can only modify the information you have provided or are there more details that we have yet to unravel? I'm assuming much more, but you know what that means.

Quite so, once we find a piece of information, for example a units size, it is ours to do with what we wish. The task right now is finding as much of this information as possible, then the sky is the limit.

[T.C.]

Documentation Tutorial

Things you'll need:

• Winhex / any decent hex editor
• Cheat Engine
• A grasp of hexadecimal numbers and the difference between data types (bools, ints, shorts, longs, strings)

Begin by creating a folder called "Logs" in your ETW data directory. Then replace the scripting.lua for the grand campaign (data/campaigns/main) with the one attached to this post. This scripting has a feature which will document the address of any unit, character or settlement you click on in the campaign map. The address will be documented in a text file in the Logs directory which you just created.

Begin by starting up Empire, preferably with no mods, and go straight into a campaign.
Then click on something and get it's address - for the moment we are focusing on units, so make sure that the log entry for "Entity" says "Pointer<UNIT>" as in the next picture (red is correct, blue is incorrect).



Note that clicking on an army always reveals the address of the unit/character in command of the army, so if you are having trouble getting a unit pointer remove any characters from the army (eg generals)

You then need to open Empire's memory in Winhex. See the picture below for the "Open RAM" button.



Once you have opened your system RAM, find Empire's process (it will have the name Empire, you'll have to scroll down a bit) and open the section called "Primary memory".

Now you need to navigate to the location in memory listed by the pointer. Do this by clicking on "Position" -> "Go To Offset", as shown in the next picture.



Make sure you select relative to beginning, not current offset.

You will then be brought to the position in memory which the pointer listed. As you will see, Primary Unit Pointers always start with the bytes 68 45 4A 01, or as seen in the interpreter "hEJ ". They end with 00 00 00 8C, if there is another Primary pointer following them (IDK what they end in if there isn't another Primary pointer following them). So now, highlight the bytes representing the Primary pointer (you should see where another PP starts by looking for "hEJ ". Then right click, select "Edit" -> "Copy Block" -> "Into New File"

You can see below a highlighted Primary unit pointer being right clicked - note the first four bytes and last four bytes.

Once you have copied the Primary pointer into a new file, you can compare it with other Primary pointers quite easily. It also makes it easy to work your way through the bytes and find the values we have identified (list kept in first post of this thread).

What you then need to do, is open Cheat Engine. Follow the tutorial presented the first time you run it, though as far as the step 3 (that's all you need that is relevant to this task).

Once you know how, you can then scan for values - use the "Memory Scan Options" to scan between the start and end of a specific Primary pointer. If you begin with an "unknown initial value scan", then proceed onto an "unchanged value" scan, you should get a good list of results. Using this method, change any values that you think represent in-game stats, and check in game to see if the change has registered. If you verify an unknown value, make a post here and let us know! If you see the OP you will also find out where the Alpha Address/Pointer is located (see my last post on the previous page for info on this). We don't know much about the data stored at the Alpha Address, but we do know some of the interesting stats (such as ammo) are stored there.

[PietroMicca]

Not really.

Topcats LUA outputs the base address of a units location in memory. For example: 0x040ffbef0. So memory scanning isn't necessary.

We then read the following bytes after the base address to give us the position of certain attributes of the unit, for example 76 bytes after a units base address is the units size.

I've been using my own program that reads the following 1000 bytes after a base address and cross references similarities to the unit. Topcat has been using cheatengine to read the memory in hex and more tediously comparing them manually.

Now I know that 76 bytes after a base address is a units size, I can do things like match a units location on the campaign map to a colourmap of europe or w/e and based on its location (if its in snow/desert) we can effect the unit. So basically from what we know it would be extremely easy to implement NTW's attrition feature into ETW.

That's just one example of a million possibilities. We can do exactly this for cities, etc, etc.

And this is two days into things... lol

Is it really possible?

[T.C.]

Is it really possible?

Given what we've achieved so far I can confidently say it's possible to design an attrition system like Napoleon's, and also a replenishment system like Napoleons. Expect both of them in the coming months

[PietroMicca]

Given what we've achieved so far I can confidently say it's possible to design an attrition system like Napoleon's, and also a replenishment system like Napoleons. Expect both of them in the coming months

It would be great. In my dream campaign map there's a supply chain. But I also think that this system should be linked to groundtypes in campaign map. So our researches should go in parallel.

[T.C.]

It would be great. In my dream campaign map there's a supply chain. But I also think that this system should be linked to groundtypes in campaign map. So our researches should go in parallel.

What's your ideas relating to ground types?

I've confirmed another pointer at the Primary Address. This one seems to relate to units_tables, and we've named it Beta pointer for now. It's structure is a little harder to understand than the Alpha pointer (unit_stats_land). The Beta pointer begins at byte 72 in the primary pointer. As usual, four byte little-endian.

There doesn't seem to be a header in these files, which is understandable as there is no difference between a naval entry and a land entry in units_tables.

From the beginning of the memory at the Beta Address:

Byte 36 - Four byte int, recruit time
Byte 40 - Four byte int, unit upkeep cost
Byte 72 - four byte int, unit limit

There's also a lot more pointers here, probably to the string based data you see in units_tables in PFM.

[.Mitch.]

It would be faaaar more effort than it's worth to use any of Warscapes data such as 'ground types' when implementing our own gameplay mechanics, we could much easier link it to our own set of maps or data.

[Hasefrexx]

Ok, I'll see if I can find anything of interest. Is data necessarily coded by 4 bytes? Are there values to avoid?

[Bolkonsky]

Is data necessarily coded by 4 bytes?

It depends. When changing numbers, most likely. Integers, with a few rare exceptions, are 4 bytes. There's a chance you'll run into other data types, like a float, but it's unlikely.

[.Mitch.]

Ok, I'll see if I can find anything of interest. Is data necessarily coded by 4 bytes? Are there values to avoid?

So far most of the values we've found are hled as single bytes, a byte (8 bits) can hold a value 0-255 which is usualy a value big enough for most unit stats needs.
All of the pointers (locations to more data) you will find will be 4 byte integers.

[T.C.]

It depends. When changing numbers, most likely. Integers, with a few rare exceptions, are 4 bytes. There's a chance you'll run into other data types, like a float, but it's unlikely.

Empire uses quite a lot of floats. If interested, you could check a variable's type by opening one of the DBFileTypes text files associated with PFM, or a schema for DBE and looking at the relevant variable.

So far most of the values we've found are hled as single bytes, a byte (8 bits) can hold a value 0-255 which is usualy a value big enough for most unit stats needs.
All of the pointers (locations to more data) you will find will be 4 byte integers.

While that's true, that's because the values are so small only a single byte is needed (imagine how many unit related stats are smaller than 255). I'm very confident that we're not dealing with any single byte numbers. So @ anyone looking to help with this - my advice would be to keep looking at four byte patterns.

[Bolkonsky]

Empire uses quite a lot of floats. If interested, you could check a variable's type by opening one of the DBFileTypes text files associated with PFM, or a schema for DBE and looking at the relevant variable.

For what you guys are changing you shouldn't be seeing to many. When you get into rendering and operations, yeah, but just simple numbers like unit size, will all be stored in integers.

[.Mitch.]

For what you guys are changing you shouldn't be seeing to many. When you get into rendering and operations, yeah, but just simple numbers like unit size, will all be stored in integers.

I've already said unit stats are stored as single bytes.. Try to keep up.

[T.C.]

For what you guys are changing you shouldn't be seeing to many. When you get into rendering and operations, yeah, but just simple numbers like unit size, will all be stored in integers.

Things like co-ordinates will be floats (I still haven't found them). Also some unit stats, like formation spacing, or damage values are floats. But by and large we've came upon integers.

[just]

Also all the models use lots of float (either 2 byte (half) or 4 byte (single)) values, that I verified from game memory. It is doubtful 2 byte floats would be used anywhere else in the game.

The ESF save game files are serialized objects, so there might/should be some similarity between the memory representation and the ESF.

[PietroMicca]

Also all the models use lots of float (either 2 byte (half) or 4 byte (single)) values, that I verified from game memory. It is doubtful 2 byte floats would be used anywhere else in the game.

The ESF save game files are serialized objects, so there might/should be some similarity between the memory representation and the ESF.

so these should be types used

• 00 - This is not a valid type code
• 01 - 8-bit, boolean (0 false, 1 true)
• 02 - 8-bit signed integer
• 03 - 16-bit signed integer
• 04 - 32-bit signed int
• 05 - 64-bit signed integer
• 06 - 8-bit unsigned int
• 07 - 16-bit unsigned integer
• 08 - 32-bit unsigned int
• 09 - 64-bit something 64-bit unsigned integer
• 0a - 32-bit single precision float
• 0b - 64-bit double precision float
• 0c - pair of 32-bit single precision floats (x,y coordinates)
• 0d - triple of 32-bit single precision floats (x,y,z coordinates)
• 0e - UTF-16 string (encoded directly in old ESF formats, or index to lookup table in S2TW's new format)
• 0f - ASCII string (encoded directly in old ESF formats, or index to lookup table in S2TW's new format)
• 10 - 16-bit angle (1 = 2π/65536)
• 40..4f - array types directly corresponding to 00..0f scalar types
• 80 - object with multiple fields
• 81 - array of objects of the same type

[risorgimento]

Another essential data type you will encounter in memory is, following my project's naming convention, NTW::Utils::Array<T>.
It's 12 bytes long: size_t max_size, size_t size, T *data.
Most of times you'll find it in Array<T*> form (= array of object pointers), but wide-char strings (Array<wchar_t>) are also very common.

Unit's 16-bit orientations were called, at least in battles, TABANGLEs. I think to remember the conversion was just matter of multiplying by a factor (0.0054931641) and performing one type casting.

Bye.

[ASSASSIN1110]

Ultimately what could be achieved with this method? give me a few examples...would this mean we could edit the campaign map?

[Primergy]

Attrition System, proper Supply Lines and something else you can think of^^

[T.C.]

Attrition System, proper Supply Lines and something else you can think of^^

Attrition is all that's guaranteed so far. And it's going to take a long time to design, any concept I've came up with so far would add a massive loading overhead. But I'll work on efficiency later As for supply lines, we've met a stumbling block. Things like ammo, unit defence values, reloading skill (i.e. static data that doesn't really change based on play) are stored in one common location for all units of the same type. So any change to, say, French line infantry would effect every single unit of French line infantry in the game. We're considering re-routing the memory to our own custom addresses, in order to assign these stats on a unit-by-unit basis. But there's a lot of issues with that, not least of all game stability. Hopefully we'll come up with something