Page 1 of 8 12345678 LastLast
Results 1 to 20 of 148

Thread: ETW Runtime Memory Stucture Documention

  1. #1
    Inevitability won
    Patrician Citizen

    Join Date
    Mar 2010
    Posts
    9,594

    Icon1 ETW Runtime Memory Stucture Documention

    This thread will be a place for myself and T.C. to document our progress on our ETW/NTW memory hook and documentation on the memory locations and structure of the running processes memory, as well as discuss its theory and implementation. If you have anything to input, please do.The thread may look bleak, but progress is actually quite astounding, theoretical implementation of what we can already do/know could mean quite astounding things. We will no doubt update everyone at a later point.


    Byte locations relative to units base address
    76 - Unit Size
    80 - Full Unit Size
    ETW:100/NTW:108 - Move Points
    ETW:112 - Unit Experience
    NTW:264 - Length of units custom name


    ETW:232 - 235: Unit stats pointer. (Numbers at this pointer are little-endian)
    +172: Four byte integer, Unit Accuracy
    +176: Four byte integer, Unit Reloading
    +192: Four byte integer, Unit Charge Bonus
    +236: Boolean, Resistant to morale shocks
    +236: Four byte integer, Unit Ammunition


    BETA POINTER:
    Quote Originally Posted by T.C. View Post
    I've confirmed another pointer at the Primary Address. This one seems to relate to units_tables, and we've named it Beta pointer for now. It's structure is a little harder to understand than the Alpha pointer (unit_stats_land). The Beta pointer begins at byte 72 in the primary pointer. As usual, four byte little-endian.

    There doesn't seem to be a header in these files, which is understandable as there is no difference between a naval entry and a land entry in units_tables.

    From the beginning of the memory at the Beta Address:

    Byte 36 - Four byte int, recruit time
    Byte 40 - Four byte int, unit upkeep cost
    Byte 72 - four byte int, unit limit

    There's also a lot more pointers here, probably to the string based data you see in units_tables in PFM.
    Want to help?
    Spoiler Alert, click show to read: 

    Quote Originally Posted by T.C. View Post
    Documentation Tutorial

    Things you'll need:

    • Winhex / any decent hex editor
    • Cheat Engine
    • A grasp of hexadecimal numbers and the difference between data types (bools, ints, shorts, longs, strings)


    Begin by creating a folder called "Logs" in your ETW data directory. Then replace the scripting.lua for the grand campaign (data/campaigns/main) with the one attached to this post. This scripting has a feature which will document the address of any unit, character or settlement you click on in the campaign map. The address will be documented in a text file in the Logs directory which you just created.

    Begin by starting up Empire, preferably with no mods, and go straight into a campaign.
    Then click on something and get it's address - for the moment we are focusing on units, so make sure that the log entry for "Entity" says "Pointer<UNIT>" as in the next picture (red is correct, blue is incorrect).



    Note that clicking on an army always reveals the address of the unit/character in command of the army, so if you are having trouble getting a unit pointer remove any characters from the army (eg generals)

    You then need to open Empire's memory in Winhex. See the picture below for the "Open RAM" button.



    Once you have opened your system RAM, find Empire's process (it will have the name Empire, you'll have to scroll down a bit) and open the section called "Primary memory".

    Now you need to navigate to the location in memory listed by the pointer. Do this by clicking on "Position" -> "Go To Offset", as shown in the next picture.



    Make sure you select relative to beginning, not current offset.

    You will then be brought to the position in memory which the pointer listed. As you will see, Primary Unit Pointers always start with the bytes 68 45 4A 01, or as seen in the interpreter "hEJ ". They end with 00 00 00 8C, if there is another Primary pointer following them (IDK what they end in if there isn't another Primary pointer following them). So now, highlight the bytes representing the Primary pointer (you should see where another PP starts by looking for "hEJ ". Then right click, select "Edit" -> "Copy Block" -> "Into New File"

    You can see below a highlighted Primary unit pointer being right clicked - note the first four bytes and last four bytes.




    Once you have copied the Primary pointer into a new file, you can compare it with other Primary pointers quite easily. It also makes it easy to work your way through the bytes and find the values we have identified (list kept in first post of this thread).

    What you then need to do, is open Cheat Engine. Follow the tutorial presented the first time you run it, though as far as the step 3 (that's all you need that is relevant to this task).

    Once you know how, you can then scan for values - use the "Memory Scan Options" to scan between the start and end of a specific Primary pointer. If you begin with an "unknown initial value scan", then proceed onto an "unchanged value" scan, you should get a good list of results. Using this method, change any values that you think represent in-game stats, and check in game to see if the change has registered. If you verify an unknown value, make a post here and let us know! If you see the OP you will also find out where the Alpha Address/Pointer is located (see my last post on the previous page for info on this). We don't know much about the data stored at the Alpha Address, but we do know some of the interesting stats (such as ammo) are stored there.
    Last edited by .Mitch.; July 18, 2012 at 01:21 PM.

  2. #2

    Default Re: ETW Runtime Memory Stucture Documention

    Reserved.
    My Tools, Tutorials and Resources

    Was running out of space, so see the full list here!

    Consider the postage stamp: its usefulness consists in the ability to stick to one thing till it gets there.- Josh Billings
    The creatures outside looked from pig to man, and from man to pig, and from pig to man again; but already it was impossible to say which was which.- George Orwell

  3. #3

    Default Re: ETW Runtime Memory Stucture Documention

    Byte 112 confirmed as unit experience.

    If anyone is interested in helping us with this I can write up a tutorial. It's quite interesting and rewarding, and very simple to do.
    My Tools, Tutorials and Resources

    Was running out of space, so see the full list here!

    Consider the postage stamp: its usefulness consists in the ability to stick to one thing till it gets there.- Josh Billings
    The creatures outside looked from pig to man, and from man to pig, and from pig to man again; but already it was impossible to say which was which.- George Orwell

  4. #4

    Default Re: ETW Runtime Memory Stucture Documention

    What are you doing, using a memory scanner, searching for the number, changing it, and searching the first search's result to see what changed?
    Under the Patronage of Leonidas the Lion|Patron of Imperator of Rome - Dewy - Crazyeyesreaper|American and Proud

  5. #5
    Inevitability won
    Patrician Citizen

    Join Date
    Mar 2010
    Posts
    9,594

    Default Re: ETW Runtime Memory Stucture Documention

    Not really.

    Topcats LUA outputs the base address of a units location in memory. For example: 0x040ffbef0. So memory scanning isn't necessary.

    We then read the following bytes after the base address to give us the position of certain attributes of the unit, for example 76 bytes after a units base address is the units size.

    I've been using my own program that reads the following 1000 bytes after a base address and cross references similarities to the unit. Topcat has been using cheatengine to read the memory in hex and more tediously comparing them manually.

    Now I know that 76 bytes after a base address is a units size, I can do things like match a units location on the campaign map to a colourmap of europe or w/e and based on its location (if its in snow/desert) we can effect the unit. So basically from what we know it would be extremely easy to implement NTW's attrition feature into ETW.

    That's just one example of a million possibilities. We can do exactly this for cities, etc, etc.

    And this is two days into things... lol

  6. #6

    Default Re: ETW Runtime Memory Stucture Documention

    Yeah I'm a huge fan of this stuff, though I've only ever made simple trainers (infinite lives, etc.) I'll be following your progress, this is some nice stuff. The real challenge is going to be creating functions that modders can reference.
    Under the Patronage of Leonidas the Lion|Patron of Imperator of Rome - Dewy - Crazyeyesreaper|American and Proud

  7. #7

    Default Re: ETW Runtime Memory Stucture Documention

    Quote Originally Posted by Bolkonsky View Post
    Yeah I'm a huge fan of this stuff, though I've only ever made simple trainers (infinite lives, etc.) I'll be following your progress, this is some nice stuff. The real challenge is going to be creating functions that modders can reference.
    That's going to be easy on our behalf; the problem for other modders is acquiring the memory pointers, which takes a bit of scripting knowledge to do. In fact I'd say I'm one of very few poeple who knows how to access these pointers, at least I haven't seen of any other active modders utilising them for scripting. I'll have to get writing tutorials again

    If you want to help out with this Bolk we can get you off the ground quite easily.
    My Tools, Tutorials and Resources

    Was running out of space, so see the full list here!

    Consider the postage stamp: its usefulness consists in the ability to stick to one thing till it gets there.- Josh Billings
    The creatures outside looked from pig to man, and from man to pig, and from pig to man again; but already it was impossible to say which was which.- George Orwell

  8. #8
    Inevitability won
    Patrician Citizen

    Join Date
    Mar 2010
    Posts
    9,594

    Default Re: ETW Runtime Memory Stucture Documention

    Quote Originally Posted by Bolkonsky View Post
    The real challenge is going to be creating functions that modders can reference.
    I've already got the concept in my head of how we'll make this available to others.

    The main application of this will be implementation of extra game mechanics and so without sounding greedy anything 'amazing' will likely be done natively in the executable (sort of like a second game engine) instead of via any secondary functionality.

    I don't know if LUA is capable of working with an executable (infact I know it is.. I just have no experience of LUA implementation into C/C++/C#), if I do then I'll likely make an API of sorts to interface with it and call the functions available to the executable. Via the scripting.lua of the game you can do literally just about anything you want, you just can't make any effects into the game. So for instance you can find out alot of information, make a million if statements have a really l33t system in place, but you cant make it effect the campaign. That was our whole idea here really, to make the possibilites of what LUA can do actually do something.

    I know TC has programming experience so I'll likely just make my source code available to him at some point so he can implement his own functionality without having to bother with the shite I've gone through.

    Nomatter what though people will need LUA experience (at the very least) to make use of this, I'll likely make TC write some LUA functions that output text command lines to a file, my program will read the file and take the commands from the stack and implement them. Sort of a very very crap API.

    If I implement a modding API into warscape can I haz opifex pl0x?

  9. #9
    General Brewster's Avatar The Flying Dutchman
    Citizen Censor

    Join Date
    Jul 2011
    Location
    Kingdom of The Netherlands
    Posts
    13,865
    Blog Entries
    9

    Default Re: ETW Runtime Memory Stucture Documention

    Well T.C if it's easy to understand and learn I might be able to help you out.

  10. #10
    Tribunus
    Join Date
    Jan 2011
    Location
    Ascension, St. Helena
    Posts
    7,336

    Default Re: ETW Runtime Memory Stucture Documention

    Quote Originally Posted by .Mitch. View Post
    If I implement a modding API into warscape can I haz opifex pl0x?
    Based on this work alone you're both in a prime position.

  11. #11

    Default Re: ETW Runtime Memory Stucture Documention

    Ok so a little theory, most of which will probably only be relevant to Mitch and I until I write more background tutorials.

    Note that I'm not used to working with data structures this low level, apologies for any incorrect terminology.

    The memory chunks which unit pointers point at have the following format:

    First four bytes [0 - 3] are some sort of header; they seem to take the format 68 45 4A 01.

    The next four bytes is [4 - 7] are an index of sorts - possibly linked into the "unitXX" ID that is assigned to unit card components, though I doubt it.

    Next four unknown, however the next four after that [12 - 15] seem to always take the form D8 45 4A 01 (notice the bytes 45 4A 01, they repeat frequently throughout the memory sector)

    Apart from that we know very little, except that each unit pointer ends in 00 00 00 8C when there is another pointer following it.

    More to come.
    My Tools, Tutorials and Resources

    Was running out of space, so see the full list here!

    Consider the postage stamp: its usefulness consists in the ability to stick to one thing till it gets there.- Josh Billings
    The creatures outside looked from pig to man, and from man to pig, and from pig to man again; but already it was impossible to say which was which.- George Orwell

  12. #12

    Default Re: ETW Runtime Memory Stucture Documention

    Quote Originally Posted by General Brewster View Post
    Well T.C if it's easy to understand and learn I might be able to help you out.
    Any experience with binary? makes no difference either way
    Last edited by T.C.; June 21, 2012 at 06:01 PM.
    My Tools, Tutorials and Resources

    Was running out of space, so see the full list here!

    Consider the postage stamp: its usefulness consists in the ability to stick to one thing till it gets there.- Josh Billings
    The creatures outside looked from pig to man, and from man to pig, and from pig to man again; but already it was impossible to say which was which.- George Orwell

  13. #13

    Default Re: ETW Runtime Memory Stucture Documention

    I've just confirmed the location in memory of unit ammo, and a chunk of other static stats. Will write documentation up on this later today.

    In other terms, we will soon have supply lines that affect unit's ammo on the campaign
    My Tools, Tutorials and Resources

    Was running out of space, so see the full list here!

    Consider the postage stamp: its usefulness consists in the ability to stick to one thing till it gets there.- Josh Billings
    The creatures outside looked from pig to man, and from man to pig, and from pig to man again; but already it was impossible to say which was which.- George Orwell

  14. #14

    Default Re: ETW Runtime Memory Stucture Documention

    Hi guys.

    In the past I tracked down a few NTW's structure layouts in memory and their pointer offsets, especially concerning battle and ai.

    A couple simple examples:

    Code:
           /* Alliance structure (?? bytes). */
            struct Alliance
            {
                struct {
                    char _Pad00[8];
                    BattleModel *m_BattleModel;
                    char _Pad0C[4];
                } *m_Unk00;
                DWORD m_Unk04;
                DWORD m_Unk08;
                DWORD m_AIPlayer;
                int m_ID;
                ArmyArray m_ArmyList; // 0x1C (-8)
                char _Pad28[16];
                LandUnitArray m_VisibleEnemyUnitList;
                char _Pad4C[52];
            };
    
            /* Army structure (0x24C bytes). */
            // Army +0x128/0x12C is a m_UnitGroupList
            struct Army
            {
                DWORD m_VTable;
                int m_Id;
                DWORD m_Cache;
                char _Pad0C[204];
                Alliance *m_Alliance; // 0xD8
                char _PadDC[8];
                LandUnitArray m_UnitList; // 0xEC (-8)
                char _PadF8[340];
            };
    But I think we need a DLL code-injector first to make any decent usage of them.
    I'm waiting for a Craig's statement on this "burning issue" because it's legally very borderline (although ethically more questionable, as it's just memory patching and no copyrighted material is shared). Crossing fingers.
    Bye.

  15. #15
    General Brewster's Avatar The Flying Dutchman
    Citizen Censor

    Join Date
    Jul 2011
    Location
    Kingdom of The Netherlands
    Posts
    13,865
    Blog Entries
    9

    Default Re: ETW Runtime Memory Stucture Documention

    Quote Originally Posted by T.C. View Post
    Any experience with binary? makes no difference either way
    Nope total noob

  16. #16

    Default Re: ETW Runtime Memory Stucture Documention

    Quote Originally Posted by risorgimento View Post
    Hi guys.

    In the past I tracked down a few NTW's structure layouts in memory and their pointer offsets, especially concerning battle and ai.

    A couple simple examples:

    Code:
           /* Alliance structure (?? bytes). */
            struct Alliance
            {
                struct {
                    char _Pad00[8];
                    BattleModel *m_BattleModel;
                    char _Pad0C[4];
                } *m_Unk00;
                DWORD m_Unk04;
                DWORD m_Unk08;
                DWORD m_AIPlayer;
                int m_ID;
                ArmyArray m_ArmyList; // 0x1C (-8)
                char _Pad28[16];
                LandUnitArray m_VisibleEnemyUnitList;
                char _Pad4C[52];
            };
    
            /* Army structure (0x24C bytes). */
            // Army +0x128/0x12C is a m_UnitGroupList
            struct Army
            {
                DWORD m_VTable;
                int m_Id;
                DWORD m_Cache;
                char _Pad0C[204];
                Alliance *m_Alliance; // 0xD8
                char _PadDC[8];
                LandUnitArray m_UnitList; // 0xEC (-8)
                char _PadF8[340];
            };
    But I think we need a DLL code-injector first to make any decent usage of them.
    I'm waiting for a Craig's statement on this "burning issue" because it's legally very borderline (although ethically more questionable, as it's just memory patching and no copyrighted material is shared). Crossing fingers.
    Bye.
    Nice info! How did you come across this?

    Quote Originally Posted by General Brewster View Post
    Nope total noob
    Better wait till I write a tutorial on the subject then. In the mean time acquaint yourself with Hexadecimal numbers, data types (ints, bools, floats, shorts, strings) and the difference between bits and bytes.
    My Tools, Tutorials and Resources

    Was running out of space, so see the full list here!

    Consider the postage stamp: its usefulness consists in the ability to stick to one thing till it gets there.- Josh Billings
    The creatures outside looked from pig to man, and from man to pig, and from pig to man again; but already it was impossible to say which was which.- George Orwell

  17. #17
    General Brewster's Avatar The Flying Dutchman
    Citizen Censor

    Join Date
    Jul 2011
    Location
    Kingdom of The Netherlands
    Posts
    13,865
    Blog Entries
    9

    Default Re: ETW Runtime Memory Stucture Documention

    Sure.

  18. #18

    Default Re: ETW Runtime Memory Stucture Documention

    If I understand how it works, I'll give it a shot. Will likely need the tuto first though.

  19. #19

    Default Re: ETW Runtime Memory Stucture Documention

    Ok, here's a complete run down of what I know about the memory structure used by ETW, followed by a tutorial on documentation.

    Empire's Lua API provides a lot of functions, the most interesting of which are in the library named CampaignUI. An example function is InitialiseUnitDetails(), which returns almost all of the known details about the unit queried. The unit is queried by means of a unit pointer - a pointer to an address in memory. An example use follows:

    Code:
    CampaignUI.InitialiseUnitDetails(0x01f4bb388)
    Note that the above code would never work, as the addresses are dynamic and change quite a lot. We can use other CampaignUI functions to find unit pointers, I'll write a tutorial on that at a later stage.

    So Mitch and I began by actually looking up this address in Empire's memory using a hex editor. The name for the sector at which these addresses point shall be the "Primary Address" from here on, to avoid confusion. At the Primary Address (PA) there is a distinctive pattern in memory - 248 bytes of memory starting with the header 65 48 4A 01 and ending with 00 00 00 8C (in 99% of cases). Obviously this 248 bytes isn't near enough space to store all of the info on a unit, so from that we correctly concluded that these 248 bytes were a mixture of some unit data and more memory pointers. The unit data stored at the PA is mostly dynamic data - data which can, and does, change frequently during the course of play. So far we have identified at the PA:
    • Current Unit Size
    • Unit size after replenishment (ie the unit at full strength)
    • Action points (has no in game effect, but verified as Action Points by means of Lua logging)
    • Unit experience
    • Length of unit's name string


    We have also successfully identified a crucial sub-pointer in the PA. The memory at which this sub pointer points to shall from here on be referred to as the "Alpha Address". Stored at the AA are things such as unit accuracy, reloading skill, charge bonus, ammunition and flags indicating abilities (the only confirmed flag so far is "Resistant to morale shocks"). The AA contains segments of data from unit_stats_land, including what we have identified as unused columns but crucially contains no string data. It's possible that there are more sub pointers here.

    So what do we need to do now? Primarily, find out what data is pointers, and where these pointers point, both in the PA and the AA. There has to be pointers here that lead us to data such as projectiles, associated 2d and 3d artwork, text based data etc.


    Here is also my latest documentation:
    At the PA:

    Bytes 232 - 235: Pointer to the AA.

    At the AA:

    Byte 172: Four byte integer, Unit Accuracy
    Byte 176: Four byte integer, Unit Reloading
    Byte 192: Four byte integer, Unit Charge Bonus
    Byte 236: Boolean, Resistant to morale shocks
    Byte 236: Four byte integer, Unit Ammunition

    All numbers are also little-endian.

    Tutorial coming...
    Last edited by T.C.; June 23, 2012 at 09:27 AM.
    My Tools, Tutorials and Resources

    Was running out of space, so see the full list here!

    Consider the postage stamp: its usefulness consists in the ability to stick to one thing till it gets there.- Josh Billings
    The creatures outside looked from pig to man, and from man to pig, and from pig to man again; but already it was impossible to say which was which.- George Orwell

  20. #20
    Foederatus
    Join Date
    Jul 2011
    Location
    Under your sink
    Posts
    42

    Default Re: ETW Runtime Memory Stucture Documention

    Thank you TC. This is where I want to focus my modding efforts after the models portions (i'm a very moddest moddler at that). Given the information you have provided, does this mean we can only modify the information you have provided or are there more details that we have yet to unravel? I'm assuming much more, but you know what that means. Obviously I will be one of the many who will take up the torch once the tutorials have been written...

Page 1 of 8 12345678 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •