Page 1 of 29 123456789101126 ... LastLast
Results 1 to 20 of 579

Thread: The "New" Frontier of M2TW Modding: Memory Editing

  1. #1
    Zarathos's Avatar Miles
    Join Date
    Dec 2006
    Location
    Verona, Italia
    Posts
    325

    Icon4 The "New" Frontier of M2TW Modding: Memory Editing

    Hi all! Recently, I started analyzing the M2TW: Kingdoms' executable in order to have a better understanding of the way the engine works for what concerns the way it parses and uses data files (how unknown options and values affect the game and how volatile data is stored into the memory) and the causes of its more common bottlenecks, bugs and misbehaviors. A mastodontic task that, little by little, is producing excellent results, especially for the knowledge of the modding community.

    Then, suddenly, I realized that if I wanted to go further into this (fixing the game engine or modifying its behaviors) I would have to find a way to do it without breaking the game's license... and I found it: memory editing. Let me explain you this: in fact, as we all should know, it's strictly forbidden by CA to alter the game executable, as stated in its eula:

    You agree to only use the Game Software, or any part of it, in a manner that is consistent with this License and you SHALL NOT:

    {...}

    (f) reverse engineer, derive source code, modify, decompile, disassemble, or create derivative works of the Game Software, in whole or in part (except as the applicable law expressly permits, in which case all and any modifications, adaptations, improvements, etc. shall belong to, vest in and be the exclusive property of Sega and/or its licensors on creation, in any event);

    {...}

    (i) create data or executable programs that mimic data or functionality in the Game Software.
    The same rule applies for almost every software house that produce closed source applications but it's not always easy to understand what kind of limitations it imposes. In my point of view, for example, the declarations "reverse engineer" and "create derivative works of the Game Software" should immediately ban all what concerns modding because discovering the structure of binary files (meshes and textures, for example) absolutely fall into the reverse engineering category. The same goes for the statement "create data or executable programs that mimic data or functionality in the Game Software" because edited files can be considered data that mimic existing software files and the Launcher replacements that can be found in this forum can totally be considered executable programs that mimic functionality of the original application. Right?
    While we could waste our time discussing for the next few months about the legitimacy of binary data modifications and launcher replacements without finding a concrete and unanimous solution (it could just be considered ok, we can be sure about the fact that, by all means, releasing a modified executable could lead to very disagreable consequences (signally, a legal action from CA).

    And here is where memory editing comes into play: the first thing I learned when I started working with debuggers, decompilers and disassemblers is that while it's totally forbidden to edit executables, it's 100% legit for a user to edit its own memory without any consequences. Memory editing is a grey zone in which, nowadays, there are still no real rules and this is due to the fact that all what is being put inside the virtual memory of a computer switches its ownership from the productor of a packed entity (the executable compiled by CA) to a consumer of volatile data (yourself). It's your memory, you bought it, you can do whatever you want with it.
    When executables are launched, they are loaded and mapped into the computer's memory and, there, their instructions can be edited as will without breaking any law. This is the correnct and only path we must follow to circumvent the eula's restrictions and bypass any limitation. How? Using a custom launcher/patcher that creates a fully deployed game process in a suspended state, modifies some small parts of it in order to change its behavior and finally resumes its execution. This one, to be exact:



    At present, the application has been created for my personal use only and has never been distributed, but it received a go ahead from the forum's administration. Here is their verdict, that has been sent to me via PM:

    Quote Originally Posted by Gigantus
    Hi Zarathos,

    the issue has been discussed in depth and the decision\guideline is as follows:

    You have a go ahead from GrnEyedDvl (TWC owner) and Jack Lusted (CA member). No altered exes, and no game exes provided, but altering values in the memory is ok.

    If you have further questions, do not hesitate to contact me.

    Greets
    Gig
    By all means, patching the executable can lead to infinite opportunities to improve the engine (adding new features, fixing bugs, improving existing features, removing limitations and so on) of a game that left its development stage long, long time ago and it's no more being improved and maintained by its creators, but that still has a great and very active modding community and is still being sold (blink blink). Here is what I achieved so far:
    • changed the EDU units limit from 500 to 1000 (just some memory allocation adjustments and pointer shiftings remain to be done);
    • changed the soldiers limit from 4/100 to 1/150 (if I'll find the multiplier being used when you set the CFG option unit_size to a value different than normal we will finally be able to have real one-man units);
    • made the executable aware of large memory addresses on 64 Bit systems with at least 6 Gb of RAM;
    • optimized the game engine for single core execution.

    All I need is collaboration. Analyzing an executable is a very difficult and time consuming task to accomplish that requires a high knowledge of software architecture and assembly code. Anyone willing to help me is more than welcome provided he knows how to use the necessary tools (IDA Pro, OllyDBG or another similar debugger and an hexadecimal editor).
    Last edited by Zarathos; May 19, 2014 at 07:35 PM.

  2. #2
    leo.civil.uefs's Avatar É nóis que vôa bruxão!
    Join Date
    Sep 2010
    Location
    Brazil
    Posts
    3,135

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing



    Zarathos is king under the mountain now, his word is law and we must obey.

    Testudo formation, 1 men per unit, disabling terrain flattering beyond the hardcoded height limit...everything.

    Remember help can come from many ways, if you know someone who understand about coding this type of thing, call it, show him that thread, go my fellows, go and spread the good news...

    Here goes my 50 cents:

    THINGS TO CHANGE ON THE .EXE


    -Height limit for terrain (battlemap is flattered beyound 80+ height, thats why battles above mountains will load totlly flat battlemaps)
    - Age limit (elves should live forever!)
    -Faction limit
    -Culture limit
    -Testudo formation.
    -Single man per unit.
    -Trolls/beasts to hit gates (like elephants did in rome 1)
    -Enable arrow slots firing from siege towers again (it is a working feature in rome1)
    -Ajust screen resolutions for widescreen monitors.
    -Remove settlement core type icon from ui of unique settlements.
    -Remove stratmap models from settlements showing even under fog on stramap (it actually shows vanilla med2 models even if your settlement has a custom model).
    -Allowing certain settlements to not require siege engine to be attacked even if its level is one of that needs it (rivendell for example is a large city but has no gates).
    -Creating a new unit type besides cavalary, infantry etc, that would not respect pathfinding both on battlemap and campaign map (for flying units!)
    -Creating a new infantry unit type that would not be able to climb ladders or walls (trolls)
    -Acivating the settlement view feature from Rome.
    Last edited by leo.civil.uefs; June 03, 2014 at 04:27 PM.

  3. #3

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    Nice to see someone bringing M2TW to modern times.

    Good luck with this project mate, hope you make it and allow all bunch of M2TW users to get the best out of this old engine using their PC builds.
    American, French, Israeli and British government's ILLEGAL aggression against the Syrian people, without any proof for chemical attacks in Douma, and without waiting for OPCW to conduct their investigation..
    Sons of *******, leave that poor, war torn country in peace.
    If you are a citizen of one of these countries, then DO NOT ask any help from me on these forums, since, in protest against this aggression by your governments, I do not provide assistance/help anymore.
    Let Syria be finally in peace.

    A video of false chemical attack in Douma, Syria, which led to Western illegal attacks.

  4. #4

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    Awesome work!

  5. #5

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    A big thanks, it's just amazing.

    A new power is rising. Its victory is at hand.
    Last edited by Arkay; May 18, 2014 at 03:55 AM.

  6. #6
    Araval's Avatar Protector Domesticus
    Join Date
    Nov 2010
    Location
    Tartu, Estonia
    Posts
    4,754

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    I was hoping this would happen one day! Sadly I have no experience in this field. Give Zarathos a huge reputation!

    And it's awesome that CA and Administration allowed this!

  7. #7

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    I'd like to help. I study computer science specialized in low level programming and embedded systems. I've written programs in assembly (specifically for the ARM Cortex-M0, but I can easily learn other instruction sets) and am pretty handy with C/C++. Things like memory allocation, how pointers work exactly, hexadecimal representations, endianness, principles of object oriented software architecture, etc are common knowledge for me. I don't have much personal experience with reverse engineering game engines, but I've followed the development of the Elder Scrolls script extenders pretty closely in the past four years or so, so I've got an idea of what's involved. I know I can do this kind of thing, especially with the occasional help from someone more experienced. Let me know if you want to have me.

    Btw, will the development be completely open source? Will it use source control like Mercurial/Git/SVN?

  8. #8
    Vipman's Avatar Protector Domesticus
    Join Date
    Jul 2010
    Location
    Romania
    Posts
    4,405

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    Well I don't know anything about these stuff, I only know of this Large adress aware tool that supposedly makes games use up to 4gb or RAM. In the screen you posted those are just things anyone can edit in the cfg, but if you say it's possible to increase hardcoded limitations like the 500 units or the number of factions maybe, I can't wait to see such a thing with my own eyes! I wish you good luck then.

  9. #9
    irishron's Avatar Cura Palatii
    Join Date
    Feb 2005
    Location
    Cirith Ungol
    Posts
    47,023

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    I have a couple of ui memory issues needing attention when this bears fruit. Congratulations on getting permission and best of luck persuing this. My programming skills are zero.

  10. #10

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    fantastic, would be nice to have upgraded graphics but that's probably out of the question.

    Memory editing sounds very short term until it's saved, then I suspect it breaches the law?

    Anyway Zarathos, fingers crossed it wll be legal and open up many changes.

    R
    oOo

    Rome 2 refugee ...

    oOo

  11. #11
    irishron's Avatar Cura Palatii
    Join Date
    Feb 2005
    Location
    Cirith Ungol
    Posts
    47,023

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    Quote Originally Posted by Rorarii View Post
    fantastic, would be nice to have upgraded graphics but that's probably out of the question.

    Memory editing sounds very short term until it's saved, then I suspect it breaches the law?

    Anyway Zarathos, fingers crossed it wll be legal and open up many changes.

    R
    Big difference.
    If the changes overwrite the .exe, against the law.
    If the changes are saved in savegame, perfectly legal even in the US.

  12. #12
    Zarathos's Avatar Miles
    Join Date
    Dec 2006
    Location
    Verona, Italia
    Posts
    325

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    Many thanks for your replies guys, this is a very nice motivation boost for me

    Quote Originally Posted by Maegfaer View Post
    I'd like to help.

    {...}

    Btw, will the development be completely open source? Will it use source control like Mercurial/Git/SVN?
    You are more than welcome. The last time I did this kind of tasks I was between 16 and 19 years old... many, many years have passed and, meanwhile, I forgot many things and even more things have changed in programming technology.
    The source code of the KingdomsPatcher will NEVER be publicly released for the following reasons:
    1) the purpose of this project is to create a sort of M2TW: Kingdoms 1.6 unofficial patch, so the patcher must be able to work on every PC and on every system in the same way if we want it to be available for everyone still playing the game (exactly as official patches work)... if everyone is going to make its own patcher to modify the game executable the way he prefers, soon or later, the community will be overflowed by patchers that modify the hardcoded limit to different values for different reasons and this fact will pose major compatibility issues and a lot of confusion;
    2) speaking with people about this, I noticed that everyone is pressing to solve his own issues: mappers want mapping problems to be fixed, modelers want better graphics and more complex models, scripters want more commands, and so on... but the real problem is that without a focused effort, this project will go nowhere;
    3) every single modification made to the executable has to be done cum grano salis: it has to be tested and tested and tested again but, most important, has to take into account the limits of a single CPU engine with very old graphic libraries and old school programming approaches... I'm pretty sure that releasing the source code of the patcher will lead to people pushing the hardcoded limits to a level that cannot be tolerated by the game engine, compromising the stability of this work.
    What I can grant instead is that everybody, in the development team, will have full access to my IDA Pro database containing:
    - the disassembled executable with a lot of members (arguments, enumerators, methods, static instances, structures, variables, virtual function tables, etc...) renamed in a coherent and human-readable way (just an example: sub_008EEF56 has been renamed to ParseEDUFile);
    - many inline comments that explain what the executable is doing and how;
    - the binary patches being applied to the memory mapped version of the executable (it's just a simple txt-like file with every single line in the form of <OFFSET>:<OLDBYTE>:<NEWBYTE> which is being used by my KingdomsPatcher also to modify the process into memory by just converting the offsets to a value relative to the process' image base address);
    4) if we let people know how we modify the executable, soon or later modified executables will pop out leading to bad consequences, maybe for us too.

    Quote Originally Posted by Vipman View Post
    Well I don't know anything about these stuff, I only know of this Large adress aware tool that supposedly makes games use up to 4gb or RAM. In the screen you posted those are just things anyone can edit in the cfg, but if you say it's possible to increase hardcoded limitations like the 500 units or the number of factions maybe, I can't wait to see such a thing with my own eyes! I wish you good luck then.
    What you can see is an application I started to develop as a normal launcher replacement for my personal use: of course, all those options can be manually set into the CFG file, but changing the game options before launching it in this way is much more simpler and quick.
    Anyway, this is not the point. That launcher is now being used as a base for my patcher: it changes the CFG file options, it starts the game with the selected mod and then it also patches its memory before resuming the main thread. Of course... executable modifications have no related buttons because they are pretty standard and the memory editing process is totally invisible

    Quote Originally Posted by Rorarii View Post
    fantastic, would be nice to have upgraded graphics but that's probably out of the question.

    Memory editing sounds very short term until it's saved, then I suspect it breaches the law?
    Memory editing will change the executable instructions once it has been loaded and mapped into the RAM. It's a very short term modification, yes, because unlike modifying the executable itself (which is illegal), you need to do this every time you launch the game and, as soon as you close it, every change will be lost.
    And yes... upgraded graphics are almost impossible to be achieved. This would require the use of new graphics library being injected at runtime into the assembly and a major rewriting of every rendering process. I don't even want to think about how much work this would require, it's a nightmare.
    Last edited by Zarathos; May 18, 2014 at 10:24 AM.

  13. #13

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    That is incredible news! Can't wait to see what you are doing to do with this new patcher.

    Does multicore support can be enabled? What are the next of your plans?

    Mundus Bellicus - TWC - ModDB - Discord - Steam
    ~ Patronized by Gaius Baltar, son of the Great Family of imb39, of the House of Garbarsardar, of the Noble House of Wilpuri.

  14. #14

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    Outstandings news!

    I have no knowledge in what you are doing but i have two questions:

    -Will it be possible to have a free camera and an auto-resplenish system like in the warscape total war?

  15. #15

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    Memory editing is not against EULA or any law, so that's why you were allowed this. Many moons ago there was some Spanish modder who did the same thing and even modified the battlefield expansion. He was Agostinos or Agartino something and there's a thread for that MMM modding sth in the medieval 2 mod workshop.

    If you need help with this project then i suggest you contact wilddog and tellos. They might be able to help you out nad they have the proper skills.

    Congrats and good luck with your project..

  16. #16
    Zarathos's Avatar Miles
    Join Date
    Dec 2006
    Location
    Verona, Italia
    Posts
    325

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    Quote Originally Posted by Nelduin View Post
    Does multicore support can be enabled? What are the next of your plans?
    Multicore support cannot be enabled. It's almost impossible, because it's linked to the architecture of the whole executable and the way it works. Next plans?
    - Completing the EDU limits increase.
    - Increasing hidden resources limit from 64 to 128 (I already spotted the necessary ASM opcodes to modify).
    - Creating better exception handling messages to help modders discovering the errors quickly.
    - Fixing major CTD issues.
    - Improving height maps management with heights being truncated to 80.
    - Increasing the regions limit from 200 to 300 (but probably not the map maximum size)
    - Much more...

    Quote Originally Posted by Brune View Post
    -Will it be possible to have a free camera and an auto-resplenish system like in the warscape total war?
    It can be done. In fact, almost everything except improving the game rendering and enabling multicore support can be done... it all depends on how many people will join me for this project, how fast we will discover things and how the game engine has been programmed by CA.

    Quote Originally Posted by Ishan View Post
    Memory editing is not against EULA or any law, so that's why you were allowed this. Many moons ago there was some Spanish modder who did the same thing and even modified the battlefield expansion. He was Agostinos or Agartino something and there's a thread for that MMM modding sth in the medieval 2 mod workshop.

    If you need help with this project then i suggest you contact wilddog and tellos. They might be able to help you out nad they have the proper skills.

    Congrats and good luck with your project..
    I know that thread (by Argantonio), I almost know it by heart. It was very helpful for me, unfortunately it was created for M2TW and not for Kingdoms, so some things have been changed meanwhile by CA developers.
    I'll try to contact those guys, I hope we can set up a small working team of at least 5 persons, because if we can split up the tasks and work together we should go very fast.
    Last edited by Zarathos; May 18, 2014 at 11:33 AM.

  17. #17
    Vipman's Avatar Protector Domesticus
    Join Date
    Jul 2010
    Location
    Romania
    Posts
    4,405

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    Quote Originally Posted by Zarathos View Post
    Multicore support cannot be enabled. It's almost impossible, because it's linked to the architecture of the whole executable and the way it works. Next plans?
    - Completing the EDU limits increase.
    - Increasing hidden resources limit from 64 to 128 (I already spotted the necessary ASM opcodes to modify).
    - Creating better exception handling messages to help modders discovering the errors quickly.
    - Fixing major CTD issues.
    - Improving height maps management with heights being truncated to 80.
    - Increasing the regions limit from 200 to 300 (but probably not the map maximum size)
    - Much more...
    Too bad about the multicore support, but these things you plan sound great, especially fixing ctd issues and improving error reports in the log to find the problems easier sound fantastic! I really hope you will manage to achieve such things and release it

    How would the compatibility with mods be? If with this patcher I select the mod I want to start will it work with any kingdoms mod or will it require some changing some mod files?

  18. #18

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    One Launcher to rule them all, One Launcher to find them,
    One Launcher to bring them all and in the darkness bind them

    Sauron would be proud Zarathos! Congratulations!

  19. #19

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    Grande fratello, hai fatto una cosa colossale!!! Non vedo l'ora... Medieval II è l'unico vero Total War!

    Long life to Medieval II !


  20. #20
    Morrowgan's Avatar Centenarius
    Join Date
    Nov 2010
    Location
    Germany
    Posts
    880

    Default Re: The "New" Frontier of M2TW Modding: Memory Editing

    This seems to be the future of modding
    Member of the Beyond Skyrim Project

Page 1 of 29 123456789101126 ... LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •