Page 1 of 7 1234567 LastLast
Results 1 to 20 of 137

Thread: Empire Mod Architecture Primer

  1. #1

    CA Developer Icon Empire Mod Architecture Primer

    Hi All,

    Since the mod community seems to be moving full speed ahead and you've already got some basic mods up and running I thought I’d explain the way the Empire modding system is intended to work. This should enable you to work with it, not against it, and hopefully avoid breaking things released by CA in the future.

    First of all, the ETW data is meant to stay packed. The pack file system and the virtual file system that it builds internally within the game are central to the patching and modding systems, so please don’t ask users to unpack their data in order to run with your mods.

    So how should you mod the game? Well, it works like this. Empire loads –all– files of type .pack in the data directory in a certain order determined by the pack’s type. There are three basic types – release, patch and mod. The fifth byte in the pack file header is an numerical id which tells the game what a pack is, release packs are 0 or 1, patch packs are 2, mod packs are 3. The game then loads release packs first, then patch packs, then mod packs.

    At game startup, the file entries in all of these packs are mounted into a virtual file system, and if the game finds an entry in a later pack which has the same name and location as an already existing entry, the new entry replaces the old. This means that a mod pack can replace any file in the vfs and add new ones as well, without asking the person who downloaded the mod to overwrite files on disk and destroy data.

    So basically, a mod is intended to be a single “mod pack” file, and installing a mod is just placing that pack inside the game’s data directory, and uninstalling it is removing it. There’s a little bit more to it than that, for example there is a mechanisms to control mod pack load order, but that’s it in a nutshell.

    Now, a note on how to work with the databases. You’ll have noticed that each table in the databases (inside the db folder) is both a folder and a binary file. When the game instantiates the databases, which happens after the virtual file system is built, it scans each directory and parses –all– of the binary files it finds. This means that a mod pack can add a new binary data fragment inside the db/unit_stats_land_table folder which adds several new units, and the game will add them to its land units database. The fragments are processed in the same order as the pack files, and any new database records with the same name as an old record will cause the new to replace the old. This means that you can replace any record within the database without touching the rest of the data or modifying the CA-supplied files on disk.

    For example, say the CA packs contain a unit record for ‘french_dragoons’, which is one entry out of hundreds inside the shipped unit_stats_land_table data fragment, which is used to build the unit table that was available at launch. Now say you create a mod pack which places a new data fragment, say called my_mods_unit_stats_land inside db/ unit_stats_land_table, which contains just one data record for ‘french_dragoons’. What will happen is your new record will end up replacing the current record for that unit in the game data, and the rest will stay as it was originally.

    So… hopefully you’ll now have a good idea of how the modding system was architected, and how you should build tools to work with it. There’s also another interesting consequence of how the system works, which is that it opens the door for mods built on top of other mods…

    One more thing to keep in mind is that we've not yet finished testing the mod support fully - that's part of what's happening here before the mod dev package gets released. Also we’re aware that there is currently an upper bound of about 6000 files in a single pack. And you should know that there is some overhead associated with mounting a pack, and a data directory filled with scores of them will make the game startup time significantly longer.

    The other thing I should mention – since you’re going to be experimenting with this – is that the game checks the final database integrity after loading mods. So if you add a pack file which includes a new unit record which doesn’t include a display model or is invalid in some other way, the game will unload that mod pack (though it keeps the rest). This isn’t the main error checking mechanism in the ETW data pipeline, that’s done elsewhere, but as modders you should be aware of it. Unfortunately there currently isn’t any log output on the reasons why the mod failed to load, we’re looking into adding that when the modding dev kit is released.

    Armed with this info you’ll at least be able to work with the system as it was intended, without having to wait for the mod dev package. I have to say it's great seeing you guys making headway so quickly. The pack and database inspection tool was a great start, I think, and really bodes well for the future of the Empire modding scene.

    Good luck!

  2. #2
    l33tl4m3r's Avatar A Frakkin' Toaster
    Join Date
    Mar 2009
    Location
    Find me in "Soldier of Fortune"
    Posts
    6,328

    Default Re: Empire Mod Architecture Primer

    Thanks for the info!

    Makes me all warm and fuzzy to see this kind of support from a developer.
    [House of Caesars|Under the Patronage of Carl von Döbeln]

  3. #3

    Default Re: Empire Mod Architecture Primer

    Thanks, this is really useful.

    First of all, the ETW data is meant to stay packed.
    Just a small correction: the game flawlessly runs with unpacked files as well. Which means you don't have to repack everything to a .pack file. Actually it's quite useful to keep everything unpacked, so that you don't have to mess with repacking after changing a single file.

    edit: just a quick question, will we receive an importer for 3ds Max (most likely an maxscript) to import the model files? And then, of course and we'd need an exporter created by CA to re-export the models back into the game. Thanks for the answer in advance.
    Last edited by Aldgarkalaughskel; March 19, 2009 at 05:52 AM.

  4. #4

  5. #5

    Default Re: Empire Mod Architecture Primer

    Thanks for the info

  6. #6
    izanagi11's Avatar Semisalis
    Join Date
    Jul 2006
    Location
    Chicago, IL
    Posts
    417

    Default Re: Empire Mod Architecture Primer

    Cool beans! Great info on the header data and any logging would be great. Spent a few hours trying to figure out how to turn that on, I feel better now...
    "The shape of you, the shape of me, the shape of everything I see.."

  7. #7
    Mooncabbage's Avatar Ducenarius
    Join Date
    Oct 2005
    Location
    Australia
    Posts
    990

    Default Re: Empire Mod Architecture Primer

    Yay, thanks for the support
    Not actually on the moon.

    Empire: Tactical Warfare Mod v1.0

  8. #8

    Default Re: Empire Mod Architecture Primer

    Thanks from me too.

    Very good that you CA's keep a focus on all these talented guys around.
    And even better that you provide useful hints!

    Thank you!!!

  9. #9

    Default Re: Empire Mod Architecture Primer

    Hah, this is beautiful. The information about how mod packs work is very useful, and puts my mind at ease. Once the pack editor implements the use of mod packs, I'll be able to restore all the CA packs as they were and (more importantly) we can all free ourselves from the fear that a sudden patch will eat all our mods. The ability to add single-entry database files is absolutely wonderful, too.

    And I'm very excited about the upcoming mod dev package!

    Thanks so much for letting us know. I am now much at ease.

  10. #10

    Default Re: Empire Mod Architecture Primer

    Quote Originally Posted by Iaius Statius Laurentius View Post
    Hah, this is beautiful. The information about how mod packs work is very useful, and puts my mind at ease. Once the pack editor implements the use of mod packs, I'll be able to restore all the CA packs as they were and (more importantly) we can all free ourselves from the fear that a sudden patch will eat all our mods. The ability to add single-entry database files is absolutely wonderful, too.

  11. #11

    Default Re: Empire Mod Architecture Primer

    YAY!

    Just when I needed info about load order
    I speak Spanish to God, Italian to women, French to men, and German to my horse. -Charles V

    My Mods- PipMod, LeatherUI, Spain Retexture+flag swap
    My website

  12. #12
    Yoda Twin's Avatar Vicarius
    Join Date
    Oct 2006
    Location
    Melbourne, Victoria, Australia
    Posts
    2,761

    Default Re: Empire Mod Architecture Primer

    Sweet
    this has to be stickied
    Last edited by Yoda Twin; March 19, 2009 at 06:08 AM.
    Minister for Home Affairs of the Commonwealth v Zentai [2012] HCA 28 per Heydon J at [75]

    Analysis should not be diverted by reflections upon the zeal with which the victors at the end of the Second World War punished the defeated for war crimes. The victors were animated by the ideals of the Atlantic Charter and of the United Nations. The Universal Declaration of Human Rights was about to peep over the eastern horizon. But first, they wanted a little hanging.

  13. #13
    alpaca's Avatar Harbinger of saliva
    Join Date
    Sep 2005
    Location
    Germany
    Posts
    4,811

    Default Re: Empire Mod Architecture Primer

    I hate to be a party pooper but Jerome, this doesn't work as you say. The mod packs are not read at all instead of updating the patch packs. I tested this with an export_triggers.lua file that works (tested on the filesystem first) and with a trait levels effects db that I tested in patch.pack; last but not least I also tried adding a new binary piece to the trait levels effects table but none of the three worked.

    I already tried creating a pack with 3 as a flag a week ago but they're simply ignored by the game

    No thing is everything. Every thing is nothing.

  14. #14

    Default Re: Empire Mod Architecture Primer

    Great, i go under the shower, come back and read this

    Now my skin will hate me for that but if i go again will there be mod tools when i come back ?

    Thanks for the info, it looks pretty much like the nwn architecture exept that data fragments which i did not quite understand

  15. #15

    Default Re: Empire Mod Architecture Primer

    I think Lt. Chambers' editor already implements it, it's just mislabeled - he has

    boot
    release
    patch
    movie

    instead of

    release
    release
    patch
    mod

    Just go to Choose pack action>change pack type.
    I speak Spanish to God, Italian to women, French to men, and German to my horse. -Charles V

    My Mods- PipMod, LeatherUI, Spain Retexture+flag swap
    My website

  16. #16
    alpaca's Avatar Harbinger of saliva
    Join Date
    Sep 2005
    Location
    Germany
    Posts
    4,811

    Default Re: Empire Mod Architecture Primer

    Quote Originally Posted by waronmars View Post
    I think Lt. Chambers' editor already implements it, it's just mislabeled - he has

    boot
    release
    patch
    movie

    instead of

    release
    release
    patch
    mod

    Just go to Choose pack action>change pack type.
    No, this is incorrect. Movie packs have an ID of 4 while mod packs are supposed to have 3. If you set the pack to be a movie pack, however, the changes do show up in the game. It's just not supposed to work that way I guess. We can use it as a hack mod pack type though.

    No thing is everything. Every thing is nothing.

  17. #17

    Default Re: Empire Mod Architecture Primer

    Does the movie type override patch.pack?

  18. #18
    Ebusitanus's Avatar Senator
    Join Date
    Oct 2004
    Location
    Insula Augusta
    Posts
    1,334

    Default Re: Empire Mod Architecture Primer

    Sticky please!
    Read a napoleonic first hand account of a Hessian serving under the french flag

    Athenians: For ourselves, we shall not trouble you with specious pretenses - either of how we have a right to our empire because we overthrew the Mede, or are now attacking you because of wrong that you have done us- and make a long speech which would not be believed;.......... since you know as well as we do that right, as the world goes, is only in question between equals in power, while the strong do what they can and the weak suffer what they must.

    Part of the Melian Dialogue in The History of the Pelopenessian War by Thucydides.

  19. #19
    D.B. Cooper's Avatar Tribunus
    Join Date
    Jul 2007
    Location
    Ottawa, Canada
    Posts
    7,119

    Default Re: Empire Mod Architecture Primer

    That's interesting. Thanks alot!


  20. #20
    alpaca's Avatar Harbinger of saliva
    Join Date
    Sep 2005
    Location
    Germany
    Posts
    4,811

    Default Re: Empire Mod Architecture Primer

    Oh, Jerome, if you have time for it (and sorry for going somewhat off-topic), could you try to ask the expert for that what you have to do to add a trait? Whenever I try to meddle with trait ids, the changes won't be read so I'm beginning to suspect that the list of trait ids that can be passed to effect.trait() is hard coded instead of being read from the db.

    I wrote a bit more about my attempts here: http://www.twcenter.net/forums/showthread.php?t=239316

    No thing is everything. Every thing is nothing.

Page 1 of 7 1234567 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •