Page 1 of 2 12 LastLast
Results 1 to 20 of 31

Thread: Database Table Fragments

  1. #1
    crzyrndm's Avatar Artifex
    Join Date
    Feb 2011
    Location
    New Zealand
    Posts
    2,576

    Default Database Table Fragments

    One basic tool that anyone modifying database files should know and use, for reasons of compatibility and sanity. Sadly, this is heavily underutilised.


    You don't need to include entire tables to make your changes!

    Daniu has already covered in his Modding Primer how the different pack types are loaded in a set order, and how table overwrites are handled. As modders, we all make use of this structure to ensure our modified values overwrite those provided by CA, independent of naming or other annoyances. There is however a little more to it.

    You may have noticed that inside the db folder of data_rome2.pack, each data table is inside it's own little folder. Would it surprise you to learn that the name of the data table itself doesn't actually matter? Below is a capture of the file structure inside my replenishment mod. You will notice that none of the tables shown use the CA naming scheme, and that I have used three seperate tables for building_effects_junctions. Why? Because it is easy for me to work with, and minimises compatibility issues with other mods.



    When tables are being loaded, the folder ("building_effects_junctions_tables") is what controls how a table is interpreted. It doesn't matter what I call those tables, it will still look at each entry using the schema for that folder, and only compare it with previously loaded tables in the same category.

    Why rename tables?
    When tables are being loaded, if the name of your data table matches a previously loaded data table in the same folder, the old table will be removed completely and only dat from the new table will remain. If a previous table with the same name can't be found, individual entries will instead be matched using the key columns, allowing data from a previously loaded table to coexist with new data from your modified table. CA's building_effects_junction table is in the order of 4500 entries yet, because I'm using tables with a different name, no table in my mod exceeds 30 entries. When using this mod, the data used comes from both my fragments, and the CA file loaded prior to it.

    Why is this important to modders and mod users?
    As a mod user the single best thing about fragments is how mods can coexist with no issues even while all modifying the same CA table, so long as they modify different parts of CA's table or add unique entries.
    As a modder, the advantage lies in dealing with significantly reduced quantities of data. Why bother with all 4500 entries in building_effects_junction if you only want to make changes to 26 entries.

    Is there a reason to ever use the official name?
    There does remain one reason not to use fragments. Because they do not replace entire tables, there is no way to remove previously loaded entries without using the same table name. This is the only reason a mod maker should be using a table with the default naming scheme ever. Even when doing a complete overhaul of every single entry in a CA table, renaming adds another layer of protection for your users (eg. if CA patches the game and adds a new entry, your modification will continue to function without issue for your users (no crash due to attempts to reference an undefined entry))

    Fingers crossed more mods start making proper use of this

    Fair winds and following seas to all


    EDIT
    Small example just to show that there is no naming convention for the fragments
    Spoiler Alert, click show to read: 
    Quote Originally Posted by crzyrndm View Post


    Suffix, prefix, rename. It doesn't make a difference (the nomad fragment is for enabling the nomad infantry, nothing to do with Romans)

    Ofcourse, a sensible naming scheme is preferable
    Last edited by crzyrndm; December 08, 2013 at 02:04 AM.
    Its better to excite some and offend others than be bland and acceptable to all
    Creating a mod.pack with PFM - Database Table Fragments

  2. #2
    Silven's Avatar Biarchus
    Join Date
    Jan 2007
    Location
    The Blazing Hot Sun, Arizona
    Posts
    646

    Default Re: Database Table Fragments

    Thanks for this crzyrndm, it would seem that using this format would also lessen the workload of the modder when updating his mod to match or accommodate the changes made by new CA patches.

    I imagine the initial effort to make ones mod format match this change would be pretty substantial, but it would be so worth it for future efforts and mod updates.

  3. #3
    Kurisu Paifuaa's Avatar Biarchus
    Join Date
    Mar 2009
    Location
    Pennsylvania, US
    Posts
    621

    Default Re: Database Table Fragments

    Huh. I didn't realize you could use anything other than a prefix for fragments. I also thought that the base table name was a required element. When did that happen?



    Anyway... hooray for suffixes! The caboose of naming conventions

  4. #4
    crzyrndm's Avatar Artifex
    Join Date
    Feb 2011
    Location
    New Zealand
    Posts
    2,576

    Default Re: Database Table Fragments

    Quote Originally Posted by Silven View Post
    Thanks for this crzyrndm, it would seem that using this format would also lessen the workload of the modder when updating his mod to match or accommodate the changes made by new CA patches.

    I imagine the initial effort to make ones mod format match this change would be pretty substantial, but it would be so worth it for future efforts and mod updates.
    The initial setup effort is definitely a little more if you want to keep it to only modified rows in a mod the size of yours. The workflow advantages though if you split tables like I do...
    Quote Originally Posted by Kurisu Paifuaa View Post
    Huh. I didn't realize you could use anything other than a prefix for fragments. I also thought that the base table name was a required element. When did that happen?



    Anyway... hooray for suffixes! The caboose of naming conventions
    Just to prove the point...


    Suffix, prefix, rename. It doesn't make a difference (the nomad fragment is for enabling the nomad infantry, nothing to do with Romans)

    Ofcourse, a sensible naming scheme is preferable
    Last edited by crzyrndm; November 20, 2013 at 09:43 PM.
    Its better to excite some and offend others than be bland and acceptable to all
    Creating a mod.pack with PFM - Database Table Fragments

  5. #5

    Default Re: Database Table Fragments

    Thanks for this, I hadn't realized how useful this is!

    ----> Website -- Patreon -- Steam -- Forums -- Youtube -- Facebook <----

  6. #6
    crzyrndm's Avatar Artifex
    Join Date
    Feb 2011
    Location
    New Zealand
    Posts
    2,576

    Default Re: Database Table Fragments

    Quote Originally Posted by Dresden View Post
    Thanks for this, I hadn't realized how useful this is!
    I had the thought the other day that there hasn't really been any explicit mention of fragments and how they work for a long time, if ever. Checking a dozen random mods on the workshop and finding only 2 using fragments (and even those in a very minor sense). Thought it was time to do a refresher.
    Its better to excite some and offend others than be bland and acceptable to all
    Creating a mod.pack with PFM - Database Table Fragments

  7. #7
    Kurisu Paifuaa's Avatar Biarchus
    Join Date
    Mar 2009
    Location
    Pennsylvania, US
    Posts
    621

    Default Re: Database Table Fragments

    Quote Originally Posted by crzyrndm View Post
    I had the thought the other day that there hasn't really been any explicit mention of fragments and how they work for a long time, if ever. Checking a dozen random mods on the workshop and finding only 2 using fragments (and even those in a very minor sense). Thought it was time to do a refresher.
    You're quite right. The majority of mods that I see replace the whole table.

    The only instructive reference I can recall dates back to Empire when a dev gave us a brief introduction to the virtual database. The rest has been pretty much scattered about in discussion.

    I thought I recalled there being some tables that wouldn't take piecemeal modification (campaign_variables comes to mind) but that has evolved throughout game releases.

  8. #8
    crzyrndm's Avatar Artifex
    Join Date
    Feb 2011
    Location
    New Zealand
    Posts
    2,576

    Default Re: Database Table Fragments

    Quote Originally Posted by Kurisu Paifuaa View Post
    I thought I recalled there being some tables that wouldn't take piecemeal modification (campaign_variables comes to mind) but that has evolved throughout game releases.
    I haven't found anything in the database tables for Rome 2, and I play with a lot of them

    .loc tables on the other hand don't use the folder structure, so you do have to include the entire table. However they don't normally cause crashes either...
    Its better to excite some and offend others than be bland and acceptable to all
    Creating a mod.pack with PFM - Database Table Fragments

  9. #9
    Kurisu Paifuaa's Avatar Biarchus
    Join Date
    Mar 2009
    Location
    Pennsylvania, US
    Posts
    621

    Default Re: Database Table Fragments

    It would be great if everyone used table fragments and also if PFM would gain the ability to overlay modified data onto the original database (i.e. aggregate data view).

    During my time with a TW game, I typically end up looking at a lot of people's work and combining what I like along with my own changes and additions. It's cumbersome hunting for modified entries and reconciling conflicts, so if I settle on someone's overhaul as a base, I'll use the post-process functions in PFM (which are a great addition) to trim and rename the tables. Though I'd still like a way to compare to a base table other than eyeballing or exporting (to WinMege, NP++, Excel etc.) which is clunky.

    Been giving some thought to importing R2 data and schema into an external database to gain more functionality. Does anyone do this? I'd love some advice before jumping in.

  10. #10
    Blitzkrieg80's Avatar Miles
    Join Date
    Aug 2005
    Location
    Netherlands
    Posts
    313

    Default Re: Database Table Fragments

    Have you actually tested this much? I just tried it and it doesnt work and i verified that my mod works every which way otherwise except how you describe (renaming tables)...

    i'm wondering though - is this process entirely dependent on using Mod Manager? maybe thats why i get different results... im using a movie pack

  11. #11
    crzyrndm's Avatar Artifex
    Join Date
    Feb 2011
    Location
    New Zealand
    Posts
    2,576

    Default Re: Database Table Fragments

    Quote Originally Posted by Blitzkrieg80 View Post
    Have you actually tested this much?
    Have a look at any of my stuff on the workshop (except the aide de camp, which removes entries). If anything I overuse it, because I use fragments to sort everything that I'm working on in each table grouping. The process doesn't depend on pack format as that is only load order, and has nothing to do with the mod manager (as mentioned above, the system has existed since Empire which had no manager).

    It would be very annoying if it suddenly stopped working

    Click image for larger version. 

Name:	Fragments.JPG 
Views:	44 
Size:	46.9 KB 
ID:	297453

    EDIT

    As LestaT mentions below (and in the OP: "Is there a reason to ever use the official name?"), any tables that you want to remove entries from you can't rename. Only changes to existing entries and new entries can go into fragments.
    Last edited by crzyrndm; November 26, 2013 at 09:37 PM.
    Its better to excite some and offend others than be bland and acceptable to all
    Creating a mod.pack with PFM - Database Table Fragments

  12. #12
    AngryTitusPullo's Avatar Comes Limitis
    Join Date
    Jan 2006
    Location
    Kuala Lumpur
    Posts
    13,018

    Default Re: Database Table Fragments

    Not all type modifications can use this. For example the special unit ability file if you have deleted some info, optimize the file and rename it will just read from vanilla the missing links so your mod will have the super power units still even if you intend not to.


    CIVITATVS CVM AVGVSTVS XVI, MMVI
    IN PATROCINIVM SVB Dromikaites SVB MareNostrum SVB Quintus Maximus
    Want to know more about Rome II Total Realism ? Follow us on Twitter & Facebook

  13. #13
    Blitzkrieg80's Avatar Miles
    Join Date
    Aug 2005
    Location
    Netherlands
    Posts
    313

    Default Re: Database Table Fragments

    perhaps the problem is that i am renaming the beginning of the table? is that something you are purposely not doing and commented on or did i miss that?


    ___

    [edit]
    im thinking maybe this is a pfm error like when formula / math does not work properly in bulk without saving?

  14. #14
    crzyrndm's Avatar Artifex
    Join Date
    Feb 2011
    Location
    New Zealand
    Posts
    2,576

    Default Re: Database Table Fragments

    Quote Originally Posted by Blitzkrieg80 View Post
    perhaps the problem is that i am renaming the beginning of the table? is that something you are purposely not doing and commented on or did i miss that?
    The fragment name can be anything you like (post #4), so long as the folder they reside in isn't affected. I use suffixes purely because I find them more readable at a glance
    Its better to excite some and offend others than be bland and acceptable to all
    Creating a mod.pack with PFM - Database Table Fragments

  15. #15
    Blitzkrieg80's Avatar Miles
    Join Date
    Aug 2005
    Location
    Netherlands
    Posts
    313

    Default Re: Database Table Fragments

    I think perhaps my problem is i am testing through Campaign Difficulty Handicap table which does not allow this?

  16. #16
    crzyrndm's Avatar Artifex
    Join Date
    Feb 2011
    Location
    New Zealand
    Posts
    2,576

    Default Re: Database Table Fragments

    Quote Originally Posted by Blitzkrieg80 View Post
    I think perhaps my problem is i am testing through Campaign Difficulty Handicap table which does not allow this?
    Well thats a first for me. You appear to be correct, campaign handicaps isn't working for me either. Makes no sense at all that different tables would be treated differently...

    Adding a new section to the OP
    Its better to excite some and offend others than be bland and acceptable to all
    Creating a mod.pack with PFM - Database Table Fragments

  17. #17
    Blitzkrieg80's Avatar Miles
    Join Date
    Aug 2005
    Location
    Netherlands
    Posts
    313

    Default Re: Database Table Fragments

    Thank you, atleast i'm not crazy... well if everything BUT that works - great... i'll stay away from it and try other stuff hehe

  18. #18
    AngryTitusPullo's Avatar Comes Limitis
    Join Date
    Jan 2006
    Location
    Kuala Lumpur
    Posts
    13,018

    Default Re: Database Table Fragments

    Quote Originally Posted by crzyrndm View Post

    .loc tables on the other hand don't use the folder structure, so you do have to include the entire table. However they don't normally cause crashes either...
    Some works while some not. For example just extract the ui_component (or something, don't remember the exact name) and change the New Campaign to Rome Total realism (or something ) and then add prefix to the file like mymod_ui_component) you can see that it works. I have not tested fragmented one though. Need more test.

    New unit added I think you can added prefix (not tested fragmented) but renaming vanilla units will not captured if you rename the original file.


    CIVITATVS CVM AVGVSTVS XVI, MMVI
    IN PATROCINIVM SVB Dromikaites SVB MareNostrum SVB Quintus Maximus
    Want to know more about Rome II Total Realism ? Follow us on Twitter & Facebook

  19. #19
    Blitzkrieg80's Avatar Miles
    Join Date
    Aug 2005
    Location
    Netherlands
    Posts
    313

    Default Re: Database Table Fragments

    yeah, its weird :O) i just experimented and part of the problem is the pack movie format (in concern to Campaign Difficulty Handicaps).... thats also why Dresdens Food Fix works on the same table i guess (good thing Dresden already tried or wouldn't have bothered)

  20. #20
    crzyrndm's Avatar Artifex
    Join Date
    Feb 2011
    Location
    New Zealand
    Posts
    2,576

    Default Re: Database Table Fragments

    Quote Originally Posted by Blitzkrieg80 View Post
    yeah, its weird :O) i just experimented and part of the problem is the pack movie format (in concern to Campaign Difficulty Handicaps).... thats also why Dresdens Food Fix works on the same table i guess (good thing Dresden already tried or wouldn't have bothered)
    So it does work now? I mustve screwed something up when I quickly tested it then (reminder to self, don't rush)
    Its better to excite some and offend others than be bland and acceptable to all
    Creating a mod.pack with PFM - Database Table Fragments

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •