One basic tool that anyone modifying database files should know and use, for reasons of compatibility and sanity. Sadly, this is heavily underutilised.
You don't need to include entire tables to make your changes!
Daniu has already covered in his Modding Primer how the different pack types are loaded in a set order, and how table overwrites are handled. As modders, we all make use of this structure to ensure our modified values overwrite those provided by CA, independent of naming or other annoyances. There is however a little more to it.
You may have noticed that inside the db folder of data_rome2.pack, each data table is inside it's own little folder. Would it surprise you to learn that the name of the data table itself doesn't actually matter? Below is a capture of the file structure inside my replenishment mod. You will notice that none of the tables shown use the CA naming scheme, and that I have used three seperate tables for building_effects_junctions. Why? Because it is easy for me to work with, and minimises compatibility issues with other mods.
When tables are being loaded, the folder ("building_effects_junctions_tables") is what controls how a table is interpreted. It doesn't matter what I call those tables, it will still look at each entry using the schema for that folder, and only compare it with previously loaded tables in the same category.
Why rename tables?
When tables are being loaded, if the name of your data table matches a previously loaded data table in the same folder, the old table will be removed completely and only dat from the new table will remain. If a previous table with the same name can't be found, individual entries will instead be matched using the key columns, allowing data from a previously loaded table to coexist with new data from your modified table. CA's building_effects_junction table is in the order of 4500 entries yet, because I'm using tables with a different name, no table in my mod exceeds 30 entries. When using this mod, the data used comes from both my fragments, and the CA file loaded prior to it.
Why is this important to modders and mod users?
As a mod user the single best thing about fragments is how mods can coexist with no issues even while all modifying the same CA table, so long as they modify different parts of CA's table or add unique entries.
As a modder, the advantage lies in dealing with significantly reduced quantities of data. Why bother with all 4500 entries in building_effects_junction if you only want to make changes to 26 entries.
Is there a reason to ever use the official name?
There does remain one reason not to use fragments. Because they do not replace entire tables, there is no way to remove previously loaded entries without using the same table name. This is the only reason a mod maker should be using a table with the default naming scheme ever. Even when doing a complete overhaul of every single entry in a CA table, renaming adds another layer of protection for your users (eg. if CA patches the game and adds a new entry, your modification will continue to function without issue for your users (no crash due to attempts to reference an undefined entry))
Fingers crossed more mods start making proper use of this
Fair winds and following seas to all
EDIT
Small example just to show that there is no naming convention for the fragments
Spoiler Alert, click show to read:
Originally Posted by crzyrndm
Suffix, prefix, rename. It doesn't make a difference (the nomad fragment is for enabling the nomad infantry, nothing to do with Romans)
Ofcourse, a sensible naming scheme is preferable
Last edited by crzyrndm; December 08, 2013 at 01:04 AM.
Thanks for this crzyrndm, it would seem that using this format would also lessen the workload of the modder when updating his mod to match or accommodate the changes made by new CA patches.
I imagine the initial effort to make ones mod format match this change would be pretty substantial, but it would be so worth it for future efforts and mod updates.
Huh. I didn't realize you could use anything other than a prefix for fragments. I also thought that the base table name was a required element. When did that happen?
Anyway... hooray for suffixes! The caboose of naming conventions
Thanks for this crzyrndm, it would seem that using this format would also lessen the workload of the modder when updating his mod to match or accommodate the changes made by new CA patches.
I imagine the initial effort to make ones mod format match this change would be pretty substantial, but it would be so worth it for future efforts and mod updates.
The initial setup effort is definitely a little more if you want to keep it to only modified rows in a mod the size of yours. The workflow advantages though if you split tables like I do...
Originally Posted by Kurisu Paifuaa
Huh. I didn't realize you could use anything other than a prefix for fragments. I also thought that the base table name was a required element. When did that happen?
Anyway... hooray for suffixes! The caboose of naming conventions
Just to prove the point...
Suffix, prefix, rename. It doesn't make a difference (the nomad fragment is for enabling the nomad infantry, nothing to do with Romans)
Ofcourse, a sensible naming scheme is preferable
Last edited by crzyrndm; November 20, 2013 at 08:43 PM.
Thanks for this, I hadn't realized how useful this is!
I had the thought the other day that there hasn't really been any explicit mention of fragments and how they work for a long time, if ever. Checking a dozen random mods on the workshop and finding only 2 using fragments (and even those in a very minor sense). Thought it was time to do a refresher.
I had the thought the other day that there hasn't really been any explicit mention of fragments and how they work for a long time, if ever. Checking a dozen random mods on the workshop and finding only 2 using fragments (and even those in a very minor sense). Thought it was time to do a refresher.
You're quite right. The majority of mods that I see replace the whole table.
The only instructive reference I can recall dates back to Empire when a dev gave us a brief introduction to the virtual database. The rest has been pretty much scattered about in discussion.
I thought I recalled there being some tables that wouldn't take piecemeal modification (campaign_variables comes to mind) but that has evolved throughout game releases.
I thought I recalled there being some tables that wouldn't take piecemeal modification (campaign_variables comes to mind) but that has evolved throughout game releases.
I haven't found anything in the database tables for Rome 2, and I play with a lot of them
.loc tables on the other hand don't use the folder structure, so you do have to include the entire table. However they don't normally cause crashes either...
It would be great if everyone used table fragments and also if PFM would gain the ability to overlay modified data onto the original database (i.e. aggregate data view).
During my time with a TW game, I typically end up looking at a lot of people's work and combining what I like along with my own changes and additions. It's cumbersome hunting for modified entries and reconciling conflicts, so if I settle on someone's overhaul as a base, I'll use the post-process functions in PFM (which are a great addition) to trim and rename the tables. Though I'd still like a way to compare to a base table other than eyeballing or exporting (to WinMege, NP++, Excel etc.) which is clunky.
Been giving some thought to importing R2 data and schema into an external database to gain more functionality. Does anyone do this? I'd love some advice before jumping in.
Have you actually tested this much? I just tried it and it doesnt work and i verified that my mod works every which way otherwise except how you describe (renaming tables)...
i'm wondering though - is this process entirely dependent on using Mod Manager? maybe thats why i get different results... im using a movie pack
Have a look at any of my stuff on the workshop (except the aide de camp, which removes entries). If anything I overuse it, because I use fragments to sort everything that I'm working on in each table grouping. The process doesn't depend on pack format as that is only load order, and has nothing to do with the mod manager (as mentioned above, the system has existed since Empire which had no manager).
It would be very annoying if it suddenly stopped working
EDIT
As LestaT mentions below (and in the OP: "Is there a reason to ever use the official name?"), any tables that you want to remove entries from you can't rename. Only changes to existing entries and new entries can go into fragments.
Last edited by crzyrndm; November 26, 2013 at 08:37 PM.
Not all type modifications can use this. For example the special unit ability file if you have deleted some info, optimize the file and rename it will just read from vanilla the missing links so your mod will have the super power units still even if you intend not to.
perhaps the problem is that i am renaming the beginning of the table? is that something you are purposely not doing and commented on or did i miss that?
___
[edit]
im thinking maybe this is a pfm error like when formula / math does not work properly in bulk without saving?
Last edited by Blitzkrieg80; November 26, 2013 at 08:52 PM.
perhaps the problem is that i am renaming the beginning of the table? is that something you are purposely not doing and commented on or did i miss that?
The fragment name can be anything you like (post #4), so long as the folder they reside in isn't affected. I use suffixes purely because I find them more readable at a glance
I think perhaps my problem is i am testing through Campaign Difficulty Handicap table which does not allow this?
Well thats a first for me. You appear to be correct, campaign handicaps isn't working for me either. Makes no sense at all that different tables would be treated differently...
.loc tables on the other hand don't use the folder structure, so you do have to include the entire table. However they don't normally cause crashes either...
Some works while some not. For example just extract the ui_component (or something, don't remember the exact name) and change the New Campaign to Rome Total realism (or something ) and then add prefix to the file like mymod_ui_component) you can see that it works. I have not tested fragmented one though. Need more test.
New unit added I think you can added prefix (not tested fragmented) but renaming vanilla units will not captured if you rename the original file.
yeah, its weird :O) i just experimented and part of the problem is the pack movie format (in concern to Campaign Difficulty Handicaps).... thats also why Dresdens Food Fix works on the same table i guess (good thing Dresden already tried or wouldn't have bothered)
yeah, its weird :O) i just experimented and part of the problem is the pack movie format (in concern to Campaign Difficulty Handicaps).... thats also why Dresdens Food Fix works on the same table i guess (good thing Dresden already tried or wouldn't have bothered)
So it does work now? I mustve screwed something up when I quickly tested it then (reminder to self, don't rush)
Database Table Fragments
Originally Posted by crzyrndm
Reply With Quote
