DB<->TSV converter for Shogun 2 Total War [released!]

[taw]

I did look at the xmls, but had forgotten about the xsd schemata. I'll check them out.

Well a string is a string, not a different data type; and including the possibility of having other data optional although it isn't used is just more flexible and has no disadvantages.
But you know, I doubt the "optional" string idea to begin with. It doesn't make any sense at all: if it's there, it's just a byte more; if it's not there, you still have to include the 0 byte which you would need anyway to denote the empty string.

I'm quite certain that this isn't an "optional" string at all, but a string array. This would fit the "sequence" elements in the xsd files (and there are array types in ESF as well).
If you think about it, an "optional string" is just a special case of an array allowing only length of 0 or 1.
I haven't gone too deep into looking at the data files in a hex editor, but if arrays of ints were also possible, that would easily confuse people trying to analyse the structure.

I opened loads of binary formats used by Total War, and they don't code arrays like that ever.

Arrays are always: [uint32le element count] [element 0] [element 1] etc.

Strings are always [uint16le codepoints count] [uint16le codepoint 0] [uint16le codepoint 1] etc.

Single byte 0/1 prefix is just for optstrings, or nullable strings, or whatever you want to call them. They are not arrays.

They only example of array indexes being not uint32le are ESF tag numbers, counts of which is uint16le, but they are all uint16le numbers, so there's literally no way it can exceed this number.

Just to be sure: those are included in the schema.xml you checked in at github yesterday, right? Because that's what I'm working with right now.

It's all on github, you can always check latest version and history of every file if you need to.

[daniu]

Okay, the array thing was just an idea.

I now finished transferring the DBFileTypes into the schema format. I also "unified" names (lowercase and underscores for spaces) in both files and sorted yours by table name so I'll have an easier time comparing them. The zip file I attached contains both xmls.
There are still a lot of differences between the two, I'll just test both by brute force opening all db files in all pack files to find the ones that work best. Probably yours since you said you tested those already

Once I got all the data consistent, I can start working on integrating them into the GUI tools.

[daniu]

Okay it looks to me like I got the reading of the schema.xml right.
I think I changed it in some places along the way, if you could run the tests you do with the one I attached, that would be good as a double-check.
I added the tag "version_end" which is analogous to "version_start" to be able to remove columns in later versions; but that was only really needed once in an NTW agents_tables.
Oh, I did strip the references for now.

I can open a large range of db files for all of etw, ntw and stw now.
Here are my test results; "passed" means I saved and reloaded the files, to verify I got the same content and don't corrupt it along the way.
I used the most recent patches for all TWs and, for S2, threw in Radious' mod for good measure.
I only checked the db/* subtree.

ETW

testing pack file /opt/mono/packs/etw/etw-patch2.pack
type: Patch/1, supersedes patch.pack, contains 2414 files
passed 108/108
unknown files: 1
models_building

NTW

testing pack file /opt/mono/packs/ntw/ntw-patch2.pack
type: Patch/1, supersedes patch.pack, contains 770 files
passed 19/19
testing pack file /opt/mono/packs/ntw/ntw-patch4.pack
type: Patch/1, supersedes patch3.pack, contains 2554 files
passed 117/117
unknown files: 0

STW

testing pack file /opt/mono/packs/stw/Radious_Total_War_Mod.pack
type: Movie/0, supersedes , contains 386 files
passed 56/56
testing pack file /opt/mono/packs/stw/Radious_Total_War_RotS.pack
type: Movie/0, supersedes , contains 407 files
passed 76/76
testing pack file /opt/mono/packs/stw/s2-patch.pack
type: Patch/0, supersedes , contains 13081 files
passed 428/428
testing pack file /opt/mono/packs/stw/s2-patch10.pack
type: Patch/1, supersedes patch9.pack, contains 608 files
passed 517/517
testing pack file /opt/mono/packs/stw/s2-patch12.pack
type: Patch/1, supersedes patch11.pack, contains 698 files
passed 517/517
testing pack file /opt/mono/packs/stw/s2-patch9.pack
type: Patch/1, supersedes patch8.pack, contains 7922 files
passed 517/517
unknown files: 23
avatar_ancillaries_to_categories_juncs
avatar_campaign_maps
avatar_dojo_codependencies
avatar_gempei_dojos
avatar_skill_sets
campaign_bonus_value_ids_unit_records
cdir_events_mission_incidents
events_effect_group_junct
events_hist_chars_junct
faction_variables
faction_variables_optional_overrides
groupings_cultures_junct
groupings_empires_junct
groupings_subcultures_junct
models_building
models_naval
mp_force_gen_upgrade_junctions
resource_effects
sea_climate_details
ship_names
small_vegetation_climates_jct
trigger_event_to_excluded_agent_types
uniform_naval_to_faction_colours

I've done all this on a command line interface; I'll start integrating it into the UI of PFM and later DBE now.

[taw]

Hi there,

I've been busy and then away on holidays, and this fell through cracks.

What are we doing here again? Should I take the last schema.zip you sent (from this post), test it against all db tables in all games,
fix any problems that occur, and upload it to etwng?

[daniu]

No, hold on, I think I have a newer version that I didn't upload because you didn't respond, but I don't have access to it right now.

[taw]

I've waited over a month, I can wait a bit more ;-)

[daniu]

Okay, here we go. I hope it's the correct one, but I think so.
The tests I wrote for it were running fine, but it's obviously possible I did something wrong writing them.
What I (hope I) do in them is
- load the db file with the schema.xml
- verify that the entry count is correct
- write the file into a temp one, then compare the two

I run this with all these packs:
etw:
patch2.pack

ntw:
patch2.pack patch4.pack

stw:
Radious_Total_War_Mod.pack patch10_mod.pack patch12.pack patch.pack
Radious_Total_War_RotS.pack patch10.pack patch9.pack


I probably introduced the index difference back into the xml, but I didn't know how else to handle this; after all, some older packs don't even have a version so I need to assume something (chose 0), and otherwise I use the value from the pack file. I don't know why the DBEditor uses a different indexing, didn't check the code.

[taw]

Two problems:
* My converter doesn't support version_end yet sadly. Now I get it that it's needed for S2TW but your schema seems to be using it even for ETW/NTW versions, where it is definitely not used ever. A lot of your version_end entries seem incorrect (things like version_end=5 optstring, followed by version_start=5 boolean "unused" in the same place - yeah right...)
* All your version numbers are off by one

There's no such thing as db files that don't have versions. No version header means version 1 (by DBEditor convertion). Version headers start with version 2 (which is encoded as hex 01) and go on.

Could you upload schema with version numbers fixed?

Once you do that, I'll test tables that don't rely on version_end first. Once I have a bit more free time, I'll add support for version_end (without breaking DB.xsd conversion I mean, that's the main hard part, I'll need to make something that exports multiple DB.xsd versions from one schema.xml or somesuch).

[Bad213Boy]

Sorta new to modding S2TW. If I understand this correctly, this tool will allow new textures for units? What kind of file does this edit and where can I locate these files in the games pack files? I would like to start playing around with all these tools so I can get a better understanding of modding. Thx.

[taw]

I've added support for a bunch of new tables from DB.xsd I got sent ages ago ;-)

[kungfuserge]

Can you support a UI converter for STW2 ?

[taw]

It would probably take a lot of time and effort to support that, and I cannot currently do that.

[M@GIC]

have a tsv file, with ripple tool I can open the tsv file????

and how do I make a tsv file from a. pack file?

[taw]

I think you need to unpack it first, get Pack File Manager for easy user interface for that.

[daniu]

Hey taw,
been looking into your code to unpack the models_building tables to add an editor to PFM. Looks like you only handle up to NTW right now?
If I'm not entirely off, all you have to do to get the S2 ones going is to just read the start of the file like a normal DB file header (GUID, version etc) and the rest will work with the code you already have.

[taw]

You are correct, and S2 tables are now supported - it was just extra GUID.

Does anybody ever use models_building tables anyway...?

[alhoon]

Ehh... which file should we download to turn ESF to XML and vice versa?

[007Jua007]

how to install this?? in the download page there are many files but i have no idea how to download them...

[Junaidi83 de Bodemloze]

how to install this?? in the download page there are many files but i have no idea how to download them...

DB <-> TSV is kind of old method back when pfm not perfect like now when most modder just try modding shogun 2 , maybe i wrong , its just my mind is foggy.

Is just like old version of pfm maybe ?

[007Jua007]

hmm, anyway, I cannot use PFM here, im on a mac, PFM works only for extracting tables.. so I was extracting and editing with text editor directly but it would not work, I thought maybe I had to convert first... do you know any other method of editing S2 tables without PFM??