DB<->TSV converter for Shogun 2 Total War [released!]

[taw]

Also taw or someone else can you look to repair the file building_level_armed_citizenry_junctions_tables
The second column includes strange numbers which will surely corrupt the game if we want to add new lines there.

Spoiler Alert, click show to read: 

version 1, schema building_level_armed_citizenry_junctions_tables, guid d3f63af9-6f91-4117-85ca-132dde881364
unknown1(string) unknown2(int) unknown3(float) unknown4(int) unknown5(string)
"SHO_Castle_5_Citadel" 0 0 2 "Inf_Spear_Yari_Ashigaru_Citizenry"
"SHO_Castle_4_Castle" 102 0 5 "Inf_Heavy_Onna_Bushi"
"SHO_Castle_5_Citadel" 109 0 2 "Inf_Spear_Yari_Ashigaru_Citizenry"
"SHO_Castle_5_Citadel" 112 0 3 "Inf_Spear_Yari_Ashigaru_Citizenry"
"SHO_Castle_5_Citadel" 117 0 2 "Inf_Missile_Bow_Ashigaru_Citizenry"
"SHO_Castle_5_Citadel" 124 0 4 "Inf_Sword_Samurai_Retainers"
"SHO_Castle_5_Citadel" 134 0 3 "Inf_Missile_Bow_Ashigaru_Citizenry"
"SHO_Castle_5_Citadel" 143 0 4 "Inf_Sword_Samurai_Retainers"
"SHO_Castle_5_Citadel" 149 0 5 "Inf_Heavy_Onna_Bushi"
"SHO_Castle_5_Citadel" 153 0 4 "Inf_Sword_Samurai_Retainers"
"SHO_Castle_5_Citadel" 156 0 3 "Inf_Missile_Bow_Ashigaru_Citizenry"
"SHO_Castle_5_Citadel" 159 0 2 "Inf_Missile_Bow_Ashigaru_Citizenry"
"SHO_Castle_3_Fortress" 16 0 2 "Inf_Spear_Yari_Ashigaru_Citizenry"
"SHO_Archery_4_Legendary_Dojo" 198 0 5 "Inf_Missile_Bow_Samurai_Dojo"
"SHO_Cavalry_4_Legendary_Bajutsu_Dojo" 207 0 5 "Cav_Spear_Yari_Cavalry_Dojo"
"SHO_Siege_4_Arsenal" 213 0 5 "Inf_Special_Fire_Bomb_Throwers_Dojo"
"SHO_Yari_4_Legendary_School" 220 0 5 "Inf_Spear_Yari_Samurai_Dojo"
"SHO_Sword_4_Legendary_Dojo" 224 0 5 "Inf_Sword_Katana_Samurai_Dojo"
"SHO_Castle_2_Stronghold" 23 0 2 "Inf_Spear_Yari_Ashigaru_Citizenry"
"SHO_Ikko_Temple_1_Jodo_Shinshu_Temple" 233 0 4 "Inf_Heavy_Naginata_Warrior_Monks_Citizenry"
"SHO_Ikko_Temple_2_Jodo_Shinshu_Monastery" 237 0 4 "Inf_Heavy_Naginata_Warrior_Monks_Citizenry"
"SHO_Ikko_Temple_3_Jodo_Shinshu_Fortified_Monastery" 243 0 4 "Inf_Heavy_Naginata_Warrior_Monks_Citizenry"
"SHO_Ikko_Temple_4_Jodo_Shinshu_Honganji" 252 0 4 "Inf_Heavy_Naginata_Warrior_Monks_Citizenry"
"SHO_Ikko_Temple_2_Jodo_Shinshu_Monastery" 254 0 4 "Inf_Heavy_Naginata_Warrior_Monks_Citizenry"
"SHO_Ikko_Temple_3_Jodo_Shinshu_Fortified_Monastery" 263 0 4 "Inf_Heavy_Naginata_Warrior_Monks_Citizenry"
"SHO_Ikko_Temple_3_Jodo_Shinshu_Fortified_Monastery" 269 0 4 "Inf_Heavy_Naginata_Warrior_Monks_Citizenry"
"SHO_Ikko_Temple_4_Jodo_Shinshu_Honganji" 278 0 4 "Inf_Heavy_Naginata_Warrior_Monks_Citizenry"
"SHO_Ikko_Temple_4_Jodo_Shinshu_Honganji" 280 0 4 "Inf_Heavy_Naginata_Warrior_Monks_Citizenry"
"SHO_Ikko_Temple_4_Jodo_Shinshu_Honganji" 287 0 4 "Inf_Heavy_Naginata_Warrior_Monks_Citizenry"
"SHO_Castle_1_Fort" 36 0 1 "Inf_Sword_Samurai_Retainers"
"SHO_Castle_2_Stronghold" 40 0 1 "Inf_Sword_Samurai_Retainers"
"SHO_Castle_3_Fortress" 62 0 3 "Inf_Missile_Bow_Ashigaru_Citizenry"
"SHO_Castle_3_Fortress" 73 0 4 "Inf_Sword_Samurai_Retainers"
"SHO_Castle_3_Fortress" 74 0 3 "Inf_Spear_Yari_Ashigaru_Citizenry"
"SHO_Castle_4_Castle" 78 0 2 "Inf_Spear_Yari_Ashigaru_Citizenry"
"SHO_Castle_4_Castle" 81 0 3 "Inf_Spear_Yari_Ashigaru_Citizenry"
"SHO_Castle_4_Castle" 83 0 2 "Inf_Missile_Bow_Ashigaru_Citizenry"
"SHO_Castle_4_Castle" 90 0 4 "Inf_Sword_Samurai_Retainers"
"SHO_Castle_4_Castle" 95 0 3 "Inf_Missile_Bow_Ashigaru_Citizenry"
"SHO_Castle_4_Castle" 96 0 4 "Inf_Sword_Samurai_Retainers"

unknown2 is definitely a 32-bit integer. It cannot possibly be anything else.

What's the meaning of this integer I have no clue. It's probably index to some list.
I have no way of figuring this out by just analyzing db tables, you'll need someone
to look at rest of the game files.


On the other hand unknown3 is probably int not float, there's no way to tell if it's all zeroes,
as float zero and uint32 zero look identical (00 00 00 00).


Mistaking int vs float columns is the most common mistake autodetector makes.
I'm sure there are a few such mistakes in DB.xsd somewhere.

If you see some really weird integers or really weird floats, try changing type in DB.xsd and check if
they make more sense after that. If so, report this finding back to me so I can integrate this.

[husserlTW]

If you hate making large scale changes in TSV(s) from db conversion using text editor (since spread sheets have problem with ") here is how I work it out. http://www.twcenter.net/forums/showthread.php?p=9276922#post9276922. Perhaps it make your life easier...

[Dogmeat]

To be honest building_level_armed_citizenry_junctions_tables looks like a mess to me.

Column two is generally an accending scale from lv1 castle to lv 5 but with some wierd exceptions. Plus what looks like some out of place lines.

Also it seems that many of the garrisoned troops that should be available aint,

And like darth there are problems if u edit the table then try to repack with taw. (not a problem just though id mention)

[Swiss Halberdier]

Not sure if this is because TAW uses the Demo that i assume doesnt include the patch packs. Maybe some philanthropic modders should send TAW a stream gift of STW2.

I gifted STW2 to taw over Steam for his valuable and amazing work for the community!

That he always has the latest files and can enjoy the work of all modders who use his awesome tools.

Hello taw, here are my thoughts about the column names in the "unit_special_abilities_tables". I'm interested what other modders think about it.

Spoiler Alert, click show to read: 

unknown1(string)
Duration of the ability (seconds)

unknown2(float)
Cool down time of the ability (seconds)

unknown3(float)
amount of use (I guess that "-1" means unlimited)

I don't know what the other columns mean yet.

"encourage_fleet" 45 60 -1 300 true 5 0 nil false -1 nil
"inspire_unit" 60 300 -1 0 false 0 0 nil false 120 nil
"inspire_unit_naval" 60 300 -1 0 false 0 0 nil false 60 nil
"rally" 120 240 -1 75 true -1 0 nil false -1 nil
"second_wind" 40 300 -1 40 true 3 0 nil false 150 nil
"stand_firm" 45 300 -1 40 true 3 0 nil false 60 nil

[Yarkis de Bodemloze]

I have the impression that many table entries have some kind of index number now. Hard to tell if these are just random or follow a certain order.

[Dogmeat]

I gifted STW2 to taw over Steam for his valuable and amazing work for the community!

Good Karma Swiss !

I have the impression that many table entries have some kind of index number now. Hard to tell if these are just random or follow a certain order.

agreed theres the strange building garrison column discussed above, and in units table the last column is a index like starting from 1 - 186 except for what looks like a few they forgot an were given longer numbers like "Art_Large_Shafted_Fire_Rockets_Hojo" 236

maybe this column unit table index could be llinked to the sound voices - hard to tell not speakin japanese.....

[The Hedge Knight]

No voices are in unit stats land.

Just consider them random ids with missing entries for deleted units.

[Yarkis de Bodemloze]

No voices are in unit stats land.

Just consider them random ids with missing entries for deleted units.

When I encountered them in tables and added new entries I created some random ones myself. So far no side effects I noticed.

[The Hedge Knight]

Darth has this for builds_armed_citizenry_junctions... Presumed the file was corrupted, i mean honesty its just a number...

The process at current for many seems to be

1: see strange thing in tables
2:Complain about somthing being wrong

It should be

1:See strange thing
2:test
3:report findings
4:If it doesnt work, why not?

[Dogmeat]

No complaints here,

On unit building garrisons - i have tried adding new lines there add just adding a new index number that was unique an in the range of the others.

Got errors when i try an unpack or repack with taw tools - so me be highy un technical dosnt really know what that would mean.

EDIT: tried the proccess again with 0 in the unknown indexy column and all works fine.

Last column may well be just random id's

On sound index in Unit table im just specultating that sound might have some link there like there is for unit class and caste - sounds being class based in the type, ie etw 'Lite infanty reporting' and that in etw an ntw there was a bug when adding units to the unit table would mess up the sound responces for many untouched vanilla units. -but i guess this is going off topic a bit.

EDIT: Seems CA have very nicely put 3 columns in the unit stats land that deal with the sound responces for units in battle. So thats last column in units looks more an mre like a simple index.

[The Hedge Knight]

Hey Taw, the new pfm has partially parsed some new tables

Eg: Unit stats land
Unit stats naval
Uniforms

COuld you use these to parse the rest?

eg:

Unit stats land

unit_stats_land Unit IDRef,String;
Unit Size,UInt32;
Number of Horses,UInt32;
Number of Artillery,UInt32;
Officer,String;
Has Drummer?,Boolean,1;Drummer,String;
Has Standard Bearer?,Boolean,1;Standard Bearer,String;
Model Name,String;
Infantry Battle Entity,String;
Animation Table,String;
Warscape Equipment Theme IDRef,String;
Armor Value,UInt32;
Armor Type,String;
unknown,UInt32;
Has Horse 1?,Boolean,1;Horse 1,String;
Has Horse 2?,Boolean,1;Cavalry Battle Entity,String;
Has Horse 3?,Boolean,1;Horse 3,String;
Unknown,Int32;
Unknown,Int32;
unknown,3;
Has Canon?,Boolean,1;Canon,String;
Unknown,Boolean;
Has Artillery 1?,Boolean,1;Artillery 1,String;
Has Artillery 2?,Boolean,1;Artillery 2,String;
Accuracy,UInt32;
Reloading Skill,UInt32;
unknown,String;
unknown,String;
Has Projectile?,Boolean,1;Projectile IDRef,String;
Ammo,UInt32;
unknown,Boolean,1;unknown,String;
Melee Weapon,String;
Melee Skill,UInt32;
Charge Bonus,UInt32;
Defense Skill,UInt32;
unknown,UInt32;
unknown,UInt32;
unknown,UInt32;
unknown,String;
unknown,String;
Drill Set,String;
Training,String;
Morale,UInt32;
Deployed Ranks,UInt32;
File Spacing,Single;
Rank Spacing,Single;
unused,Single;
unused,Single;
Dismounted File Spacing,Single;
Dismounted Rank Spacing,Single;
unknown,Single;
unknown,Single;
unknown,Single;
Anti Cavalry Bonus,UInt32;
Can Throw Grenades,Boolean;
Can Skirmish,Boolean;
Can Guard,Boolean;
Can Dismount,Boolean;
Unknown,Boolean;
Is Slow Moving,Boolean;
Can Rally Comrades,Boolean;
Can Hide in Woodland,Boolean;
Can Hide in Light Scrub,Boolean;
Can Hide in Long Grass,Boolean;
Can Hide in Buildings,Boolean;
Can Hide when Walking,Boolean;
Can Hide when Firing,Boolean;
Can Fire from Horseback,Boolean;
Spotting value 1,Single;
Spotting Value 2,Single;
Spotting Value 3,Single;
Paths Seldom Trod,Boolean;
Garrison Police Bonus,Boolean;
Morale Shock Resistant,Boolean;
unknown,Boolean;
Scares Horses,Boolean;
Scares Enemies,Boolean;
Inspires Allies,Boolean;
Good Stamina,Boolean;
Use Grappling Hooks,Boolean;
Heat Resistant,Boolean;
Cold Resistant,Boolean;
unknown,Boolean;
Can Use Pike Wall,Boolean;
Can Use Pike Square,Boolean;
Immune to Attrition,Boolean;
Can Shoot While Moving,Boolean;
->,Boolean,1;Flutist,String;
->,Boolean,1;Limber Destruction Model1,String;
->,Boolean,1;Limber Destruction Model1,String;
unknown,Boolean;
Unknown,String;
Unknown,String;
Unknown,String;
Unknown,String;
Unknown,String;
Unknown,String;
Unknown,String;
Unknown,String;
Unknown,5

But its missing some of the napoleon bits so the 24 most likely has

Paths Seldom Trod,Boolean;
Garrison Police Bonus,Boolean;
Morale Shock Resistant,Boolean;
unknown,Boolean;
Scares Horses,Boolean;
Scares Enemies,Boolean;
Inspires Allies,Boolean;
Good Stamina,Boolean;
Use Grappling Hooks,Boolean;
Heat Resistant,Boolean;
Cold Resistant,Boolean;
unknown,Boolean;
Can Use Pike Wall,Boolean;
Can Use Pike Square,Boolean;
Immune to Attrition,Boolean;
Can Shoot While Moving,Boolean;
->,Boolean,1;Flutist,String;
->,Boolean,1;Limber Destruction Model1,String;
->,Boolean,1;Limber Destruction Model1,String;
unknown,Boolean

and somthing else in it.

[truongquanghung]

I used to be able to edit the tsv files with win xp but after I upgraded to win 7 it always mess up the file when I tried to add or remove things. Can anyone help ?

P/S: Nvm, I thought Notepad++ is included in Win 7. Notepad++ works like a charm.

[son_tac]

I used Pack File Manager 1.6.4 to unpack "unit_stats_land" from patch2.pack and use Taws to convert db to tsv in dos but i got the following error:
"Conversion of db\unit_stats_land_tables/unit_stats_land(version 14) failed"
can some one tell what might cause this error to appear ?

[Dogmeat]

son_tac Taw's tools cant pack/unpack unit_stats_land table yet - just edit that table from Pack File Manager. It also has an export tsv function from there.

But there are tables that PFM cant edit yet but TAW tools can pack/unpack. Tools are still in dev stage so until they get fully updated using both tools is what i do.

and of course thanks again to TAW and Co for the continuing efforts.

[L'Ost]

I gifted STW2 to taw over Steam for his valuable and amazing work for the community!

That he always has the latest files and can enjoy the work of all modders who use his awesome tools.

Hello taw, here are my thoughts about the column names in the "unit_special_abilities_tables". I'm interested what other modders think about it.

Spoiler Alert, click show to read: 

unknown1(string)
Duration of the ability (seconds)

unknown2(float)
Cool down time of the ability (seconds)

unknown3(float)
amount of use (I guess that "-1" means unlimited)

I don't know what the other columns mean yet.

"encourage_fleet" 45 60 -1 300 true 5 0 nil false -1 nil
"inspire_unit" 60 300 -1 0 false 0 0 nil false 120 nil
"inspire_unit_naval" 60 300 -1 0 false 0 0 nil false 60 nil
"rally" 120 240 -1 75 true -1 0 nil false -1 nil
"second_wind" 40 300 -1 40 true 3 0 nil false 150 nil
"stand_firm" 45 300 -1 40 true 3 0 nil false 60 nil

4th column migth be range ?

[son_tac]

thanks dogmeat for the info gonna try that tonight !!!!

[Swiss Halberdier]

4th column migth be range ?

Yes that's right!

Spoiler Alert, click show to read: 

unknown1(string)
Duration of the ability (seconds)

unknown2(float)
Cool down time of the ability (seconds)

unknown3(float)
amount of use (I guess that "-1" means unlimited)

unknown4(float)
range (radius of the ability)

"encourage_fleet" 45 60 -1 300 true 5 0 nil false -1 nil
"inspire_unit" 60 300 -1 0 false 0 0 nil false 120 nil
"inspire_unit_naval" 60 300 -1 0 false 0 0 nil false 60 nil
"rally" 120 240 -1 75 true -1 0 nil false -1 nil
"second_wind" 40 300 -1 40 true 3 0 nil false 150 nil
"stand_firm" 45 300 -1 40 true 3 0 nil false 60 nil

[drake178]

Yes that's right!

Spoiler Alert, click show to read: 

unknown1(string)
Duration of the ability (seconds)

unknown2(float)
Cool down time of the ability (seconds)

unknown3(float)
amount of use (I guess that "-1" means unlimited)

unknown4(float)
range (radius of the ability)

"encourage_fleet" 45 60 -1 300 true 5 0 nil false -1 nil
"inspire_unit" 60 300 -1 0 false 0 0 nil false 120 nil
"inspire_unit_naval" 60 300 -1 0 false 0 0 nil false 60 nil
"rally" 120 240 -1 75 true -1 0 nil false -1 nil
"second_wind" 40 300 -1 40 true 3 0 nil false 150 nil
"stand_firm" 45 300 -1 40 true 3 0 nil false 60 nil

I'd recommend calling it "area of effect" instead of range (radius is good). Unknown5 is likely personal (true) / targeted (false), Unknown6 should be max # of affected units, Unknown7, 8, 9, and 11 are unused in these ones (i'm too lazy to look at the whole file), and Unknown10 is initial cooldown in seconds (at the start of the battle). Looks like there's no real range column that would define how far you can target the non-personal ones.

Edit: after looking at the whole file in PFM Unknown6 is max # of friendly units affected, Unknown7 is max # of enemy units affected, -1 means all that are in the area. You've got the rest already.

[Swiss Halberdier]

I'd recommend calling it "area of effect" instead of range (radius is good). Unknown5 is likely personal (true) / targeted (false), Unknown6 should be max # of affected units, Unknown7, 8, 9, and 11 are unused in these ones (i'm too lazy to look at the whole file), and Unknown10 is initial cooldown in seconds (at the start of the battle). Looks like there's no real range column that would define how far you can target the non-personal ones.

Edit: after looking at the whole file in PFM Unknown6 is max # of friendly units affected, Unknown7 is max # of enemy units affected, -1 means all that are in the area. You've got the rest already.

Thanks a lot drake and very good find! +rep

[The Hedge Knight]

On that i found if you open excel first (dont double click the file).

Then in advanced options change the text seperator form " to none.

Then once you have saved open it with a text editor and use find and replace to change """ to ".