DB<->TSV converter for Shogun 2 Total War [released!]

[taw]

So I tried to unpack the dbs on my old XP PC and it didnt work either. Till now no success with Win7 64bit, Vista 32 and XP 32. Everytime I've received the same error message posted before. I guess the Unpacker and Windows aint no friends.

This error is probably related to version of JRuby or nokogiri, not to version of Windows.

I'll fix it when I have some free time. If you're desperate, find Linux LiveCD somewhere or try it with older JRuby/nokogiri.

[taw]

nokogiri claims to have fixed their Ruby Windows installer.

Try this if you have problems (no guarantee).

[Swiss Halberdier]

nokogiri claims to have fixed their Ruby Windows installer.
Try this if you have problems (no guarantee).

Thanks for this information!

Do you have a priority list of which tables should be parsed first in your next release?

Like these:

Spoiler Alert, click show to read: 

battles_tables
building_units_allowed_tables
factions_tables
naval_uniforms_tables
projectiles_tables
technologies_tables
uniform_naval_to_faction_colours_tables
uniform_to_faction_colours_tables
uniforms_tables
unit_abilities_tables
unit_stats_land_tables
unit_stats_naval_crew_tables
unit_stats_naval_tables
unit_special_abilities_tables > the General abilities
units_tables

I don't want to stress you and it's just a suggestion. We can be lucky and grateful enough for your effort that your amazing tools work in Shogun 2. +rep

[taw]

Thanks for this information!

Do you have a priority list of which tables should be parsed first in your next release?

Yes. My priority is to fix ETW tables which got accidentally broken by all this.

[Agostinos]

Success
I uninstalled nokogiri with jgem uninstall nokogiri. Then I installed the latest ruby and entered in cmd gem install nokogiri. This way I was able to convert the files. I will test this on my other PC aswell. I guess this is the solution to this problem.

[taw]

307 tables (15 empty tables, and 292 non-empty tables) from S2TW convert now.

Including exactly 1 of requested tables (uniform_naval_to_faction_colours_tables), which is empty, so it was pretty easy.

Tables you see are what I could get out of modified ETW auto-detector without too much effort.

I might tweak it to analyze a few moderately complicated tables later,
but don't expect huge tables like units_tables and such anytime soon.

In the mean time, please find correct column names for new tables and send them to me.
Editing DB.xsd without DBEditor working properly would be hard, so just send me what you've got,
and I'll figure out some way to integrate it.

Anyway, here are some samples. To get more download the most recent code:

version 1, schema _kv_key_buildings_tables, guid 8fd1a011-7093-4b05-9ee7-51528601b10a
unknown1(string)        unknown2(float)
"archery_dojo_ammo_replenish_rate_local_modifier"       0.2
"archery_dojo_reload_skill_global_modifier"     35
"farmhouse_fatigue_modifier"    -0.5
"farmhouse_speed_global_modifier"       0.2
"fort_tower_fire_frequency_large"       4
"fort_tower_fire_frequency_medium"      4
"fort_tower_fire_frequency_small"       4
"fort_tower_fire_frequency_ultra"       4
"gate_capture_time"     15
"keep_capture_time"     60
"key_building_capture_time"     90
"local_bonus_radius"    55
"shrine_morale_global_modifier" 5
"shrine_morale_local_modifier"  0
"sword_dojo_charge_bonus_global_modifier"       4
"sword_dojo_charge_bonus_local_modifier"        0
"sword_dojo_melee_attack_global_modifier"       5
"sword_dojo_melee_attack_local_modifier"        0
"tower_capture_time"    15
"workshop_armour_global_modifier"       4
"workshop_armour_local_modifier"        0
"workshop_defence_global_modifier"      3
"workshop_defence_local_modifier"       0

version 2, schema avatar_xp_rewards_tables, guid 405f1614-a49f-40b2-b722-914f24aaa081
unknown1(string)        unknown2(float)
"battle_lost"   50
"battle_won"    150
"enemy_avatar_killed"   25
"enemy_avatar_routed"   25
"enemy_avatar_shattered"        25
"got_an_achievement"    25
"most_enemy_units_routed_on_team"       25
"most_enemy_units_wiped_out_on_team"    25
"most_kills_on_team"    25
"most_units_rallied_on_team"    25
"pb_best_streak"        25
"pb_didnt_get_worst_kills_(once_only)"  25
"pb_didnt_lose_(once_only)"     25
"pb_enemy_units_wiped_out"      25
"pb_most_enemy_units_shattered" 25
"pb_most_kills" 25
"pb_most_units_rallied" 25
"pb_special_abilities_used"     25
"rank_differential_modifier_negative"   0.1
"rank_differential_modifier_positive"   0.1
"result_defeat_close"   10
"result_defeat_crushing"        0
"result_defeat_decisive"        0
"result_defeat_valiant" 10
"result_draw"   0
"result_victory_close"  15
"result_victory_decisive"       30
"result_victory_heroic" 60
"result_victory_pyrrhic"        10
"special_abilities_used_on_team"        25

[Kurtisb]

Upon installing Ruby and Nokogiri for it, using the new version of the DB unpack worked. Well done.

[Swiss Halberdier]

307 tables (15 empty tables, and 292 non-empty tables) from S2TW convert now.

Including exactly 1 of requested tables (uniform_naval_to_faction_colours_tables), which is empty, so it was pretty easy.

In the mean time, please find correct column names for new tables and send them to me.
Editing DB.xsd without DBEditor working properly would be hard, so just send me what you've got,
and I'll figure out some way to integrate it.

Thanks a lot for your huge effort and the new version of your amazing tools! It works with Ruby and Jruby! +rep

I will inform you about my findings of column names in the tables.

[The Hedge Knight]

Taw it looks like unit stats land is just the napoleon one with eight strings bunged on the end, can you confirm this?

[Swiss Halberdier]

Hello taw, today I unpacked the db tables of the data.pack without any problems to TSV files.

The patch.pack db tables are causing this error here:

Spoiler Alert, click show to read: 

EDIT: The new patch2.pack can be converted to TSV and it includes the same amount of tables. Thank you so much for your useful tools! +rep again

[Adeptus Modifcus]

Firstly muchly thanks TAW for the work done so far in unlocking the db tables for our modding pleasure.

secondarily

EDIT: The new patch2.pack can be converted to TSV and it includes the same amount of tables. Thank you so much for your useful tools! +rep again

Swiss Not sure but after latest patch i didnt get any new patch2.pack in data dir


according to time stamps these are the only files i saw changed/

shogun2.dll
Safe Browsing Bloom Filter 2
Safe Browsing Bloom
awesomium (an empty text file ???)

A new folder named Default

DATA dir shows new time stamp on folder but all files inside show orig install date time stamp.

So im just wondering if what i got was a mixed up vers of patch.

Anyway soz for goin off topic an thnx again TAW for the tools.

EDIT: Part Answer seems there were bugs with the patch for some so the update has been pulled or a smaller hotfix released instead. Seems alot didnt get the patch2 pack either.

[husserlTW]

Any news for important tables like units, building_levels, etc.?

[taw]

Any news for important tables like units, building_levels, etc.?

No. I don't currently have time for any modding activities.

And neither have I received any schemas, or even column annotations so far.
Come on guys, pretty much everybody should be able to label columns in newly discovered tables, right?

[husserlTW]

Where can we test them? Can I add in db2tsv DB.xsd and test until succeed?

[taw]

Pretty much, but DB.xsd format sucks and it has three entries for each table
Plus forth for foreign keys, which DB Editor uses, but db<->tsv converter couldn't care less about.

If you just add new tables, you can skip primary keys as well.
If you want to rename columns, you'll have to rename existing primary key entry too.
(but I have scripts to detect DB.xsd inconsistencies, so if there are any problems, I will know)

The worst problem is with tables which changed format between versions.
DB.xsd has support for them of course, but it's so horribly failful I don't even want to think about it.

Three entries per table look like this:

<xs:complexType name='avatar_xp_rewards_tables'>
  <xs:attribute name='unknown1' type='xs:string' use='required'/>
  <xs:attribute name='unknown2' type='xs:float' use='required'/>
</xs:complexType>

<xs:element form='unqualified' name='avatar_xp_rewards_tables' type='mstns:avatar_xp_rewards_tables'/>

<xs:unique msdata:PrimaryKey='true' name='avatar_xp_rewards_tables_tables_Constraint1'>
  <xs:selector xpath='.//avatar_xp_rewards_tables'/>
  <xs:field xpath='@unknown1'/>
</xs:unique>

I don't recommend people trying to figure out this, because I have autodetector which I can use to open
large number of tables at once, and manual work for ETW took forever and didn't even open half of them.
But if you're in a hurry, go on and try.

[The Hedge Knight]

Ill update some column headings when i have time too and send you the updated schema.

[KozCDVI]

I've looked at DB format for S2TW, and it is very similar to ETW/NTW.

• Optional GUID: - the only thing new in S2TW, everything else is old
  
  • Bytes FD FE FC FF
  • uint16 GUID length in UTF-16 characters (always 36)
  • UTF-16 String guid 3913248c-da3e-4fe9-889b-e018ef212bfd
  
  
• Version number, either:
  
  • Byte 01
  
  
• or:
  
  • Bytes FC FD FE FF
  • Byte - version number minus one (so 01 here is newer than previous 01)
  • Bytes 00 00 00 01
  
  
• Uint32 number of rows
• data

EDIT: Converter now works, and is available here.
I broke far too many ETW tables in process (due to using very strict table version checks),
so I'll have to go back and fix them soon.

195 of S2TW tables with ETW-compatible schemas are supported.

No new S2TW are supported yet, but I have db schema autodetector
I'll use to quickly add support for them at some point (definitely after fixing ETW tables I broke).

Any volunteers to update DB Editor as well (and does it at least handle NTW yet)?

This sounds awesome, but what does it actually mean?

[Regs]

Obviously this is used to unpack the pack files for io-- first mods? Is there a readme showing how to unpack the files and using them?

Thanks for all your help! You guys are the greatest.

[taw]

Obviously this is used to unpack the pack files for io-- first mods? Is there a readme showing how to unpack the files and using them?

Thanks for all your help! You guys are the greatest.

Yes, there's a README.txt included.

[The Hedge Knight]

So taw how does one edit the .xsd to change the column descriptions, i tried to but it now doesnt convert anything but the blank tables.