Corrupted Sounds Solutions and Discussion

[crux3D]

Sure thing.

Uploaded the parts of Radious' mod I'm using, my units mod (don't understand why it's so big for adding 4 units...), and my packed sounds.

It looks like there is a bug in the way I'm handling completely replaced unit tables. I was able to reproduce your issue easily - it is indeed the Ikko Ikki Bow Ashigaru that demonstrate it (other units may do as well). If you simply rename your units table from Seyal_Shoujo_Bushi.pack to something else (i.e. db/units_tables/units to db/units_tables/units_seyal) then Symphony seems to work. I'm not sure I've ever tested a complete units table replacement before. I think I've only tested self-generated table additions and Radious unit mods, neither of which replace the db/units_tables/units file, they just add additional db files in the db/units_tables directory. I'll look into this as soon as I have the time but for now I suggest using table additions rather than replacing db/units_tables/units (I realise this isn't ideal since it doesn't allow you to remove units).

For Crux.7z

This includes the new packed sounds I made with your tool and the core DarthMod pack which has the unit_tables.
I hope this helps.

I've not been able to reproduce your reported issue Darth. When I load without your mod Date Bow Ashigaru have 50% more range than Date Matchlock Ashigaru. When I enable your mod both have the same range. If you'll permit me to speculate, I'll guess that your user.script.txt file is not saved as Unicode and so is not read correctly by Shogun 2. This would mean that the game was always loading loose data until you renamed the db folder, at which point it was no longer loading replacement db files at all. I suspect you then re-exported the loose db files or re-named the db folder back at the same time as you deleted your preferences script and disabled the Symphony created pack, at which point everything worked again.

So, while Symphony throws no errors, the unknowns generated seem like a big issue and I get the same sound mismatch as using my test mod by itself.

I'll have to compare this to the results from running against a full FotS installation when I get the chance, but something seems wonky with the extraction in this case. Or, are the unknowns ok and my corrected pack simply not loading properly? What do you think?

Below are my Symphony output screens. I can attach my files and extraction directory if you want them. In any case, thanks very much for your tool and efforts at solving this

Your steps taken look absolutely perfect. The unknowns and reduced output relative to Empire/Napoleon are entirely to be expected. CA shipped unpacked sound files with Empire and Napoleon which made it much easier to reverse engineer the file format. Unfortunately they haven't done the same for Shogun 2 so I've only gotten around to partially handling one sound bank thus far and everything else remains unknown. I guess you're running into the same problem as Seyal since you're replacing db/units_tables/units. Could you try a table addition instead and see if Symphony works for that?

[Steph]

Removing units can be dangerous anyway, as to avoid crashes one would need to remove it from everywhere.
I think it safer to keep the unit to removed in the unit table, but then in unit_to_exclusive_faction, make it unavailable to every faction allowed to have it in the first place.
That's what I did in MOE, I have a "unit removal" table, which is actually a renamed uni_to_exclusive_faction where the units are set to false.

[Kurisu Paifuaa]

Your steps taken look absolutely perfect. The unknowns and reduced output relative to Empire/Napoleon are entirely to be expected. CA shipped unpacked sound files with Empire and Napoleon which made it much easier to reverse engineer the file format. Unfortunately they haven't done the same for Shogun 2 so I've only gotten around to partially handling one sound bank thus far and everything else remains unknown. I guess you're running into the same problem as Seyal since you're replacing db/units_tables/units. Could you try a table addition instead and see if Symphony works for that?

Yes, thanks. Working on it. Made some minor edits to units table and re-packed. I just changed cost, upkeep and recruitment time for Yari Ashigaru to verify my pack is effective. Changes show in campaign and sounds are correct (for the few units to which I have access), however, not so in custom battle where I still get fire rockets for matchlock ashigaru, possibly a rifle sound for mangonel. I've not tried a custom unit. By table addition, do you mean something like xxx_units_table with just the edited row? I'll do that next.

edit: ah, just caught that explanation in your reply to Seyal

These are all projectile fire sounds obviously... can I check those directly by extracting tables from my Symphony generated pack? I've briefly examined sound_bank_projectile_fire.xml and sound_events.csv in the vanilla extract (heh... ). Not familiar with the layout just yet, but those look relevant. Any insight or pointer to an old discussion would be much appreciated.

It would also be helpful if I could be absolutely certain that the fixed sound pack is loading. The encoding/formatting of my script seems ok as my test mod loads. Both entries are continuous strings (e.g. fixed_sounds.pack) with no quotes and a terminating ; Previously I discovered that the script doesn't seem to like mixed file name formats (like "test units.pack"; along with fixed_sounds.pack; ) which I didn't recall being the case. There I thought I had success (all sounds were correct), but it turned out the units pack wasn't loading and probably not the corrected sound pack either.

Putting the packs together didn't work before, but I'll try that again.


edit: What I'm finding, after countless trials, is that the mere presence of Symphony's output pack in the data directory stops my test mods from loading at all. Even if it is removed from the user script, it still gets referenced and mod loading seems to halt. This is true also if I run my test mod as a movie type. Otherwise, every simple test mod I've constructed works as expected, whether it's removing a unit from the base units table (or not) and adding it back via a single row supplementary table, or modifying the base table directly. I have to change Symphony's output file extension to effectively hide it. I don't know if that's revealing of anything other than possibly the sounds_packed directory may be a special case in the loading scheme.

[Seyal]

It looks like there is a bug in the way I'm handling completely replaced unit tables. I was able to reproduce your issue easily - it is indeed the Ikko Ikki Bow Ashigaru that demonstrate it (other units may do as well). If you simply rename your units table from Seyal_Shoujo_Bushi.pack to something else (i.e. db/units_tables/units to db/units_tables/units_seyal) then Symphony seems to work. I'm not sure I've ever tested a complete units table replacement before. I think I've only tested self-generated table additions and Radious unit mods, neither of which replace the db/units_tables/units file, they just add additional db files in the db/units_tables directory. I'll look into this as soon as I have the time but for now I suggest using table additions rather than replacing db/units_tables/units (I realise this isn't ideal since it doesn't allow you to remove units).

Oh I see. That is interesting.

Figures I would be doing something unaccounted for

Could I just have a units_seyal with just my 3 totally new units and mod the Warrior Nuns from the base units file, would that work just as well? I probably should just go ahead and make the Warrior Nuns for my mod with a new ID and such for simplicity's sake, though.

Thanks for looking into it. I'd give rep, but it won't let me

[crux3D]

It is encouraging that you see the unit ranges for Shogun 2 units, but have you seen for FOTS units if ranges are vanilla or modded?
I will double check if my Shogun units have the modded ranges as yours, I am not sure for that but I am sure that FotS units had range inconsistencies.

I have tested the following FotS units in custom battle (Aki), lining the units up against the back of the deployment area and measuring the firing arcs against each other and the 200m depth of the deployment area:
Unit, Projectile, Vanilla Range, Darthmod Range
Boshin_Cav_Missile_Carbine_Cavalry, boshin_carbine, 125, 175
Boshin_Cav_Missile_Revolver_Cavalry, boshin_revolver, 70, 120
Boshin_Gen_Generals_Bodyguard, boshin_officer_revolver, 70, 120
Boshin_Inf_Elite_Imperial_Guard_Infantry, boshin_breech_loading_rifle, 125, 190
Boshin_Inf_Light_Sharpshooters, boshin_sharpshooter_rifler, 150, 250
Boshin_Inf_Line_Red_Bear_Infantry, boshin_rifle, 125, 180
Boshin_Inf_Light_Tosa_Riflemen, boshin_repeating_rifle, 150, 250
Boshin_Inf_Militia_Levy_Infantry, boshin_musket, 125, 150
Boshin_Inf_Missile_Matchlock_Kachi, boshin_matchlock, 100, 150
I have both darthmod_shogun_units.pack and DMS_Sounds_Packed.pack in my data folder. With an empty user script I see vanilla ranges for all the listed units. With both pack files in my user script I see Darthmod modded ranges for all listed units.

In case this was a campaign-only issue I then tried a campaign battle with the following units:
Unit, Projectile, Vanilla Range, Darthmod Range
Boshin_Inf_Line_Infantry, boshin_rifle, 125, 180
Boshin_Inf_Militia_Levy_Infantry, boshin_musket, 125, 150
Boshin_Inf_Missile_Matchlock_Kachi, boshin_matchlock, 100, 150
Once again, with an empty user script all units had vanilla ranges and with both mod packs activated all units had Darthmod modded ranges.

There is clearly something different between our installs. I have tried running with unpacked loose data in addition to the packfiles, but that didn't make any difference. If you and/or Kurisu Paifuaa are willing to perform some tests to help me narrow down the issue I would be very grateful. Here is a list of some things that could be tried:

Running a completely vanilla install with just the two pack files Darth uploaded. Aim is to identify if some other file and/or setting is interfering.
Running with loose sound pack files from Symphony instead of packed files (create a sounds_packed folder in the data directory and pass that as the destination for Symphony, i.e. Symphony pack <shogun-2-install> <unpacked-sounds-directory> <user-script-path> shogun-2-install/data/packed_sounds). Aim is to identify if particular format of packfile is the cause.
Removing the Symphony produced packfile and extracting sounds_packed/sound_events from patch20.pack and sounds_packed/sound_bank_database from patch16.pack, repacking them into a new packfile and running the game. Aim is to identify if mere presence of additional packed sound files causes the issue.
Taking sounds_packed/sound_events from patch20.pack and sounds_packed/sound_bank_database from Symphony and vice-versa. Aim is to identify if one of the two sound files causes the issue.
Recruiting some additional testers, preferably a mix of mod-makers and mod-users, to test the packs Darth posted. Aim is to identify how widespread the problem is and whether any correlations can be observed.
I may also try and put together a tool to list and checksum all involved files and collect the preferences and user_script files, so I can try and exactly duplicate your working environments (or as close as possible). Would either of your be willing to run this tool and provide me with the output if I did so?

Oh I see. That is interesting.

Figures I would be doing something unaccounted for

Could I just have a units_seyal with just my 3 totally new units and mod the Warrior Nuns from the base units file, would that work just as well? I probably should just go ahead and make the Warrior Nuns for my mod with a new ID and such for simplicity's sake, though.

Thanks for looking into it. I'd give rep, but it won't let me

You could have a units_seyal with your 3 new units plus your modified warrior nuns. That should work just fine.

[Kurisu Paifuaa]

There is clearly something different between our installs. I have tried running with unpacked loose data in addition to the packfiles, but that didn't make any difference. If you and/or Kurisu Paifuaa are willing to perform some tests to help me narrow down the issue I would be very grateful. Here is a list of some things that could be tried:

Running a completely vanilla install with just the two pack files Darth uploaded. Aim is to identify if some other file and/or setting is interfering.
Running with loose sound pack files from Symphony instead of packed files (create a sounds_packed folder in the data directory and pass that as the destination for Symphony, i.e. Symphony pack <shogun-2-install> <unpacked-sounds-directory> <user-script-path> shogun-2-install/data/packed_sounds). Aim is to identify if particular format of packfile is the cause.
Removing the Symphony produced packfile and extracting sounds_packed/sound_events from patch20.pack and sounds_packed/sound_bank_database from patch16.pack, repacking them into a new packfile and running the game. Aim is to identify if mere presence of additional packed sound files causes the issue.
Taking sounds_packed/sound_events from patch20.pack and sounds_packed/sound_bank_database from Symphony and vice-versa. Aim is to identify if one of the two sound files causes the issue.
Recruiting some additional testers, preferably a mix of mod-makers and mod-users, to test the packs Darth posted. Aim is to identify how widespread the problem is and whether any correlations can be observed.
I may also try and put together a tool to list and checksum all involved files and collect the preferences and user_script files, so I can try and exactly duplicate your working environments (or as close as possible). Would either of your be willing to run this tool and provide me with the output if I did so?
.

Yes, I can run more tests at some point this weekend. This machine doesn't have FotS, but I can grab that from Steam at any point, unless it would be useful to keep this un-FotSed for now. The only difference I've noted between the two installations is the absence of the patch_fots packs. Those contain some fots specific audio resources (sfx and music) but no data files.

I did try running with a loose sounds_packed directory, but I extracted that from Symphony's file with PFM. I'll give the other methods you mentioned a try. Also, switching Symphony's generated pack to patch type allowed my test mod to load (which it wouldn't otherwise) but there were still audio mismatches.

Shall I stick with my one line test mod or use Darth's files?

[Seyal]

You could have a units_seyal with your 3 new units plus your modified warrior nuns. That should work just fine.

Cool deal, thanks again for your help and this utility.

[DARTH VADER]

I've not been able to reproduce your reported issue Darth. When I load without your mod Date Bow Ashigaru have 50% more range than Date Matchlock Ashigaru. When I enable your mod both have the same range. If you'll permit me to speculate, I'll guess that your user.script.txt file is not saved as Unicode and so is not read correctly by Shogun 2. This would mean that the game was always loading loose data until you renamed the db folder, at which point it was no longer loading replacement db files at all. I suspect you then re-exported the loose db files or re-named the db folder back at the same time as you deleted your preferences script and disabled the Symphony created pack, at which point everything worked again.

You think very correctly but I had already tried all that to be sure. Yes my script is saved as unicode, so no problem there. It is encouraging that you see the unit ranges for Shogun 2 units, but have you seen for FOTS units if ranges are vanilla or modded?
I will double check if my Shogun units have the modded ranges as yours, I am not sure for that but I am sure that FotS units had range inconsistencies.

[DARTH VADER]

edit: What I'm finding, after countless trials, is that the mere presence of Symphony's output pack in the data directory stops my test mods from loading at all. Even if it is removed from the user script, it still gets referenced and mod loading seems to halt. This is true also if I run my test mod as a movie type. Otherwise, every simple test mod I've constructed works as expected, whether it's removing a unit from the base units table (or not) and adding it back via a single row supplementary table, or modifying the base table directly. I have to change Symphony's output file extension to effectively hide it. I don't know if that's revealing of anything other than possibly the sounds_packed directory may be a special case in the loading scheme.

Yes exactly. I noticed the same problem.

[Kurisu Paifuaa]

Yes exactly. I noticed the same problem.

This is exactly the sort of situation where being able to turn on vfs logging would be more than a little useful

That switch in the preferences never actually worked, did it, Darth?



edit: If I change Symphony's generated pack to patch type, it no longer prevents my mod from loading. However, it still doesn't fix anything.

[DARTH VADER]

Thank you a lot Crux for your reply.

I will do a test again myself, but please all others use my files and report back if you can.
Since Crux you see the correct ranges, can you test if the sound issues are fixed?
Play some custom battles with Sengoku or Genpei era archers and hear if they sound like cannons/rifles.
Also take the Wooden Frigate FOTS and hear if there is no sound at all when it fires.
These are the basic tests I do to see if the sounds get messed.

PS. I have also some other mods enabled by movie type but have nothing to do with the unit_tables. I will re-upload the test pack in a few minutes. I will also organise a test in my forum.

EDIT:
DMS 4.1+ Test PACKS.7z
Here is the new test pack

• Copy/Paste all the included packs in the Data folder of the game e.g "C:\Program Files (x86)\Steam\steamapps\common\total war shogun 2\data"
• Follow the included instructions on how to enable the script

*WARNING*
--Be sure not to use other mods and do not use the DarthMod Launcher for this test.
--Back up your older DarthMod files if you already use DarthMod: Shogun v4.1.


*TO REVERT*
--To revert to vanilla setting delete the script and the packs you copy/pasted.
--To revert to older DarthMod Setting, delete the scripts and the packs you copy/pasted, and replace them with your backup.

[crux3D]

PS. I have also some other mods enabled by movie type but have nothing to do with the unit_tables. I will re-upload the test pack in a few minutes. I will also organise a test in my forum.

Thankyou for your help Darth. We are making progress! With your latest packfiles I am able to reproduce your issue. I don't have much time right now but I will start to investigate.

Thank you a lot Crux for your reply.

I will do a test again myself, but please all others use my files and report back if you can.
Since Crux you see the correct ranges, can you test if the sound issues are fixed?
Play some custom battles with Sengoku or Genpei era archers and hear if they sound like cannons/rifles.
Also take the Wooden Frigate FOTS and hear if there is no sound at all when it fires.
These are the basic tests I do to see if the sounds get messed.

With your previous packs I tested all bow units available to Chosokabe, Date and the first of the Gempei clans. All had correct sounds, so it seems very likely that the pack-disabling issue is the only problem needing to be solved.

Yes, I can run more tests at some point this weekend. This machine doesn't have FotS, but I can grab that from Steam at any point, unless it would be useful to keep this un-FotSed for now. The only difference I've noted between the two installations is the absence of the patch_fots packs. Those contain some fots specific audio resources (sfx and music) but no data files.

I did try running with a loose sounds_packed directory, but I extracted that from Symphony's file with PFM. I'll give the other methods you mentioned a try. Also, switching Symphony's generated pack to patch type allowed my test mod to load (which it wouldn't otherwise) but there were still audio mismatches.

Shall I stick with my one line test mod or use Darth's files?

Thankyou for your willingness to help. Given that I am now able to reproduce the issue you and Darth are experiencing it would me most helpful if you could try and identify any correlations between the lastest set of packfiles Darth uploaded and your own Shogun 2 install.

[DARTH VADER]

Oh Crux I am delighted by what you have found. Splendid! +1 (Cannot +1 more, must spread it).

[epaacccc]

I added unit_tables to darthmod 4.1 vanilla_garrison_pack because I wanted to make matchlock warrior nuns. Sound bug: yumi ashigaru with cannon sounds.

Removed all entry except the one for the warrior nuns (the one I edited); no more sound corruption. I tested custom battles with all bow units in original shogun 2, not sure about naval units yet.

[Kurisu Paifuaa]

I added unit_tables to darthmod 4.1 vanilla_garrison_pack because I wanted to make matchlock warrior nuns. Sound bug: yumi ashigaru with cannon sounds.

Removed all entry except the one for the warrior nuns (the one I edited); no more sound corruption. I tested custom battles with all bow units in original shogun 2, not sure about naval units yet.

Sometimes the problem can manifest itself very subtly. For example, with one instance of a one line units_table mod, I noticed no other errors except for mangonel missing their mechanical release noise. Easy to overlook, but definitely present and tied to my single entry addition to the units table. The bug moves around if you change the conditions and isn''t always obvious.


Regarding the presence of a Symphony generated pack causing other mods not to load: testing Darth's packs, it looks like groupformations.bin is still effective despite the db tables in the same pack getting ignored. I noticed when click dragging formation shapes, the tight spacing is consistent with his modded version.

[crux3D]

Fingers crossed, I think I've "fixed" it. I think the problem was the dependencies I was generating for the packfiles created by Symphony - or equally, the lack of dependencies in most mod packs (to be fair, it's probably harder for other tools to generate dependencies). Symphony lists all pack files it loaded data from as dependencies and it looks like mod packs are used to determine dependencies, even if they are not loaded.

This could get confusing, so for the following explanation I'm going to use darthmod_shogun.pack (a pack containing db/units_tables/Darth_units) and DMS_Sounds_Packed.pack (a packfile generated by Symphony, containing sounds_packed/sound_bank_database and sounds_packed/sound_events), replace them with whatever your particular packs are called.

For most mods DMS_Sounds_Packed.pack ends up with darthmod_shogun.pack and patch21.pack as the dependencies. When Shogun 2 loads the packfiles it sees that it has to load darthmod_shogun.pack before DMS_Sounds_Packed.pack and that it must also load patch21.pack (and all its dependencies) before DMS_Sounds_Packed.pack, regardless of whether or not it is going to load DMS_Sounds_packed.pack. What it doesn't know is the relative ordering between darthmod_shogun.pack and patch21.pack. This means that, depending on the name of the mod pack, sometime it will load patch21.pack first and darthmod_shogun.pack second, in which case everything will be fine, and sometimes it will load darthmod_shogun.pack first and patch21.pack second, in which case the contents of darthmod_shogun.pack will be overwritten by the contents of patch21.pack, and you get the result that Darth and Kurisu Paifuaa saw.

I have uploaded a new version of Symphony with the dependency generation disabled. Please give it a go and see if it fixes your problems. Please be sure to delete all previously generated sound packs first!

[The Hedge Knight]

Fingers crossed, I think I've "fixed" it. I think the problem was the dependencies I was generating for the packfiles created by Symphony - or equally, the lack of dependencies in most mod packs (to be fair, it's probably harder for other tools to generate dependencies). Symphony lists all pack files it loaded data from as dependencies and it looks like mod packs are used to determine dependencies, even if they are not loaded.

This could get confusing, so for the following explanation I'm going to use darthmod_shogun.pack (a pack containing db/units_tables/Darth_units) and DMS_Sounds_Packed.pack (a packfile generated by Symphony, containing sounds_packed/sound_bank_database and sounds_packed/sound_events), replace them with whatever your particular packs are called.

For most mods DMS_Sounds_Packed.pack ends up with darthmod_shogun.pack and patch21.pack as the dependencies. When Shogun 2 loads the packfiles it sees that it has to load darthmod_shogun.pack before DMS_Sounds_Packed.pack and that it must also load patch21.pack (and all its dependencies) before DMS_Sounds_Packed.pack, regardless of whether or not it is going to load DMS_Sounds_packed.pack. What it doesn't know is the relative ordering between darthmod_shogun.pack and patch21.pack. This means that, depending on the name of the mod pack, sometime it will load patch21.pack first and darthmod_shogun.pack second, in which case everything will be fine, and sometimes it will load darthmod_shogun.pack first and patch21.pack second, in which case the contents of darthmod_shogun.pack will be overwritten by the contents of patch21.pack, and you get the result that Darth and Kurisu Paifuaa saw.

I have uploaded a new version of Symphony with the dependency generation disabled. Please give it a go and see if it fixes your problems. Please be sure to delete all previously generated sound packs first!

Alright then, would these rather snazzy user.script commands I just help.

#this is a comment#
set_pack_file_precedence a.pack b.pack #lhs pack precedes rhs pack (lhs < rhs): lhs will be searched before rhs for files#
exclude_pack_file 1.pack #Exclude pack file from the virtual file system#
#also#
#set_pack_file_dependency <pack file name> <pack file name>#
#lhs pack is required by rhs pack (lhs < rhs): lhs will be loaded first and used by rhs pack#

set_precedence and exclude seem (very rough test) to work as expected though I must admit I didn't test set_pack_file_dependancy because I was busy and couldn't think of a simple test quick test and got bored with how slowly shogun loads.

[epaacccc]

Sometimes the problem can manifest itself very subtly. For example, with one instance of a one line units_table mod, I noticed no other errors except for mangonel missing their mechanical release noise. Easy to overlook, but definitely present and tied to my single entry addition to the units table. The bug moves around if you change the conditions and isn''t always obvious.

You are right, sir. Today I finally meet the "corrupted sound". Turns out the mori bow kobaya is broken. Actually took me multiple campaign before I see this because Mori clans gets wiped out very early most of the time. I guess it is tolerable enough as long as all the commonly used units have proper sounds. IMO, this symphony thing isn't user friendly enough for most users, so it's not the best solution.

What's with the lack of support from CA?

[The Hedge Knight]

Though Im not really 'with this' would it not be best to have symphony generate a movie pack rather than a mod pack as those always load last.

Order should be
Boot
Release
Patch(1-x)
Mod (a-z)
Movie(a-z) (In the old hot patches for big Ntw mods we named them ZZ_patch.pack and it overwrote everything important every time).

[Kurisu Paifuaa]

Though Im not really 'with this' would it not be best to have symphony generate a movie pack rather than a mod pack as those always load last.

Order should be
Boot
Release
Patch(1-x)
Mod (a-z)
Movie(a-z) (In the old hot patches for big Ntw mods we named them ZZ_patch.pack and it overwrote everything important every time).

There always seem to be some odd, ever changing exceptions to that with anything sound related that make me think the process isn't entirely linear or consistent. Though I'm sure it would make sense if we could see a vfs log.

The related oddity with the previous version of Symphony (and possibly still) was it's mod type output pack getting referenced even when not listed in the user script. As far as I know, that's alternative to the usual behavior. Is sounds_packed a special case?