Corrupted Sounds Solutions and Discussion

[crux3D]

I don't know what I've said or done to upset you daniu, but you seem determined to assume the worst about me and interpret everything I say in the worst possible light.

Well, you were asked alreay.

That was clearly an enquiry into how Symphony could be used, not how it works. I answered accordingly and I assume the user was happy with my response since they did not request further clarification. To my knowledge nobody had enquired as to the technical details before you complained that I had not provided them. Furthermore, it is my experience that people generally do not care how a tool works, as long as it does indeed work. Consequently I don't usually take the time to format and present said details unless someone requests them.

I'll hardly start calling external tools from an executable.

If you don't wish to do so then that is, of course, your perogative. I merely pointed out that is was permissible.

PFM is an application, not a script.

Applications spawning additional processes in order to perform a section of work is nothing new. Your IDE does it, your internet browser almost certainly does it, even Shogun 2 does it. As I said, it's up to you how your application behaves, but please don't insinuate that my suggestion was in some way inappropriate or unrealistic. Since you do not want to invoke an external application from PFM, would you be happier if I provided a variant of Symphony as a dll?

Sure, people are asking for the source code because they won't understand it. That statement itself is an insult to the intelligence of anyone knowing more than a single programming language. Heck, I've been going through taw's Ruby code and got along fine, and Ruby is much more fundamentally different from anything common like C or Java than C++. Maybe I'm special, but do you really think you'd have a hard time understanding how the PFM works?

It certainly wasn't intended as an insult. I know professionals who aren't up to speed with the latest C++ features and would probably stuggle with elements of my code. However, if you believe you could cope with it then fine, I will accept that.

I doubt it because my assumption is that's where you looked to see how packs are assembled.

I did not use anybody else's tools in the development of Symphony. If I had I would have asked their permission and credited them. I figured out how pack files work by reverse-engineering the files and reading information posted publicly by The Creative Assembly.

And none of the libraries you state you use is "releaseable in executable but not in source form", a licensing model I only have seen in commercial product, which I must express my doubt you use; and if you do, that does also fall under "not my idea of community effort" because I have a hard time believing you can't use an equivalent open source library to keep things open.

You're probably right, there probably isn't anything in my source that would prevent me from releasing. But you'll note I didn't say that there was. I said I would have to check. I know everything in my various libaries and code snippets are releasable in executable form because I don't add anything to my respositories that does not meet this requirement. However, this is the only thing I check. I do not know the licensing requirements for a source code release of everything in my repositories, and I'm not about to open myself to liability without thorougly checking first. In fact, having just checked the Boost license I find this:

Excerpt from Boost Software License
The copyright notices in the Software and this entire statement, including
the above license grant, this restriction and the following disclaimer,
must be included in all copies of the Software, in whole or in part, and
all derivative works of the Software, unless such copies or derivative
works are solely in the form of machine-executable object code generated by
a source language processor.

If I had released and had not checked this license I would have been breaking the terms of the Boost license.

But this must all stay guesswork on my part because again you seem reluctant to give out any concrete information.

What concrete information would you like?

Not that it really matters because like I said, if people can live with using a multitude of tools to get their mods working, that's fine by me.
It's just that it conflicts with my idea of having a single tool providing as much integrated functionality as possible.
And the idea that someone might pick up on where you stop when you do, as I had to with the PFM.

Fair enough, but I have listed my reasons for not releasing my source and while you have addressed one, the other three still remain.

Can I ask why you chose such a 'locked door' solution to an open community problem?

I don't consider it a 'locked door' solution. It is a fairly standard software license and in fact much more open than most since it permits unlimited distribution. It's not an open-source license because I am not distributing my source - for the reasons explained previously.

[daniu]

It's just that it conflicts with my idea of [...] the idea that someone might pick up on where you stop when you do

Well who would have thought.

Can someone explain to me why it is that the mod users have to run Symphony to get their sounds right?
Why don't modders run Symphony after finishing their mods, then add the files it created into their mod pack?

[Steph]

Well who would have thought.
Can someone explain to me why it is that the mod users have to run Symphony to get their sounds right?
Why don't modders run Symphony after finishing their mods, then add the files it created into their mod pack?

Because the compilation of the sounds requires a "final" list of units.
So I used symphony for the Masters of Europe mod, and it works well with all the new units I created.
But if you want to add another mod, then the list of units changes, and the compiled sound bank doesn't work.

So:
- Each modder should run symphony for their own mods, and include the resulting files in their mod. Yes, you are right on this. But it will work only of no other mods changes the list of units.
- If a mod user wants to play with several mods at the same time, then he has to install all the mods, and then run Symphony for this new configuration and compiled the sounds.

Then it leads to a second level. If mods don't change the sound banks, but just add units and use existing sounds, then this would be relatively easy.
But if a mod modify the sound banks, then you'd need to merge the sound banks from the different mods before you can compile.

[rob-a-dogg]

If you want to know how Symphony works, all you have to do is ask, although there are some details that I will not reveal due to the slightly legal grey area surrounding the methods I used to discover them. I shall post a description in the Symphony thread.

Release of source code is unlikely due to a number of issues - time to cleanup, ensuring I don't release anything I don't have the right to (i.e. third party code which is releasable in executable form but not in source form), the fact that I don't even know if anyone else here can work with dense modern c++ code, removing elements I don't want to release due to the aforementioned legal grey area.

Regarding the license, no-derivatives means you are not allowed to modify the work but you are still welcome to package it with another work (i.e. PFM) and have that work use it.

Can I ask why you chose such a 'locked door' solution to an open community problem?

the fact that I don't even know if anyone else here can work with dense modern c++ code

By this logic no one should ever learn anything, because to begin with they dont know how to do whatever it is they want to learn, and so must be too stupid to even learn it because they dont already know it this is the modding workshop forums if we are not here to learn and share what are we here for? Not for the cups of tea surely

[DARTH VADER]

Daniu, I mentioned to the other thread about tables renaming. Maybe the table must not renamed at all, it must remain vanilla. The PFM currently renames all tables to version 1, version 2, version x. This is a helping new feature but it does not do good in all situations because some tables must use the same vanilla name in order to work/replace the values of the game and not make double entries.

[Kurisu Paifuaa]

I don't think crux3D's statement about the collective knowledge of the community here is meant as a judgment or an insult. More so, it's uncertainty whether he'd be liable if he publicly posted his source in it's present state and uncertainty whether absolutely everyone who touched it would be discrete with it when releasing derivative work. Using strictly open source libraries and methods in the first place is hindsight at this point. Who really knew, several years ago, that the propensity for these audio mismatches would grow instead of being fixed internally, causing the need for a more elegant community solution?

I imagine what daniu is envisioning in this case is PFM automatically generating corrected sound bank data alongside modified units tables. That and the ability to update the routine should crux3D not have the time for regular maintenance. Yes?

BTW, has anyone had any success getting this method to work reliably with S2? Haven't had time to get beyond looking at the syntax for Symphony and pulling my hair out before discovering I needed quotes around my source and destination pathnames

[rob-a-dogg]

I don't think crux3D's statement about the collective knowledge of the community here is meant as a judgment or an insult. More so, it's uncertainty whether he'd be liable if he publicly posted his source in it's present state and uncertainty whether absolutely everyone who touched it would be discrete with it when releasing derivative work. Using strictly open source libraries and methods in the first place is hindsight at this point. Who really knew, several years ago, that the propensity for these audio mismatches would grow instead of being fixed internally, causing the need for a more elegant community solution?

As mentioned by Dainu, if there is no source no one else can continue the work. This sound issue here is irrelevant for this point. As for liability...cant really see that as being an issue in a modding community like this. Egotistical maniacs complaining can be fairly safely ignored. What can they really do?
Only reason I can think of for not going open source is being too lazy to learn what is needed to do it, or you dont actually want people to have access to your source. This is not everyones view of course, and it does nothing to reduce my respect due to those who made the tool and what the tool does.

I imagine what daniu is envisioning in this case is PFM automatically generating corrected sound bank data alongside modified units tables. That and the ability to update the routine should crux3D not have the time for regular maintenance. Yes?

This is what i understand also

BTW, has anyone had any success getting this method to work reliably with S2? Haven't had time to get beyond looking at the syntax for Symphony and pulling my hair out before discovering I needed quotes around my source and destination pathnames

Am also interested in the answer to this.

[Livebyte]

Don't be so hard on the guy. I agree, it would be really nice to expand Symphony's functionality to new tools ASAP, but I don't think complaining is the correct approach. Nor do I think is saying "You wont understand dense code."
May I humbly suggest Crux and Daniu take the techniques to PM, and someone write a nice friendly use guide for the current tools? :-)

[Akaie]

Okay. I have cracked open symphony and got this:

Spoiler Alert, click show to read: 

Don't mind my pointlessly writing the file path to "unpack_sounds". It was being a prick last night, but it seems fine now. In any case, it doesn't seem to like my custom pack . I did a quick check to see if it would work without using my custom pack, and the same thing happens.

Furthermore, how flexible is this fix? Will I have to re-do this every time I add a new unit, or even tamper with pre-existing ones in the "units" table? I read that there are annoyances with the movie pack type, which is rather depressing .

[Steph]

In the case of NTW, you need to repack the sound everytime you modify the unit table to add, remove or reorder units.

[crux3D]

Okay. I have cracked open symphony and got this:

Due to lack of time I only added support for the most recent version of the Shogun 2 units table. I'm guessing your pack file was created with an earlier table version. You could try re-exporting your table in the newest format (I presume PFM or one of taw's tools will allow you to do that), or else point me at where I can get a copy of that packfile and I'll look at adding support for that version.

[Akaie]

Due to lack of time I only added support for the most recent version of the Shogun 2 units table. I'm guessing your pack file was created with an earlier table version. You could try re-exporting your table in the newest format (I presume PFM or one of taw's tools will allow you to do that), or else point me at where I can get a copy of that packfile and I'll look at adding support for that version.

Sorry for not getting back to this very quickly .

I use a table based on the version 13 layout, which is the latest. I noticed the 'expected' number (4294770429) is lower than the number (4294901244) that my custom pack provides. That same number (4294901244) also shows up when I don't use my custom pack and in doing so leads it to take the "units" table from patch20.pack.

[Seyal]

I just experienced this issue with my Miko No Yumi and it sounds like they have hand cannons instead of bows.

So I came across this:

"This is supported by the "Add to All" and "Renumber" functions in the Edit submenu of number columns' context menu in PFM."

Where is this? I right-clicked on unit ID and didn't see this in the context menu

[DARTH VADER]

Yes the same error for me too.

An error has occured:
Database table version unknown in table C:\Program Files  (x86)\Steam\steamapps\common\total war shogun  2\data\db\units_tables\units.  
Expected 4294770429, got 4294901244

[crux3D]

I've just tested Symphony against all of the version 13 units tables shipped by CA - everything from patch15.pack to patch21.pack and they all work fine for me. Are you both extracting against the original vanilla tables? Does the same problem occur with the new patch21.pack? If so, could one of you please upload your db/units_tables/units file from patch20.pack or patch21.pack so that I can examine it please.

[DARTH VADER]

I extracted with the mod script loaded but will try without it.

[crux3D]

I think I have fixed the issue. All units_tables files shipped by CA start with magic-number, guid, magic-number, version. It appears the first magic-number and guid are optional and Pack File Manager omits them when you create a new db file. I have updated Symphony accordingly. Grab the new version from the first post in the original Symphony thread.

[Akaie]

Checking now. Thanks for the swift update .

Edit: Alright. It gets through my custom units tables without any issue now . Now it gets stuck on the sound bank:

E:\Program Files\Total War Modding Tools\symphony>symphony extract "E:\Program F
iles\Steam\steamapps\common\total war shogun 2" "C:\Users\Anton\AppData\Roaming\
The Creative Assembly\Shogun2\scripts\user.script.txt" unpack_sounds
Detected "shogun2.exe"
Loaded "sounds_packed/sound_events" from "E:\Program Files\Steam\steamapps\commo
n\total war shogun 2\data\patch20.pack"
Loading database table "db/units_tables"
        Loaded "units" from "E:\Program Files\Steam\steamapps\common\total war s
hogun 2\data\The_Sekigahara_Campaign_V0.85f_Database.pack"
        Loaded "remove_units - version 13" from "E:\Program Files\Steam\steamapp
s\common\total war shogun 2\data\The_Sekigahara_Campaign_V0.85f_Database.pack"
        Loaded "navy_units" from "E:\Program Files\Steam\steamapps\common\total
war shogun 2\data\The_Sekigahara_Campaign_V0.85f_Database.pack"
        Loaded "generals_units" from "E:\Program Files\Steam\steamapps\common\to
tal war shogun 2\data\The_Sekigahara_Campaign_V0.85f_Database.pack"
Loaded "sounds_packed/sound_bank_database" from "E:\Program Files\Steam\steamapp
s\common\total war shogun 2\data\patch16.pack"
An error has occured:
Index of type "sound bank parameter unit" out of bounds.
        Value: 340
        Maximum Allowed Value: 334
Exiting.
E:\Program Files\Total War Modding Tools\symphony>

[DARTH VADER]

My error is similar and I tried both ways (with script loaded and not loaded)

An error has occured:
Index of type "sound bank parameter unit" out of bounds.
        Value: 394
        Maximum Allowed Value: 382

[crux3D]

You need to extract against a completely vanilla install. When extracting Symphony looks up the units each sound event refers to from the units table. Since you're both running with modded data with fewer units it finds sound events which refer to non-existant units and reports an error (and even if it didn't it would associate each sound event with the wrong units). Once you've extracted the original data you can then pack it against your modded data to produce working packed sound files. If any of the unpacked sound events reference units you have removed you will have to remove those references from the unpacked data before repacking.

Also, please note that Steph reported having problems working with multiple modded unit table files in a single packfile. You may have to combine the files together into one to get Symphony to generate the correct results.