Symphony (sound unpacker/packer)

[crux3D]

Attached is a zip file containing symphony.exe. Symphony is a windows console application for unpacking and repacking total war sound files. It now supports the formats used in Empire: Total War and Napoleon: Total War and (to a limited extent) Total War: Shogun 2.

Usage:

symphony extract <path to root total war folder> <path to output folder>
(i.e. symphony extract "C:\Program Files\Steam\SteamApps\common\empire total war" unpacked_sounds)
Extracts sounds_packed/sound_events and sounds_packed/sound_bank_database into csv, txt and xml files and writes those files to the specified folder. You should normally run this with a vanilla install.

symphony extract <path to root total war folder> <path to user script> <path to output folder>
(i.e. symphony extract "C:\Program Files\Steam\SteamApps\common\napoleon total war" "%appdata%\The Creative Assembly\Napoleon\scripts\user.script.txt" unpacked_sounds)
Loads the mods specified in the user script and then extracts sounds_packed/sound_events and sounds_packed/sound_bank_database into csv, txt and xml files and writes those files to the specified folder. You should only extract after loading mods if you want to edit packed sounds that have been created to work with a mod.

symphony pack <path to root total war folder> <path to edited sound files> <path to output folder or pack>
(i.e. symphony pack "C:\Program Files\Steam\SteamApps\common\napoleon total war" unpacked_sounds sound_non_wavefile_data_my_unit_mod.pack)
Packs the csv, txt and xml files in the specified folder into sound_events and sound_bank_database files and writes them to the specified destination. If the destination is a folder symphony will write raw files. If the destination is a packfile which does not already exist, or is an existing packfile which does not contain any files besides sounds_packed/sound_events and sounds_packed/sound_bank_database then symphony will create a new packfile with dependencies. If the destination is a packfile which contains additional files symphony will rename it to <name>.bak and then create a new packfile which contains all of the files from that packfile, plus the new sounds_packed/sound_events and sounds_packed/sound_bank_database files. In this case the packfile dependencies will not be modified.

symphony pack <path to root total war folder> <path to edited sound files> <path to user script> <path to output folder or pack>
(i.e. symphony pack "C:\Program Files\Steam\SteamApps\common\empire total war" unpacked_sounds "%appdata%\The Creative Assembly\Empire\scripts\user.empire_script.txt" sound_non_wavefile_data_my_unit_mod.pack)
Loads the mods specified in the user script and then packs csv, txt and xml files in the manner specified above.

Symphony loads all non-mod pack files on startup, plus any mod packs specified in the user script given, if any. Normal use would be to extract with a vanilla install, optionally edit the unpacked sounds and then pack with a mod enabled. For example, to fix the sound randomization issue that occurs with unit mods you would extract with a vanilla install, enable your mod via the user script and then pack. Symphony should then produce packed sound files that reproduce the original intended sound behaviour when your mod is active. This is achieved with the following steps:

1. Disable all mods (including loose files).
2. Run: Symphony extract <path to root total war folder> <path to unpacked sound folder>
(i.e. Symphony extract "c:\Program Files\Steam\SteamApps\common\total war shogun 2" "c:\symphony")
3. Enable all mods.
4. Run: Symphony pack <path to root total war folder> <path to unpacked sound folder> <path to user script> <path to output packfile>
(i.e. Symphony pack "c:\Program Files\Steam\SteamApps\common\total war shogun 2" "c:\symphony" "%appdata%\The Creative Assembly\Shogun2\scripts\user.script.txt" "c:\Program Files\Steam\SteamApps\common\total war shogun 2\data\sounds.pack")
5. Edit your user script and add "mod sounds.pack;"

Note that if you use mods with movie-type or other non-mod type packfiles you must take care to manage them with symphony. If you have a movie-type unit mod active and extract the vanilla sound packs against it you will extract a randomized sound pack!

Symphony lists which files it is reading and writing so that you can easily confirm that the expected mod files are being read.

For a vanilla extraction you can expect output similar to:

Loaded "sounds_packed/sound_events" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\patch2.pack"
Loading database table "db/agents_tables"
        Loaded "agents" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\patch2.pack"
Loading database table "db/building_levels_tables"
        Loaded "building_levels" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\patch2.pack"
Loading database table "db/projectiles_explosions_tables"
        Loaded "projectiles_explosions" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\patch2.pack"
Loading database table "db/factions_tables"
        Loaded "factions" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\patch2.pack"
Loading database table "db/sea_surfaces_tables"
        Loaded "sea_surfaces" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\main.pack"
Loading database table "db/cultures_subcultures_tables"
        Loaded "cultures_subcultures" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\patch2.pack"
Loading database table "db/units_tables"
        Loaded "units" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\patch5.pack"
Loading database table "db/unit_category_tables"
        Loaded "unit_category" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\main.pack"
Loading database table "db/unit_class_tables"
        Loaded "unit_class" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\main.pack"
Loaded "sounds_packed/sound_bank_database" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\patch5.pack"
Wrote "unpacked_sounds\sound_categories.csv"
Wrote "unpacked_sounds\unknown_1.txt"
Wrote "unpacked_sounds\sound_events.csv"
Wrote "unpacked_sounds\ambience.csv"
Wrote "unpacked_sounds\unknown_2.txt"
Wrote "unpacked_sounds\movies.csv"
Wrote "unpacked_sounds\sound_settings.xml"
Wrote "unpacked_sounds\sound_bank_surface_impacts.xml"
Wrote "unpacked_sounds\sound_bank_projectile_fire.xml"
Wrote "unpacked_sounds\sound_bank_projectile_explosion.xml"
Wrote "unpacked_sounds\sound_bank_projectile_ground_impact.xml"
Wrote "unpacked_sounds\sound_bank_projectile_idle.xml"
Wrote "unpacked_sounds\sound_bank_region_buildings.xml"
Wrote "unpacked_sounds\sound_bank_ambient_birds.xml"
Wrote "unpacked_sounds\sound_bank_ambient_forest_birds.xml"
Wrote "unpacked_sounds\sound_bank_ambient_land.xml"
Wrote "unpacked_sounds\sound_bank_ambient_environment.xml"
Wrote "unpacked_sounds\sound_bank_building_levels.xml"
Wrote "unpacked_sounds\sound_bank_building_ambience.xml"
Wrote "unpacked_sounds\sound_bank_sea.xml"
Wrote "unpacked_sounds\sound_bank_group_movement.xml"
Wrote "unpacked_sounds\sound_bank_naval_unit_voice_events.xml"
Wrote "unpacked_sounds\sound_bank_campaign_voices.xml"
Wrote "unpacked_sounds\sound_bank_unit_voice_events.xml"
Wrote "unpacked_sounds\sound_bank_ambient_battle_forest.xml"
Wrote "unpacked_sounds\sound_bank_ambient_water_wildlife.xml"
Wrote "unpacked_sounds\sound_bank_music_state.xml"
Wrote "unpacked_sounds\sound_bank_bodyfall_light.xml"
Wrote "unpacked_sounds\sound_bank_bodyfall_heavy.xml"
Wrote "unpacked_sounds\sound_bank_man_individual_footstep_walk.xml"
Wrote "unpacked_sounds\sound_bank_man_individual_footstep_run.xml"
Wrote "unpacked_sounds\sound_bank_man_individual_footstep_turn.xml"
Wrote "unpacked_sounds\sound_bank_horse_individual_footstep_walk.xml"
Wrote "unpacked_sounds\sound_bank_camel_individual_footstep_walk.xml"
Wrote "unpacked_sounds\sound_bank_elephant_individual_footstep_walk.xml"
Done.

For a simple unit mod packing you can expect output similar to:

Loaded "output\sound_categories.csv"
Loaded "output\unknown_1.txt"
Loaded "output\sound_events.csv"
Loaded "output\ambience.csv"
Loaded "output\unknown_2.txt"
Loaded "output\movies.csv"
Loading database table "db/agents_tables"
        Loaded "agents" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\patch2.pack"
Loading database table "db/building_levels_tables"
        Loaded "building_levels" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\patch2.pack"
Loading database table "db/projectiles_explosions_tables"
        Loaded "projectiles_explosions" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\patch2.pack"
Loading database table "db/factions_tables"
        Loaded "factions" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\patch2.pack"
Loading database table "db/sea_surfaces_tables"
        Loaded "sea_surfaces" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\main.pack"
Loading database table "db/cultures_subcultures_tables"
        Loaded "cultures_subcultures" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\patch2.pack"
Loading database table "db/units_tables"
        Loaded "units" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\patch5.pack"
        Loaded "my_unit_mod" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\my_unit_mod.pack"
Loading database table "db/unit_category_tables"
        Loaded "unit_category" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\main.pack"
Loading database table "db/unit_class_tables"
        Loaded "unit_class" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\main.pack"
Loaded "output\sound_settings.xml"
Loaded "output\sound_bank_surface_impacts.xml"
Loaded "output\sound_bank_projectile_fire.xml"
Loaded "output\sound_bank_projectile_explosion.xml"
Loaded "output\sound_bank_projectile_ground_impact.xml"
Loaded "output\sound_bank_projectile_idle.xml"
Loaded "output\sound_bank_region_buildings.xml"
Loaded "output\sound_bank_ambient_birds.xml"
Loaded "output\sound_bank_ambient_forest_birds.xml"
Loaded "output\sound_bank_ambient_land.xml"
Loaded "output\sound_bank_ambient_environment.xml"
Loaded "output\sound_bank_building_levels.xml"
Loaded "output\sound_bank_building_ambience.xml"
Loaded "output\sound_bank_sea.xml"
Loaded "output\sound_bank_group_movement.xml"
Loaded "output\sound_bank_naval_unit_voice_events.xml"
Loaded "output\sound_bank_campaign_voices.xml"
Loaded "output\sound_bank_unit_voice_events.xml"
Loaded "output\sound_bank_ambient_battle_forest.xml"
Loaded "output\sound_bank_ambient_water_wildlife.xml"
Loaded "output\sound_bank_music_state.xml"
Loaded "output\sound_bank_bodyfall_light.xml"
Loaded "output\sound_bank_bodyfall_heavy.xml"
Loaded "output\sound_bank_man_individual_footstep_walk.xml"
Loaded "output\sound_bank_man_individual_footstep_run.xml"
Loaded "output\sound_bank_man_individual_footstep_turn.xml"
Loaded "output\sound_bank_horse_individual_footstep_walk.xml"
Loaded "output\sound_bank_camel_individual_footstep_walk.xml"
Loaded "output\sound_bank_elephant_individual_footstep_walk.xml"
Wrote "sound_non_wavefile_data_my_unit_mod.pack"
Done.

Notice the additional 'Loaded "my_unit_mod" from "C:\Program Files\Steam\SteamApps\common\empire total war\data\my_unit_mod.pack"' under 'Loading database table "db/units_tables"', which shows the desired mod has been loaded.

Symphony has not been heavily tested. I have also had to make a number of assumptions about how things should work. If you are having problems please post in this thread and I will attempt to resolve your issues if possible and as my time allows.

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/; or, (b) send a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain View, California 94140, USA.

Uses Boost.Array, Copyright (C) 2001-2004 Nicolai M. Josuttis
Uses Boost.LexicalCast, Copyright Kevlin Henney, 2000-2005. Copyright Alexander Nasonov, 2006-2007. Copyright (C) Antony Polukhin, 2011.
Uses Boost.Filesystem, (C) Copyright Beman Dawes, 2002-2005
Uses Boost.Integer, Copyright (C) 2001 -2009 Beman Dawes, Daryle Walker, Gennaro Prota, John Maddock
Uses Boost.Thread, Copyright (C) 2007 -8 Anthony Williams

Thanks to Steph, Seyal, Kurisu Paifuaa, DARTH VADER, Anton III and anyone else I missed for helping track down issues with Symphony and Shogun 2.

Technical information about the underlying file formats is available here.

[The Hedge Knight]

You like unlockings things . Awesome.

Got this error though




Also the usage is a bit complex, can it use unpacked files (like i have in the folder above) and create unpacked files?

[Scotland]

Ah, this is just the ticket. I eagerly await the NTW version. Good job.

Got this error though

I also got this error when attempting to extract the NTW equivalent files, I don't know if you were trying that too? However it's just like the thing for me to ask about this very file and then by coincidence someone is working on it.

My thanks to the maker again.

[crux3D]

Got this error though

It looks like you're trying to run it against a Napoleon install. At the moment symphony only supports the Empire formats.

Also the usage is a bit complex, can it use unpacked files (like i have in the folder above) and create unpacked files?

Symphony is designed to be run against a full install of a Total War game to minimise the opportunity for error. It can run be run against an unpacked install (although I haven't tested this behaviour), but it is not designed to be run against a set of data files in isolation. Symphony can output unpacked packed files (i.e. output files sound_events and sound_bank_database as individual files not in a packfile), simply specify a directory as the output path, i.e.:

symphony pack "C:\Program Files (x86)\Steam\steamapps\common\empire total war" "unpacked sounds" destination_directory

which will create files destination_directory/sound_events and destination_directory/sound_bank_database.

[The Hedge Knight]

So could i use the unpacked ntw csvs and xml from patch 5 to make sounds packed files for ntw or has the format changed?

[crux3D]

The format is different for Napoleon.

[crux3D]

Symphony now supports the formats used in Napoleon: Total War.

[The Hedge Knight]

Still getting the Vector(t) is too long error with napoleon im afraid.

windows 7 64 bit if that might make a difference.

[crux3D]

I have a suspicion as to what may be going wrong. I've uploaded a new version of symphony with some additional verification. Please give that a go and let me know the results.

[The Hedge Knight]

Got it to unpack sucessfully now, thanks.

I think the problem before might have been that i had a copy of napoleon.exe named empire.exe in my ntw install in order to perform a dbe editor work around which i then forgot about -oops.

[T.C.]

Keep getting this:



However, it works for Empire

[crux3D]

I'm guessing you have an unpacked units_tables file in your napoleon install. Turns out there was a bug with the way unpacked files were loaded. Should be fixed now.

[T.C.]

I'm guessing you have an unpacked units_tables file in your napoleon install. Turns out there was a bug with the way unpacked files were loaded. Should be fixed now.

Nice, thanks
Will you be posting your source code? Just out of interest
And any plans for more converters soon?

Forgive my complete lack of patience

[Steph]

I've been looking for this for ages!! well, perhaps not that long, but I kep pestering CA in their modding tools with that.

I need to try that as soon as I can.

[Steph]

I tried your tool with NTW, and I have some questions.
Does it handle the sounds_packed from sounds.pack, or from the patches?
There is also one in patch2, patch3, patch4 and patch5.

The file extracted with symphony, in sound_events.csv, gives lines such as

unnamed_event_374,musician_fife_drum.

And you have only one sound_events.csv file.

while if you look at patch5, it contains several sound_events.csv, including sound_events_voices_land.csv, and a sound_events_battle.csv, with a line such as

"FIFE_WESTERN_MARCHING_1","musician_fife_drum"

In other words, patch5 seems to contain already extracted xml and csv files, with named events (while your are unamed).

So

1) Does your tool support the patch packs?
2) If we want to edit them, should we only edit sound_events.csv? Should we delete it and replace it by the many sound_events_something.csv found in the patch5.pack for instance?
What should we have before we pack again to fix the voices issue?

[Scotland]

Jolly good show for the release of the NTW version.

However, I'm encountering a problem when using the files created from Symphony on my music mod for NTW. When used I don't appear to get any sound from the UI. The music and speech are there, but no button pressing noises, so to speak. All I altered was the .csv file to include more campaign map music, decrease the priority and probability of some music playing, remove the battle music loops and change the Peninsular Campaign music's file extensions from .wav to .mp3. So I'm stumped as it's nothing to do with the UI.

[crux3D]

Will you be posting your source code? Just out of interest

I don't plan to. It would take too long to clean it up to a standard I would be comfortable releasing I don't know that there are many modders here who know c++ anyway.

And any plans for more converters soon?

If I see something that looks interesting to work on, isn't being worked on by anyone (or doesn't appear to be progressing) and doesn't look like it will take too long. But to be honest symphony took a lot longer than I originally intended to spend on it and I'd like to devote some time to some other personal (and not TW-related) projects for a while.

I've been looking for this for ages!! well, perhaps not that long, but I kep pestering CA in their modding tools with that.

It was your repeated requests and Craig's response that CA wouldn't be providing tools that prompted me to start on this.

I tried your tool with NTW, and I have some questions.
Does it handle the sounds_packed from sounds.pack, or from the patches?
There is also one in patch2, patch3, patch4 and patch5.

Symphony will extract the latest files, so for vanilla Napoleon it will extract from patch5.pack.

The file extracted with symphony, in sound_events.csv, gives lines such as

unnamed_event_374,musician_fife_drum.

And you have only one sound_events.csv file.

while if you look at patch5, it contains several sound_events.csv, including sound_events_voices_land.csv, and a sound_events_battle.csv, with a line such as

"FIFE_WESTERN_MARCHING_1","musician_fife_drum"

In other words, patch5 seems to contain already extracted xml and csv files, with named events (while your are unamed).

So

1) Does your tool support the patch packs?

Yes. It will extract sounds_packed/sound_events and sounds_packed/sound_bank_database from whichever pack file it thinks the game would use (i.e. the latest patch or a mod if there is one).

2) If we want to edit them, should we only edit sound_events.csv? Should we delete it and replace it by the many sound_events_something.csv found in the patch5.pack for instance?
What should we have before we pack again to fix the voices issue?

The csv and xml files appear to be the inputs that CA used to generate sounds_packed/sound_events and sounds_packed/sound_bank_database. Symphony then tries to perform the reverse process and extract that data back into csv and xml files. Unfortunately not all of the information in the original csv and xml files is retained when they are packed, hence the loss of most of the event names and all the sound events ending up in the same file.

Theoretically you should be able to merge all the CA provided sound_events_*.csv into a single sound_events.xml and end up with a near-identical file to the one extracted by symphony (with the exception of the event names). Symphony only supports packing from a single sound_events.csv file however, so you're probably best off just editing the extracted file.

Fixing the voices issue is trivial. Simply extract the sound files from a vanilla install and then pack them again with the mod activated. Place the resulting pack file into your TW install and enable it with a mod command in your user script file.

However, I'm encountering a problem when using the files created from Symphony on my music mod for NTW. When used I don't appear to get any sound from the UI. The music and speech are there, but no button pressing noises, so to speak. All I altered was the .csv file to include more campaign map music, decrease the priority and probability of some music playing, remove the battle music loops and change the Peninsular Campaign music's file extensions from .wav to .mp3. So I'm stumped as it's nothing to do with the UI.

I'll take a look. Could you provide a copy of your modified csv file? It might help me track down the problem.

[Scotland]

Here you go. Much appreciated.

[Steph]

I tried your tool. I removed all my mods from the game, then extracted the sounds. I added a new unit in the sound_bank_unit_voice_events.xml, then put my mod back to the NTW install folder, and packed.
Then I selected my units, it never said one "infanterie suisse" (swiss infantry), so that's better. However, it always played the first sound "unit_type_selection_line_infantry_nap_01.mp3", a bit like if the other sounds from the csv file were ignored and it doesn't loop anymore.

Another question

Context:

Take the Portuguse Line Infantry in the British army (Peninusla). In the db/units table, it's Inf_Line_British_Portuguese_Infantry_Auxiliary

If I place it on a map alongside a British line infantry, and start a battle in an unmodded game, then select the units the British speaks English, the Portuguese speaks Portuguese.

In the sound_bank_unit_voice_events.xml file, this unit is referenced only in

<sound_bank_event sound_event_name="unit_voices_selected_infantry_line">

This is the generic line infantry selection event, it's not specific to the Portuguese.

and for this event, in the corresponding csv file, we find

Uk\actor_6\Unit_Type_Selection_Line_Infantry_NAP_01.mp3


Now, if we go to sounds.pack, we can listen to the sounds, and thisUk\actor_6\Unit_Type_Selection_Line_Infantry_NAP_01.mp3 is indeed the sound played for the British, but for the Portuguese, it is Po\actor_6\Unit_Type_Selection_Line_Infantry_NAP_01.mp3
So the engine can somehow replace the uk by po to play the sound in the proper language.

Now the question:

Where in the db can you specify that a unit should use portuguese, english, german or whatever language?

In the unit table, there is "voice" column, but it just says infantry_generic

[T.C.]

If I see something that looks interesting to work on, isn't being worked on by anyone (or doesn't appear to be progressing) and doesn't look like it will take too long. But to be honest symphony took a lot longer than I originally intended to spend on it and I'd like to devote some time to some other personal (and not TW-related) projects for a while.

If you ever feel like it, I'd kill for a luac decompiler updated to lua 5.1 (current is 5.0)