For Demo: UnPACKer, loc converter

[alpaca]

UnPACKer

Alright since many people seem to have problems with it I decided to create an official thread for the demo unpacker.

As the name says, this tool can be used to get at the data stored in the ETW demo's *.pack files. You can then use the unpacked files instead of the packs to run the demo with, any changes you make on them (except for those files in demo3.pak) should readily show up in the demo.

What you need to do in order to use it is:
1. Go to http://python.org, download Python 3.01 and install it (http://python.org/ftp/python/3.0.1/python-3.0.1.msi).
2. Download the attached "etw_demo_unpacker.zip", extract the contents to your demo data folder (usually something like "C:\Program Files\Steam\steamapps\common\empire total war demo\data"). You should then be set up with five new files, "etw_demo.py" and four batch files.
3. Click on "unpack_demo1.bat" or "unpack_all.bat" which will unpack the contents of demo.pack or all the packs respectively to your data directory. You should then be able to simply run the demo using the freshly unpacked files. Please note than any previously existing files are overwritten without warning!

Optionally, you can create a list of all the files contained in the packs by running "list_all.bat"

There are a few additional options for tech-savvy people:

• To unpack a specific file, use -u <file> where <file> is the name printed to the list
• You can use the -ur <pattern> option to unpack files matching a regular expression pattern (see here for the syntax), for example -ur .*lua.* would extract all files containing the string "lua" somewhere in their path
• To repack, you have to edit the file repack.xml in the archive. In this file you have to specify all the files that should be included in the rebuilt pack (it's fairly self-explanatory), then you have to run the unpacker with -r repack.xml <target> where <target> is the name of the pack file to be created. Note that the unpacker will take its indexing info from the existing pack so if you changed the name or want to repack another file than demo1.pack, use (in this example to repack demo3.pack): -p demo3.pack -r repack.xml demo3.pack_new
• You can change the output folder with the -o option

------------------------------------------------------------------------

loc converter

The loc converter (or loco for short) is a tool that allows you to convert ETW's .loc files to tab-separated .txt files that can be edited in a text editor or with a spreadsheet program like Open Office Calc or Microsoft Excel (use the Text CSV option when opening the files) and vice-versa. Instructions below:

0. Go to http://python.org, download Python 3.01 and install it (Windows installer here).
1. Download the zip package in the attachment
2. Create a working directory (for example 'C:\empire_locs') and copy the .loc files you want to work on into that directory, rename the file extension of the old files to .loc_backup
3. Extract the contents of the zip into your working directory
4. Run loc_to_txt_all.bat to convert the .loc files into text dbs with the same name
5. Make your changes
6. Run txt_to_loc_all.bat to convert the text dbs back to .loc files with the same name
7. Put the converted .loc files into the folder you copied them from

Remember to make backups of your .loc files!

[RobinCato]

Great alpaca. This will be a big help for many users. +rep

Testing it now.

[IndoPruss]

Alpaca, do you use special library to unpack the file? I'm planning to do somewhat same unpacker for future, I just want to know the algorithm to extract the file. I'm programming in C# not in Python. Thx.

[sinople]

Yeah, great

+rep


It works with Xp, for sure

[ISG01]

It seems to be working with my demo on Windows Vista 64 bits fine.

[S-te-Fan]

Jup, vista works fine.

WHY DOES IT TAKE SO LONG! :S.

[Lord Maximus]

Alpaca, do you use special library to unpack the file? I'm planning to do somewhat same unpacker for future, I just want to know the algorithm to extract the file. I'm programming in C# not in Python. Thx.

Hello IndoPruss Alpaca isn't using any special lib for this. Neither am I.
Unpacking in C# is fairly easy.

the pack file structure is very simple.

first 16 bytes. Header info (containing PACK identifier)
second 4 bytes (Int32) containing number of files.
then 4 bytes containing some unknown info.

now the list of files start with a structure like this
4 bytes (Int32) containing the size of the file. Then a varying amount of characters delimited by a zero byte.

note: Demo3.pack contains a few "references" in the filelist which have a differentstructure. they are intermixed with the standard sizes/filenames.

directly after the filelist there's the filedata in one giant block.

If you want C# upacking code just PM and/or email me. I'll get you the working code I've written so far.

I'm planning on making it a publicly available class library so other developers can use it too.

[alpaca]

Hello IndoPruss Alpaca isn't using any special lib for this. Neither am I.
Unpacking in C# is fairly easy.

the pack file structure is very simple.

first 16 bytes. Header info (containing PACK identifier)
second 4 bytes (Int32) containing number of files.
then 4 bytes containing some unknown info.

now the list of files start with a structure like this
4 bytes (Int32) containing the size of the file. Then a varying amount of characters delimited by a zero byte.

note: Demo3.pack contains a few "references" in the filelist which have a differentstructure. they are intermixed with the standard sizes/filenames.

directly after the filelist there's the filedata in one giant block.

If you want C# upacking code just PM and/or email me. I'll get you the working code I've written so far.

I'm planning on making it a publicly available class library so other developers can use it too.

Yes this is correct, as I said before writing the unpacker the file structure is very simple. The four bytes at the beginning that you don't know is the length of the file list by the way. Do you have any idea about the references to the sound pack in demo3.pack? So far I'm just copying them from the old pack.

Edit: @stefaneke: It takes so long because it has to create thousands of files and folders containing more than 2 gigs of data that has to be copied. What did you expect?

[just]

To update that information:
Bytes 0-3 are the identifier "PFH0".
Bytes 4-7 are a 32 bit long with the pack type.
- 0 = boot pack, only one boot pack allowed, it can't have dependencies on other packs, contains stuff necessary for the game start
- 1 = release pack, for regular game packs
- 2 = patch pack
- 4 = bink/movie pack, for movies
- ? mod pack, possibly type 3.
Patch packs overwrite files that already exist in packs of type 1.
Mods don't overwrite files/settings, the final game probably contains a mod manager, where you have to activate your mods, but the demo doesn't have this.

packs can have two settings, lhs and rhs, lhs files have precedence over rhs files, I don't know what the difference between lhs and rhs is

Bytes 8-15 are unknown (all null bytes)
Bytes 16-19 are a long with the number of files
Bytes 20-23 are a long with the size of the list of sizes and names
Starting from byte 24 is the list of sizes (4 byte long) and names (asciiz) of the files in the pack.
File contents start immediately after the list (offset is also list size + 24)

All longs are little endian.

[alpaca]

I think movie is 4, otherwise great work

[Lord Maximus]

Nice find Just. Thats very usefull.

About the references. I've been playing arround with them. It looks like they are referencing a few different files from one. (the files themselves are numbered: growth21, 22, 23 & 24) Some sort of Hybrid wave file.
Fancy name for a few different versions for the same basic sound so it doesn't sound the same all the time maybe?

The structure of the complete reference link is this: (the complete reference link is delimited by 0x00)

filesize (4 bytes) = 0
filename. not delimited by 0x00 but by 0x20h but followed by 2 ASCII numbers. The first being the offset in the other pack_file (a guess realy) delimited again by 0x20h the second being the size. delimited again by 0x20h (If you substract one offset from the other you get a number which corresponds to the size). Now there's some sort of identiefier 0x3b + 0x0a followed by command add_file_from_pack delimited by 0x20 followed by the packfilename delimited by 0x20 followed by the filename in the other pack file. (the offset + size + identifier + command + filename is repeated four times. and finaly there's that 0x3b + 0x0a identifier again followed by two commands build hybrid (separated by 0x20) and finalized by that identifier again 0x3b + 0x0a and it al ends with a 0x00 byte.

Logicaly there's nothing to extract here because the actual files are somewhere else (sound_sfx.pack, is this missing pack causing all the missing sound problems?) But this hybrid reference does raise some questions.

There are some other references in the packs too. But they seem to be regular one on one references.

[alpaca]

Updated with a .loc converter

[General Gordon]

Thanks, the unpacker worked.

[Augustus Lucifer]

You sure get busy fast. Great to see your work already opening doors.

[just]

Updated with a bit of pack type 2 info.

There is now enough info to create and distribute pack files that mod the game, and don't require the end user to unpack and delete his packs.

[Rorarii]

Thanks Alpaca and others, just loaded the Loco but was wondering what game attributes do LOC files cover, i'm trying to get mental picture between ETW and M2TW modding.

But this is only a demo, I 'll probably wait for the game to see the actual directory structure, and where everything is kept.

[oliphaunt119911]

Oh, alpaca, YOU ARE A GOD!! THANK YOU!!

[LtChambers]

As far as I can tell, all the game rules are in demodb, not the loc files. Loc is just for string language localization unless I'm missing something?

An important question for just about the patch type of pack: does it do a diff for the contained data, or does it simply replace files according to load order? I.e. if two patch packs modify the unit_stats_land db file in different ways (assuming the changes don't overlap bytes), will the patch system merge them?

[Vincenz0]

I'm quite interested in the C# code to pack/unpack.

Maybe put it in SVN or something.

[IndoPruss]

Nice find Just. Thats very usefull.

About the references. I've been playing arround with them. It looks like they are referencing a few different files from one. (the files themselves are numbered: growth21, 22, 23 & 24) Some sort of Hybrid wave file.
Fancy name for a few different versions for the same basic sound so it doesn't sound the same all the time maybe?

The structure of the complete reference link is this: (the complete reference link is delimited by 0x00)

filesize (4 bytes) = 0
filename. not delimited by 0x00 but by 0x20h but followed by 2 ASCII numbers. The first being the offset in the other pack_file (a guess realy) delimited again by 0x20h the second being the size. delimited again by 0x20h (If you substract one offset from the other you get a number which corresponds to the size). Now there's some sort of identiefier 0x3b + 0x0a followed by command add_file_from_pack delimited by 0x20 followed by the packfilename delimited by 0x20 followed by the filename in the other pack file. (the offset + size + identifier + command + filename is repeated four times. and finaly there's that 0x3b + 0x0a identifier again followed by two commands build hybrid (separated by 0x20) and finalized by that identifier again 0x3b + 0x0a and it al ends with a 0x00 byte.

Logicaly there's nothing to extract here because the actual files are somewhere else (sound_sfx.pack, is this missing pack causing all the missing sound problems?) But this hybrid reference does raise some questions.

There are some other references in the packs too. But they seem to be regular one on one references.

Have you received my PM?