M2TW UNPACKER ("alternate" / "homebrew" unpacker )

[Hexdragon]

M2TW UNPACKER ( "alternate" / "homebrew" unpacker )

Download link:
https://www.twcenter.net/forums/down...o=file&id=4609


1.-Description:
============
This tool unpacks the content of Medieval 2 TW Pack files located in the Pack folder.



2.-Downloadable Zip Content:
=======================
- m2tw_unpacker.py >>> This is the thing you will execute. It opens and reads the pack file, calls the decompressor module (for unpack job) and writes the unpaked assets.
- decompressor_01.py >>> This is the thing that gets the unpack work done, behind the scenes
- decompressor_02.py >>> Also, the thing that gets unpack work done, behind the scenes ( it is nice to have two of them !!).
- m2tw_unpacker_config.xml >>> This is your "user interface"
- Readme.txt >>> Read the Readme !!!



3.- Requirements:
================


-Python 2.7
This tool has been coded and tested with Python 2.7.18.
I believe it may work with ealier versions like python 2.6.8 but cannot confirm at the moment.


-Notepad++
Or anything else that allows you to edit xml files.



4.- Basic Use:
============

4.1- Put the zip content and the pack file you want to unpack in the same folder.
4.2 - Open m2tw_unpacker_config.xml and edit the parameter "pack file name" (put your pack file's name there).
4.3 - Execute m2tw_unpacker.py by either:
a) Double click, or
b) Right click + "Edit with IDLE" + "Run" (Idle's menu option) + "Run Module"

4.4 - Upon execution, the tool will display current values of parameters located in configuration file ( m2tw_unpacker_config.xml )
Review those values and then press "Enter" key.
4.5 - Furthermore, the tool will confirm your pack file was found and read. Press "Enter" key once more in order to start unpack job.



5.-Configuration Options: ( m2tw_unpacker_config.xml )
====================


5.1- Pack File Name:
Name of the pack file to be unpacked (".pack" must be included in name).


5.2- File Path String:
- This is a "search key" string meant to be used as file filter.
- Any file within the pack whose file path contains this string will be selected for unpack.
- The default value is "data" (without the quotes), with this value, every file within the pack will be selected and unpacked.
- You can edit this parameter in order to unpack a specific filetype, a specific folder or even a single file.


5.3- Execution Mode:
If execution mode = 0: The tool will not unpack anything. It will create a txt file instead. Such file will contain a list of file paths of every file selected with "File Path String".
If execution mode = 1: The tool will unpack every file selected with "File Path String" using decompressor_01.py. Folder structure will be kept. Unpacked data will be placed in "unpacked_1".
If execution mode = 2: The tool will unpack every file selected with "File Path String" using decompressor_02.py. Folder structure will be kept. Unpacked data will be placed in "unpacked_2".
If execution mode = 3 : The tool will unpack every file selected with "File Path String" using both decompressor_01.py and decompressor_02.py, hence two instances of every unpacked file will be created.


5.4- Verbosity:
Valid values 0, 1, 2, or 3. (recommended value: 1).


--------
The xml file looks like this:

<?xml version="0.135" encoding="utf-8"?>
<RootElementOfAnyName>
... <pack_file_name>data_0.pack</pack_file_name> >>> Here is where you put your pack file's name
... <file_path_string>data</file_path_string>
... <execution_mode>2</execution_mode>
... <verbosity>1</verbosity>
</RootElementOfAnyName>

--------



6.- Notes on testing:
================


6.1 Testing Results:
- Testing was executed on M2TW and Kingdoms pack files (retail version, with patch 1.5). / PC with Core-i5 2300 @ 2.80 Ghz, 16Gb Ram / Windows 7 64-bit
- For testing purposes, the tool unpacked every single file from the above mentioned pack files (54,849 files)
- Every unpacked file was compared with its equivalent file unpacked with CA's unpacker tool (binary comparison)
- All files were identical to the last bit (verified for both Execution_Mode = 1 and Execution_Mode = 2)


6.2 Execution Times (for wholesale unpacks):
- Kingdoms data packs (425mb to 600mb) took between 7 and 15 minutes to unpack (each one)
- Vanilla's M2TW data_0.pack and data_3.packs (966mb and 845mb) took between 16 and 20 minutes.
- Vanilla's M2TW data_1.pack and data_2.packs (693mb and 866mb) took between 40 and 45 minutes.
Increased times for data_1.pack and data_2.pack seem to be related to the presence of many large files (.world among others) that unpack into "settlements" folder
- Execution_Mode = 1 and Execution_Mode = 2 brings no meaningfull differences in execution time




7.- Other Notes:
============

7.1 Sources:

The primary source used to code the decompressor modules was the document: "LZO stream format as understood by Linux’s LZO decompressor". Available at:
http://www.infradead.org/~mchehab/ke...aging/lzo.html
https://www.kernel.org/doc/html/latest/staging/lzo.html


An additional resource was the source code for miniLZO ( a subset of the LZO real-time data compression library).
Found here: http://www.oberhumer.com/opensource/lzo/


-Code in decompressor_02.py relates better to the LZO/Linux document
-Code in decompressor_01.py keeps more resemblance with miniLZO library source code

A such, both files (but decompressor_01.py in particular) may be considered as partial ports of the miniLZO library to python, both featuring the minimum content required for the unpack job on M2TW pack files.
In concordance with the above statement, this tool is distributed under the terms of the GNU General Public License that apply to the aforementioned library.


7.2 Future Plans

Short term:
-Minor tweaks and bugfixes.
-Add comments here and there.

Medium Term:
- Replace xml file with a GUI (Open file dialog, filepath list etc)
- Improve file filter (more filepath string keys, inverse selection)
- Merge M2TW Unpacker with my old packfile tool "PackFile Slicer"

Long Term:
- Try to code a custom compressor module, compatible with both M2TW engine and Unpacker
- Merge such compressor with my old packfile tool "PackFile Builder"



8.- Terms of Use:
============
M2TW Unpacker is free and distributed with no warranty. Make use of this tool at your own risk.
I am hereby not responsible for any consequence derived from the use of this tool, either if your files get corrupted, your pc explodes, your sweetheart cheats on you, or twitter suspends permanently your account.


----


M2TW Unpacker. is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 2 of the License, or
(at your option) any later version.


M2TW Unpacker. is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details at:


https://www.gnu.org/licenses/old-licenses/gpl-2.0.txt
https://www.gnu.org/licenses/gpl-3.0.txt


----


Have fun
Hexdragon (George586)

[Hexdragon]

03-24-2021 Update:
---------------------------
-decompressor_01.py :
Replaced nested loops with single one, in order to improve code readability (and bugfix unrelated to M2TW Unpacker). No functional change related to M2TW Unpacker.
-decompressor_02.py :
Removed non required bits of code. No functional change related to M2TW Unpacker.

[Hexdragon]

04-20-2021 Update:
-----------------------
Uploaded two auxiliary python tools meant for test and research

"M2TW One-File Packer":
This tool reads a source file and writes a compressed one, as either M2TW's .PACK format or LZOP's .LZO format (*)
Note: It does the work, but is (very) slow.

"LZO Auxilary Decompressor":
This tool reads a compressed source file with LZOP's .LZO format (*) and writes a decompressed one.

Download link:
https://www.twcenter.net/forums/down...o=file&id=4616

(*) About LZO File format
-LZO is the native format of LZOP (Lempel-Ziv-Oberhumer Packer), command-line compression program.
-LZOP uses the LZO data compression library for compression / decompression services (this is the compression library used in M2TW pack files).
-LZOP's binary executable (version 1.03) and source code (version 1.04) can be dowloaded from its homepage: https://www.lzop.org/


--------


Pack Files created with "M2TW One-File Packer" can be decompressed with:
- CA's M2TW Unpacker
- M2TW "alternate" Unpacker (python tool)


Lzo Files created with "M2TW One-File Packer" can be decompressed with:
- LZOP tool
- LZO Auxilary Decompressor (pyhton Tool)

Lzo Files created with with LZOP can be decompressed with:
- LZOP tool
- LZO Auxilary Decompressor (pyhton Tool) (*)
(*) As long as LZOP compression was not executed with command line options that change the .lzo standard header size (like --no-checksum or --filter)


--------


More on M2TW's PACK vs LZOP's .LZO:
------------------------------------------------

M2TW's pack files have compressed data split in chunks whose uncompressed size is 65536 bytes
LZOP's lzo files have compressed data split in chunks whose uncompressed size is 65536 * 4 bytes (262144 bytes)

According to test results, CA's M2TW Unpacker tool cannot unpack .PACK files with compressed data split in chunks of size 262144 bytes
By extension it is likely that M2TW engine cannot read such files. Thus, the option for compression with those large chunks would have no practical use regarding M2TW packs.

The above would also limit the usefulness of LZOP as a resource to get compressed data that could be later put inside a M2TW pack file
Fortunately a little cute "adjustment" comes handy:
-By opening lzop.exe with hex editor we can change byte located at 0x0001969E form 0x04 to 0x01 (**)
-With the above change LZOP can create compressed lzo files with compressed data split in chunks of (uncompressed) size 65536 bytes
-With the help of another (simple to do, but yet to be done) tool such compressed files could be further converted / added to M2TW pack files.

(**)
-Apply to LZOP 1.03 Win32 console version
-Aside from some succesfull test results, I cannot warrant (for obvious reasons) the absolute absence of side effects. Make the descibed change at your own risk.
-The C-language source code of LZOP tool is available at its home page. Hence above described "adjustment" would not be required if some wise user compile a modified version.


--------


LZO file format structure:
--------------------------------
File Header
Compressed file chunk info (12 bytes)
Compressed file chunk data
Compressed file chunk info (12 bytes)
Compressed file chunk data
Compressed file chunk info (12 bytes)
Compressed file chunk data
..
..
Compressed file chunk info (12 bytes)
Compressed file chunk data
Trailing bytes (4 zeroed bytes)


File Header:
----------------
- File signature (magic number): 9 Bytes ;0x89, 0x4c, 0x5a, 0x4f, 0x00, 0x0d, 0x0a, 0x1a, 0x0a
- Tool and library version:6 bytes (3 x 2-byte version IDs)
- Decompression method and level: 2 bytes
- Operating System Identidier: 1 byte ; 0x0E >>> os: Win32
- Command line option flags: 4 bytes
- Unknown 01: 3 bytes
- Unknown 02: 8 bytes (possibly two 4-byte time stamps)
- Source_file_name length: 1 byte
- Source file name: "N" bytes
- File header checksum (*): 4 bytes: ADLER32 (default) or CRC32

(*) Does not include file signature for checksum

File Header Size = 34 + N + 4

Compressed file chunk info:
--------------------------------
-Uncompressed file chunk data length: 4 bytes ; Big endian
-Compresseed file chunk data length: 4 bytes ; Big endian
-Uncompressed data checksum: 4 bytes: ADLER32 (default) or CRC32


--------