File/Mod Compression Tutorial!

**DaDump** · February 26, 2014, 12:01 PM

What: How to properly compress files

Why: Because properly compressing a file (or mod) will make uploads and downloads significantly faster, while being less demanding on servers. The cons? It will take longer decompressing the file/mod, but at least for me, what takes longest by far is to download mods, so in overall there's only benefits.

To begin with, here's some data to show how much of a difference file compression can do:

Name of Mod
Size of mod when installed (versions as of the date of this post (February 2014))
Size of uploaded mod
Size of mod when 'reziped' (my compression settings are shown later on)
Reziped ratio: multiply it by 100 and you have the percentage size of the 'reziped' in relation to the 'uploaded'

Spoiler Alert, click show to read:

Notes on the above list:

Spoiler Alert, click show to read:

Compressing the Files

Software used: 7Zip

How to:
-Right click a folder/file, on the 7Zip menu select "Add to Archive…"
-Now you can customize the compression of the folder/file

I use:
-Archive Format: "7z", because it allows some options that are not available for other formats (I believe)
-Compression level: "Ultra", we wanna compress it damn it! :p
-Compression method: "LZMA2", you can try out the other ones, but this one seems to be the one with better compression levels and a good performance
-Dictionary size: "1024Mb" - this also determines how well compressed it will be. It influences the RAM needed though (and thus time for compression). The benefit by increment in size decreases as the size increases.
-Word size: "273" - this doesn't affect the RAM at all, so I guess a higher value will just make the compression take longer (but better).
-Solid Block Size: "Solid" - I do not advise the use of "Non-Solid". You can split the result in chunks here, which is good if you want to upload smaller files (but more of them though).
-Number of CPU threads: "3" - The more the faster (assuming you have enough threads. To know the number of threads open the task manager(varies with the version you have), or search for your processor). However, it also influences the RAM needed, so a balance needs to be found.

"Memory usage for Compressing:" With my settings, it's almost 11Gb, but I have 16Gb of RAM, which is awesome :p (just because I could, and expect to use it later on, but haven't yet (only for compressing)) Just play around 'till you find settings that suit your specs.
"Memory usage for Decompressing:" mine says '1026Mb', but I don't think it is a problem for anyone, since any system with 2/3 Gb of RAM should only suffer a small slowdown.

Please note that the above is mainly based on experience, so mistakes may have been made.

That’s All Folks!!!
DD

**DaDump** · February 26, 2014, 12:03 PM

(reserved just in case)

"spoiler" thingy test:

[CONTENTBOX]contentbox?[/CONTENTBOX]

[FIELDSET2]fieldset2??[/FIELDSET2]

Spoiler Alert, click show to read:

**Tellos Athenaios** · February 28, 2014, 05:56 PM

If you are going to compress your mod to cut down the file size, why not use the PACK file format? That's the native 'compressed archive' format M2TW knows and understands, so people downloading your mod should not have to decompress anything.

**DaDump** · March 02, 2014, 02:35 PM

In short, the goal here is to minimize the transfer size of the mod, meaning that you'll be able to download/upload faster, and not the size of the mod itself when installed (although that obviously helps as well).

I had not thought about using the .Pack, so good point. Thus, I went straight ahead into the vanilla "packs" folder, selected the 5 main .pack files (data_x.pack), and compressed them. Result:
.pack files = 3.61Gb
compressed as shown in tutorial: 1.55Gb
1.55/3.61=0.429 !!!!

Conclusion: Even if you decide to make the mod a .pack, it is worth compressing it for upload/download purposes.
I do not know how much compressed .pack is in relation to 'not-.pack', nor how the compression of the 'not-.pack' would relate to the compression of the .pack.

,DD

**Withwnar** · March 03, 2014, 09:18 PM

Interesting. I tried compressing a few packs with just basic zipping, using 7-zip's "add to xxx.zip" on its Explorer popup menu: about 88% of the original size. Not huge but still more than I was expecting.

Why: Because properly compressing a file (or mod) will make uploads and downloads significantly faster, while being less demanding on servers.

And save room on the hard drive of anybody who keeps that downloaded file.

Thanks for the info.

**Tellos Athenaios** · March 06, 2014, 08:55 AM

Originally Posted by DaDump

In short, the goal here is to minimize the transfer size of the mod, meaning that you'll be able to download/upload faster, and not the size of the mod itself when installed (although that obviously helps as well).

I had not thought about using the .Pack, so good point. Thus, I went straight ahead into the vanilla "packs" folder, selected the 5 main .pack files (data_x.pack), and compressed them. Result:
.pack files = 3.61Gb
compressed as shown in tutorial: 1.55Gb
1.55/3.61=0.429 !!!!

Conclusion: Even if you decide to make the mod a .pack, it is worth compressing it for upload/download purposes.
I do not know how much compressed .pack is in relation to 'not-.pack', nor how the compression of the 'not-.pack' would relate to the compression of the .pack.

,DD

Originally Posted by Withwnar

Interesting. I tried compressing a few packs with just basic zipping, using 7-zip's "add to xxx.zip" on its Explorer popup menu: about 88% of the original size. Not huge but still more than I was expecting.

And save room on the hard drive of anybody who keeps that downloaded file.

Thanks for the info.

PACK files use LZO compression. The idea behind LZO compression is to be fast, faster than reading uncompressed data from disk and require little additional memory. It's not meant to get the file size down the absolute lowest possible sizes.

Going the extra mile and compressing the packs can bring the file size down, if only because there is bound to be a lot of redundancy due to how the PACK file format works. (Each file in the archive is compressed separately.)

**DaDump** · March 09, 2014, 05:40 PM

Originally Posted by Tellos Athenaios

PACK files use LZO compression. The idea behind LZO compression is to be fast, faster than reading uncompressed data from disk and require little additional memory. It's not meant to get the file size down the absolute lowest possible sizes.

Going the extra mile and compressing the packs can bring the file size down, if only because there is bound to be a lot of redundancy due to how the PACK file format works. (Each file in the archive is compressed separately.)

so "packing" a mod would make the game/mod faster in relation to loading times and disc access?
...why are then all MTW2 mods I've downloaded so far not "packed"?

**irishron** · March 09, 2014, 05:45 PM

The main reason why I have not tried is in RTW/BI, you could have only one modified pack recognized by the game engine and the size was limited. If this got reworked by CA, news to me.

**Tellos Athenaios** · March 10, 2014, 12:42 AM

Originally Posted by irishron

The main reason why I have not tried is in RTW/BI, you could have only one modified pack recognized by the game engine and the size was limited. If this got reworked by CA, news to me.

The two file formats are not at all the same. In case you didn't know, all Kingdoms campaigns are implemented as mod foldered "mods" ... using their own PACK files.

**Tellos Athenaios** · March 10, 2014, 12:44 AM

Originally Posted by DaDump

so "packing" a mod would make the game/mod faster in relation to loading times and disc access?
...why are then all MTW2 mods I've downloaded so far not "packed"?

Note that the OS maintains caches for the file system anyway, so at some point lots of the assets probably wind up being available in memory anyways.

That said the answer is probably still 'yes', provided that io.file_first is turned off. Turning io.file_first off allows the game to potentially skip a lot of file system queries (which are relatively expensive, especially if you do not need/want them to happen).

**irishron** · March 10, 2014, 04:16 AM

Originally Posted by Tellos Athenaios

The two file formats are not at all the same. In case you didn't know, all Kingdoms campaigns are implemented as mod foldered "mods" ... using their own PACK files.

Even though I have not done any research on this myself, I do see the logic behind it. Would you mind doing a tutorial on this?

**Tellos Athenaios** · March 15, 2014, 07:15 PM

Originally Posted by irishron

Even though I have not done any research on this myself, I do see the logic behind it. Would you mind doing a tutorial on this?

Would need to set up a 'clean' environment to do so, or else I'm bound to omit things. Maybe once I've cleared my todo list or at least whittled it down to not irresponsibly huge.

For now, off the top of my head the following applies:

CA's Kingdoms campaigns are implemented as mods.
Specifically: they consist of PACK files in mods/$mod/packs and necessary IDX/DAT files to get sounds working in mods/$mod/data/sounds and finally animation and skeleton IDX/DAT files in mods/$mod/data/animations
Makes sense to split a mod up in multiple PACK files, so you can keep patch sizes down if you only address a particular feature or swap things in/out easily. For example CA puts all their in game descriptions and their fonts in the localized.pack file, so the difference between Italian and English versions of the game is a simple substitution of the correct localized.pack file.
You probably should disable io.file_first.
Paths in the CA PACK files of the Kingdoms campaigns start with the mods/ directory itself, which is to say paths should probably be relative to the 'main' M2TW directory.
And you can use my PACK tool to create PACK (s). Use relative paths from the appropriate working directory to get the correct packed paths. So to pack your EDU you must pack mods/$mod/data/export_descr_unit.txt.

**Vipman** · April 10, 2014, 02:20 AM

Originally Posted by DaDump

so "packing" a mod would make the game/mod faster in relation to loading times and disc access?
...why are then all MTW2 mods I've downloaded so far not "packed"?

I really wonder how big a difference in performance would be in big mods if they were packed, maybe enough to let more people play easier due to faster loading and less lag?