the .pack file compression

[HorseArcher]

Let's discuss this in a more relaxed environment than the general discussion forum.

RTW was compressed via the .pak compression, and I also think the RTW demo was the same.

In M2TW demo, there is barely any text file to edit (i think only 1, the config file) and all the textures/game files are compressed into one "data.pack" file.

What kind of compression is this? I have heard some games using the " .pack" compression, such as quake3 or warcraft 3, but their utilities released to unpack the M2TW data file are useless.

[DARTH VADER]

I remember myself modding Medal Of Honour (Yes too long ago) and had pack files which I opened with simple winzip. In order to re-enter the pack it had to be done manually and then to replace the respective modded files one by one.
You may have modded 1 mb of files and had to have available the whole pack (A lot of mega) to offer it to the community since it could not be copy/pasted selectively.

If this is true they decided to revenge the modders than help them.
But I believe there will be ways...if I do not retire by then.

[Sinuhet]

Ave!

After seeing on the MTW2 demo files in hexeditor I am not quite sure about one thing - is it "encrypted" by bitwise operations only (XORs, bit shifts, and such light stuff), or only compressed (have anybody see any direction in error strings etc?, I personally have not found any hint till this time if some concrete widely known compression method/library was used) or really encrypted (i.e. for us - demo TW players, who wants to use it in a span of 1 month or so, irreversibly lost piece of info)? So I would say in MTW2 case about pakcing/extracting than directly word "compression". In fact I think that the most probable variant that it is not encrypted at all, only mix of some bitwise operations and - maybe - also compression /here could one imply indirectly if I would know the method used in RTW, I think that Vercingetorix - if he would be so nice - can help us in a manner of giving more detatiled info about methods used in RTW ...).

The usage of packs extraction and then repacking of modified data is theoretically imoprtant for modding of the MTW2 after its full version releasing too. The reconstruction of archive structure doesnt apper to me unfeasible, the structure of data.pack is relatively good as for the logical relations between its respective sections. What is also very helpful is a fact that there is alot of ASCII text which is resembling to large extent the parts of files from RTW (for example the large amount of text in the start of EDU.txt file, so in many cases it is possible to state that we actually already know the "text" which was "encrypted". So the classical assumption for decoing seems to be fullfilled - with knowladge of result of encryption and coded original information there is always a way how to recognise the "coding key".

However, I am not sure if it is worth of my time to go in this direction, though. Whats the purpose for me? Yes, the demo playability would be enlarged, but what in official release will be packed in pack? I assume that the CA will let us the crucial (for modders crucial I mean) script files (and now also with xml files added) in normally moddable structure in Data folder .... Is it worth to unpack the bunch of cas files and similar stuff then, if the CA would used in this time eventually some more complicated system then previously in RTW? The key in my considerations is question of time and the eventual impact on my own gameplay and modding possiblities. And here I am very reserved, because after palying the MTW2 demo on my comp I have noticed not only load of my video card (what is clear because it is the one which is reccommnedd as the lowest possible for MTW2), however I have noticed relatively extremly loaded CPU also, comparable to the Civ4 full of buggs/memmory leaks after its release. So I am very careful in my considerations when I will would like to buy and play full version of MTW2 after its official release. And what a motivation I would have to work on something, which would prevent the technological dominance of RTW with its mature mods, when I myslef will mod and play with the highest probability the RTW 1.5 still several months at least.

However, if somebody gives me some hint (something like - hey I have see that its with xxx lib compresed, or similar "pia fraus") and it would be the way in one direction only (with obstacles, I can admit) then there is some probability that my intelectual ego will be stimulated ..... But I am too tired in my age to start go against the brick wall without some thinking before it and this even with aim to limit survival my currently preffered plaform of RTW 1.5 .... I think that alot of people is thinking in similar way right now. Is it worth of my effort to try spent my free time with investigation of packs???

Nevertheless, if someone more motivated would like to know when to start in this problem, I would recommend to him the section describing the respective files in data folder (find in you hexeditor "data " and you will be in the section which I mean, it is by chance the first with word "data" in it). After recognising the patterns in this section (they are there, and this part of fiel is quite "non-encrypted", so the start will be relatively light) you give the start position of respective script files. Then I will recommend (like above already) to start with EDU header.
This parts are already "encrypted" or "compressed" and here you will have to start the really hard investigations... What is only worse than in RTW demo is a fact that there is only one pack in MTW2, so mutual comparisons are not possible....

Take this all as my thoughts and remarks to the whole situation after my own trials last midnight after playing few times the right downloaded MTW2 demo.

Bye Sinuhet

[Professor420]

.pack files, like .pak files, are simply a very arbitrary file extension. The actual method and type of compression between .pack files for different games is completely unrelated... the file extension for proprietary file types is always arbitrary.

Likely someone will need to create an extractor for the .pack files, just as for the .pak files. Or, possibly, CA would release one (I am sure they will have learned some lessons from RTW modding community desires, but maybe we won't be so lucky).

All an extension does is tell the system/program what to do with that file. Formats such as exe tell Windows to 'execute,' jpegs and most other data files will tell the system to open the file in a certain program. M2TW engine knows how to use these .pack files, that's all it means. Completely arbitrary (.pack is a common extension because it is very informative of what the file's purpose is).

[Sinuhet]

.pack files, like .pak files, are simply a very arbitrary file extension. The actual method and type of compression between .pack files for different games is completely unrelated... the file extension for proprietary file types is always arbitrary.

Likely someone will need to create an extractor for the .pack files, just as for the .pak files. Or, possibly, CA would release one (I am sure they will have learned some lessons from RTW modding community desires, but maybe we won't be so lucky).

All an extension does is tell the system/program what to do with that file. Formats such as exe tell Windows to 'execute,' jpegs and most other data files will tell the system to open the file in a certain program. M2TW engine knows how to use these .pack files, that's all it means. Completely arbitrary (.pack is a common extension because it is very informative of what the file's purpose is).

Ave Professor420!

No doubts in this your statement. However, if some compression algortihm have been used at all, then you will need the info, which one, in some phase of your seeking with aim to extract the files. The usage of such an algorithm in proprietary archive files/game pack files is then sometimes directly dependent on the knowladge of the name (and sometimes even size) of the respective files to decompress. Example- there is used 7z compression, you know it, but before you extract the 7z archive you must know what piece of code from pack file is the "7z file" to extract. And these kind of info is very probable to find via the parts of the data.pack, which I have described in my post above. And after that you can use the command "decompress the 7z file named xxx (eventually of size yyy).

So, it appears to me that to define these pieces is not so non-feasible task. I see the problem in investigatig if 1. the compression was used at all, 2. which one method, 3. if there is used some bitwise operation for "masking" the things a bit, so then of course also to demask this formula (and this can be very time consuming task, while theoretically no obstacles for decryption are not not existing ...).

This is my reaction, which is trying to give into relation the post of both of us....

Bye Sinuhet

[Halie Satanus]

I got as far as being told its a part of a "multi volume archive, please insert a disk with the last part of the archive, or locate the archive on the fixed disk."

My guess is (and i'm no expert) that the .PACK can only be opened when the full game is installed. I tried a program called Servant Salamander, maybe someone with more knowlage of these things could open it but it was way beyond my fumblings.

[Sinuhet]

Ave ! after some small playing (with) the MTW2 Demo during this night it apperas to me now, that there is used lz32.dll library for compresion of the respective parts of data.pack. I assume on this only indirectly, I have not ried anythig till yet, this is only nmy observation. This is in congruens with my previous unsuccessfull seeking for some error messages for manipulating the comppresed files or some similar hint in MTW2 Demo files - all this is in Windows, because the lz32.dll is an integral part of both w2k and wXP. Bye Sinuhet

[HorseArcher]

I just invited Spurius to this forum. he has some info regarding the .pack files. I hope he can be helpful here.

[Spurius]

Hi all.

I think the pack file is a Granny 3D file, this is a tool made by the same guys that sell Bink and Miles, which we know are also used by CA. If you check the program description, it fits the bill quite nicely.

http://www.radgametools.com/gramain.htm

"They export all content to a standard, rigorously defined file format that can be easily read in using the Granny run-time library in a single call. Once read, the files are traversable as in-memory C structures, making data analysis and conversion a breeze."

So it would seem the demo is a kind of frozen snaphot of the game, indeed 'not indicative of the final product' or however they put in on the splash screen and in the readme.

If you check the pack for the occurrence of 'Granny', the program name appears in full.

Pretty high-tech tooly this, you could ask radtools for a copy of the SDK, but they only have like 600 paying customers for it, so they will probably keep a good eye on who can sniff at it...

[Sinuhet]

Hi all.

I think the pack file is a Granny 3D file, this is a tool made by the same guys that sell Bink and Miles, which we know are also used by CA. If you check the program description, it fits the bill quite nicely.

http://www.radgametools.com/gramain.htm

"They export all content to a standard, rigorously defined file format that can be easily read in using the Granny run-time library in a single call. Once read, the files are traversable as in-memory C structures, making data analysis and conversion a breeze."

So it would seem the demo is a kind of frozen snaphot of the game, indeed 'not indicative of the final product' or however they put in on the splash screen and in the readme.

If you check the pack for the occurrence of 'Granny', the program name appears in full.

Pretty high-tech tooly this, you could ask radtools for a copy of the SDK, but they only have like 600 paying customers for it, so they will probably keep a good eye on who can sniff at it...

Ave Spurius!

This was my idea after ssing the granny2.dll in the demo folder too. However granny is not for packing, its graphical library, similarly as bink is video library and miles is sound library. A lot of intances of miles and bink is in the pack too, so this is a bit unrelevant argument. I think that with highest probability this library has nothing to do with any manipulating with the packs, unfortunatelly.

Bye Sinuhet

[Spurius]

Yes, I see your point exactly - it's not its intended, original development-side function, like Miles or Bink but read the description further:

"While you will most likely want to take advantage of Granny's wide array of run-time library features, it is also trivial to use Granny solely as your export layer. You can read the Granny files that come out of the exporter directly, and use this data in your application without using any other features of the run-time library, or you can write a trivial preprocessor that uses Granny to extract data from the Granny files and packages them into your own proprietary format for later use by your application."

So, I think this pre-processor trick is what is essentially driving the demo, it can also incorporate non-graphical data. Compression it knows all about, ofcourse.

The fact you cannot edit any unit sizes upwards, only downward, strenghtens my idea it is only a snapshot, with exactly only in it what is needed to play those battles, and not a byte more.

Also, the empty HD space required for the full game, 11 gigs, is quite a whopping bit more than this demo, and yet it runs, so there must be some trickery afoot to make it even look like the full game, even without all that data.

So I still think its a dump file, generated with one of the myriad functions of Granny. Good way to make a demo I would say - you are not giving anything away yet.

edit: almost forgot, I also wondered why several things were NOT in the pack, if it is a 'normal' packed data file, namely sound (I really don't see why not - unless the packer CANNOT handle it) the animated cursors (interactive, also necessary during the LOADING of the menu already, so cannot be in some huge packed file) some anims (menu again? didn't unpack them yet) and one textfile, also used exclusively for the menu. So a menu plus interactive 3d movie + sound, is all we got, I think.

Nothing really moddable, I fear.

[Sinuhet]

Yes, I see your point exactly - it's not its intended, original development-side function, like Miles or Bink but read the description further:

"While you will most likely want to take advantage of Granny's wide array of run-time library features, it is also trivial to use Granny solely as your export layer. You can read the Granny files that come out of the exporter directly, and use this data in your application without using any other features of the run-time library, or you can write a trivial preprocessor that uses Granny to extract data from the Granny files and packages them into your own proprietary format for later use by your application."

So, I think this pre-processor trick is what is essentially driving the demo, it can also incorporate non-graphical data. Compression it knows all about, ofcourse.

The fact you cannot edit any unit sizes upwards, only downward, strenghtens my idea it is only a snapshot, with exactly only in it what is needed to play those battles, and not a byte more.

Also, the empty HD space required for the full game, 11 gigs, is quite a whopping bit more than this demo, and yet it runs, so there must be some trickery afoot to make it even look like the full game, even without all that data.

So I still think its a dump file, generated with one of the myriad functions of Granny. Good way to make a demo I would say - you are not giving anything away yet.

edit: almost forgot, I also wondered why several things were NOT in the pack, if it is a 'normal' packed data file, namely sound (I really don't see why not - unless the packer CANNOT handle it) the animated cursors (interactive, also necessary during the LOADING of the menu already, so cannot be in some huge packed file) some anims (menu again? didn't unpack them yet) and one textfile, also used exclusively for the menu. So a menu plus interactive 3d movie + sound, is all we got, I think.

Nothing really moddable, I fear.

Ave Spurius!

Very interesting answer, thank you and sorry my previous misunderstanding of your argument. Well, I am going to see on this granny2 stuff more deeply next week. Maybe this is the good way to be able to extract at least some of the most important text files..... I hope in this because the exe must have some way how to read the files to be able to use them. However, you are true with high probability that as for some modifying and repacking this will not be feasible.

Thanks for sharing your info on this problem and bye, Sinuhet

[Darkragnar]

Maybe you'll have more luck the new demo that comes out on the 24th , its suppose to be derived from the full game and it might be more Modable then this beta version one.
And also if Sega push's CA then they might not do a very good job of makeing the demo secure from modding.

[Shigawire]

All future modding of M2TW depends on the work you good sirs may perform in unwinding the .pack

Also it's highly optimistic to think CA will release modding-tools, but you never know. If they do release tools, you would've wasted time.

A moderator of the .org (also a member of EB) is going to visit CA Aus in a few days time, we have given him some tough questions to present - we simply can't let such a unique opportunity go to waste.

Meanwhile, it doesn't hurt to keep pondering this ole Granny thing.

[Spurius]

Wow, cool invite (although I have no clue as to hot it is downunder right now).

Pls let them fix the bug in the demo that makes the mousecursor move to the second screen if you want to rotate the camera, instead of rotating properly. I NEED my second monitor when I mod. DON'T let them migrate that particular one to the full game. Kowtow, beg, make empty threats if need be.

BTW - Granny is just a hunch, my take is that you will not see the .pack format in the full game, just the normal old .pak files. This is just for the demo so nothing leaks. They just made a kind of cache file for the game with everything in it to run two batlles, not unlike map.rwm in a way.

And since all the other stuff appears to be neccessary during the menu (including the sound ofcourse - that is why it's not packed, it plays during the menu) it becomes clear they just found a serious way of making sure noone jumps the gun on their game.

Turns out I heard right about the 11 gigs. yikes. Now where was that modswitcher link again....

[Shigawire]

BTW - Granny is just a hunch, my take is that you will not see the .pack format in the full game, just the normal old .pak files. This is just for the demo so nothing leaks. They just made a kind of cache file for the game with everything in it to run two batlles, not unlike map.rwm in a way.

Yep that makes sense. We'll see when the full game comes.. I preordered it from Amazon now. I had it preordered from Play.com, but then I heard of the second demo that will come to Amazon preorders. :p

[Sinuhet]

Ave!

Some new remarks of mine on the topic of pack extraction, take it as by-product my playing with the MTW2 demo with aim to modify the Historical battles in it, so this is not result of some serious and metodical research. Nevertheless, it seems to me now that neither granny2.dll nor lz32.dll are used for "reading" of files from the data.pack.

Btw, my information is from reports of my debugger, yes, the MTW2 Demo (i.e. the medieval.exe) is debugger-friendly !!!!, nice feature compared to RTW.exe from full game where is protection against any debugger running in the same time. The MTW2 Demo is by this quite unique source for information how the engine is working and what features are implemeted in it.

To the thing, when one retrieve debug information specific for granny2.dll there is only usage of import from this library during plaing demo with choosing one battle and then ending it and qiting the game, which is pack reading unrelated almost certainly (from top of head, something about pointers or similar "only graphical" stuff). As for lz32.dll usage with the same approach I was not able retrieve any trace of any import from lz32.dll.

So, there must be some other way how the exe is acquiring the info to the memory. The hint could be for somebody eventually the fact of existence these several strings (aproximatelly, again from head only): mounted pack, open pack, pack data(), close pack with structure of strings in manner %s,%s,%s or such. Some of the syntax of this is also accesible for reading in system.log.txt.

For example in the beginning of the logging

00:12:27.156 [system.io] [always] mounted pack packs/data.pack
bla bla bla bla
00:12:27.281 [system.io] [warning] pack open,packs/data.pack,data/text/descr_shortcuts.txt,,not found

Inferring from this, there is relatively structured amount of code in the exe for handling of pack file and its content, and some of tese routines could be theoretically speaking used or manipulated for usage of at least reading the files in data.pack file and evetually to write them in some folder evetually too (i.e. extracting the included files).

Bye Sinuhet