PackFile Tools: Packfile-Builder (re-packer)

[Hexdragon]

Finally, getting some progress with the "how to re-pack" thing.

I have attached a homemade packfile. It works with "retrofit" mod.

Repacked files are not compressed but the game reads them from inside the packs.

If research goes fine I hope to re-start work on "dummy-packer" (repacker phyton script) as soon as possible


====

09-Jun-2014
Edit:

I have uploaded a minor update for the tool set: Version 1.03
Here: (former link for version 1.01)
http://www.twcenter.net/forums/downl...o=file&id=2094

====

[Hexdragon]

Uploaded here:
http://www.twcenter.net/forums/downl...o=file&id=2094

Packfile-Builder 1.01 (aka dummy-packer)
================

What it does:
Well, It builds Packs. It is a packer / repacker
I mean, It can put 90% to 100% of your modded gamefiles inside a packfile which can be read
by the game from ../mods/your_mod_name_here/packs.
So you can release your mod "packed".
Also you can use 2, 3 or more custom packs in your mod folder to set up easy removable mini-mods (*).

(*) IF that is the case, remember that between 2 versions of the same file, the game takes the one inside
the first pack the game reads (as far as I have tested). However, If you are using a mod folder you don't
need to worry about the game reading packs inside the vanilla "packs" folder before your packs.

The files inside the packs built with this script are not compressed, but the game can reads them.

How to use:
==========

1.- Download and install Python 2.5 from http://www.python.org/download/ (Python 2.5.2 Windows installer in most cases)

2.- Put these files inside the main Medieval II Totalwar folder
Packfile-builder.py
Pack_contents_epsilon.txt

3.- Put the files you want to repack inside ../mods/your_mod_name_here/data/.. etc etc (*)
(*) The placement of each file must be the same shown inside pack_contents_epsilon.txt

4.- Run the script
Output: "data_epsilon.pack" <<<<<<<<<< this is your new pack (rename as you wish).

5.- If you want to unpack your file to view what you got inside, don't forget
to add or change the "base_pack_path" command line in the unpacker's batch file:
--base_pack_path=mods/your_mod_folder_name_here/data/

Delay:
If new pack has +/-350 files, output should delay less than 1 minute
IF new pack has +/-2500 files, output could delay up to 12 minutes

Tested with:
Windows XP
Core 2 Duo, 2.4 GHz
2Gb RAM

Repacking more than 2500 files at once is not recommended.

Pack_contents_epsilon.txt
=========================
This file is a list of folder path (starting at "mods/your_mod_name_here/data..") of every file which goes to be repacked

Is the same kind of list you get by running M2TW Unpacker ("pack_contents.txt")
with command line --mode=1

(*) The placement of each file inside ../mods/your_mod_name_here/data/.. etc etc must be the same shown inside this file.

You will find a tool to make file list here
http://www.twcenter.net/forums/showthread.php?t=111055
(I have not tried it)

In any case remember:
- do not write blank spaces at the end of the lines
- make sure you have not inserted blank spaces below the last folder path
- make sure cursor can go down one (only one) line below the last folder path, just do not insert blank spaces

I'm not sure if alphabetic order is a must. Some testing suggested that.

=====
A sample pack_contents_epsilon.txt and sample minimod to be repacked are provided inside the downloadable zip
They work along with retrofit mod.
To test them:
- Put the minimod's files inside ..mods/retrofit/ etc etc
- Put the sample pack_contents_epsilon.txt inside the main Medieval II Totalwar folder
- Run the script (build the pack)
- Put the new built pack in ..mods/retrofit/packs/
- Start a custom battle using retrofit mod
- choose england as your faction
- recruit merchant cavalry militia and take a look on it.
=====

Update History:

30-08-2008 V1.01
Changed the way on which the script writtes Pack's header (no visible change for users)

28-08-2008 V1.00 beta
First release


enjoy :original:

Jorge35 / Hex35

[unDa]

ok. i will test

[alpaca]

Interesting. Did you make any performance comparisons?

[Hexdragon]

If I understand correctly, you want to know if I tested how fast the game runs using the new built packs. (vs original packs and vs unpacked)

To date, I have not carried out those comparisons.
In fact I'm no sure about how to set up a serious test
(any help is welcome here)

I've only ran binary comparisons to make sure the repack is working fine.

1st time repacked 3 files (the sample i've included with the download)
Then repacked 322 files from unpacked british isles mod
Then repacked 2516 files from unpacked british isles mod

Each time unpacked the new built pack and made a binary comparison between folders (used beyond compare v 3.02) The result was always OK. No differences among files.

I had a failure once:
Today tried to repack all 4217 files from british_isles.
That time the script started to work but stopped after +/-15 minutes of computing and gave no output. Maybe a memory overflow. I'm not sure

I had no problem with that because if somebody wants to pack +/-4000 files there is always the option to repack them inside 2 packs with +/-2000 each instead of 1 with +/-4000.

regards, and thanks for the reply

Jorge35 ( Hex35, just i've joined http://www.totalwar.org with this name )

[B. Ward]

Do you want this put in the Index of Tutorials, Guides and Modding Tools Thread?

[Hexdragon]

That would be fine indeed. Thanks.

The main thread for the tool (download link and instructions) is posted here:

http://www.twcenter.net/forums/showthread.php?t=188452

---
Edit: Maybe you could find convenient to merge that thread with this one.
---

There is another tool. If possible, consider it for the index also.

http://www.twcenter.net/forums/showthread.php?t=181350

regards

Jorge

[B. Ward]

Merged per request.

[makanyane]

sorry for the very long range *bump* but did anyone ever get this to work?

I was trying it because game takes a huge amount of time to read through/load our models_strat/residences files - searching for faction variants that don't actually exist (31 x factions x cultures x each city level) - and does that every time the strat map is reloaded after battle. I thought if I could pack those bits it might load quicker....

tool seems to make a pack OK, but game doesn't attempt to look in it, even though it says it's loaded
e.g. if I call my new pack 'localised.pack' and put in top level packs folder and call old one 'xlocalised.pack' the log trace shows it's looking for stuff in 'xlocalised.pack' and not in my new one
(didn't work from mod-folder/packs either)

[Ishan]

I have no idea on the issue you are having but did you try defragmentation of your Drive, Mak?
I know for certain it makes a difference but how much in your case i'm not certain.

Or perhaps compress that entire folder (gets highlighted in blue when done)?

Spoiler Alert, click show to read: 

I know it's ineffective when done on .exe files etc but in bulk on files like .txt and stuff it's worth it. Although i haven't tried all this on M2TW besides defrag.

[Tellos Athenaios]

sorry for the very long range *bump* but did anyone ever get this to work?

I was trying it because game takes a huge amount of time to read through/load our models_strat/residences files - searching for faction variants that don't actually exist (31 x factions x cultures x each city level) - and does that every time the strat map is reloaded after battle. I thought if I could pack those bits it might load quicker....

tool seems to make a pack OK, but game doesn't attempt to look in it, even though it says it's loaded
e.g. if I call my new pack 'localised.pack' and put in top level packs folder and call old one 'xlocalised.pack' the log trace shows it's looking for stuff in 'xlocalised.pack' and not in my new one
(didn't work from mod-folder/packs either)

Have you checked if the game doesn't simply mount *any* pack found in the packs folder? I.e. does it make a difference if you move the xlocalisted.pack from, say, mods/mymod/packs to mods/mymod/dump ?

[makanyane]

I have no idea on the issue you are having but did you try defragmentation of your Drive, Mak?
I know for certain it makes a difference but how much in your case i'm not certain.

will have a go at that but this is actually on one of the only relatively clean partitions on my hd's.... so not sure it will help.

Have you checked if the game doesn't simply mount *any* pack found in the packs folder? I.e. does it make a difference if you move the xlocalisted.pack from, say, mods/mymod/packs to mods/mymod/dump ?

Well yup it mounts all the packs within the packs folders... my problem isn't stopping it doing that - it's that despite the fact that it apparently mounted my new pack - it doesn't try and look inside it for the missing bits when they are required.

I'm assuming from the fact that it is looking inside the renamed 'xlocalised.pack' that the problem isn't to do with the pack name or location but instead something integral to the pack; like the header information... judging by the age of this thread I'm wondering if the tool isn't compatible with the later patches ?

[irishron]

By the date of the OP, it should. Are you still using XP. mak?

[makanyane]

By the date of the OP, it should. Are you still using XP. mak?

yup I'm on XP - does that make a difference to this?

[irishron]

I was reading the bottom of the OP and he built it on XP. The way Win 7 can get cranky at times, I wanted to eliminate it as a possibility.

What happens if you move the xlocalization.pack to your desktop and try to force it to read yours?

[makanyane]

I was reading the bottom of the OP and he built it on XP. The way Win 7 can get cranky at times, I wanted to eliminate it as a possibility.

What happens if you move the xlocalization.pack to your desktop and try to force it to read yours?

If I move the original localization pack, however named / replaced, out of m2/packs folder then game crashes on load (I am trying this on a still packed data folder as I was trying to check if mod worked for people who hadn't unpacked...) and erm, it definitely doesn't without that original file....

[irishron]

Dumb question or dunb idea, take your choice.

Are you trying to unack the vanilla's localisation pack add the mod's files to it and repack it with the same name?

Maybe a holdover from RTW/BI, why not name yours mod_1.pack?

[makanyane]

what I was trying to explain was that if I rename the existing packs it still looks in them if still in /packs folder... and whatever I call my new pack mod_0 or 1 or anything else whilst it 'mounts' it, the game doesn't actually try to open it to find stuff...

[irishron]

Sorry, I'm out of ideas, then. His instructions seem to be straight forward. King kong will probably be looking for something like this for his new Third Age settlements. I wonder if he's had any luck, yet.

[Tellos Athenaios]

I'm assuming from the fact that it is looking inside the renamed 'xlocalised.pack' that the problem isn't to do with the pack name or location but instead something integral to the pack; like the header information... judging by the age of this thread I'm wondering if the tool isn't compatible with the later patches ?

Doesn't seem to be the case, though. Had a looksee at the Python source code & a few PACK files in a hexeditor and while it's not actually commented or anything it does seem quite clear that the header data consists of just the following:

The ASCII characters 'PACK', the bytes \x00 \x00 \x03 \x00, and 3 unsigned ints representing length of offset tables. Nothing to suggest order.

I expect that the game simply does a directory query for <folder>\packs\*, mounts any pack it finds, and is simply not prepared to detect or handle the case when two packs contain names that collide. Presumably it overwrites its internal data structure with data for whatever happens to be the last file found in the packs with the same name, but AFAIK you can't really make any prediction based off this assumption because directory stats do not return results in a well defined order anyway. IOW it is not possible to predict with 100% certainty what the pack file is which will be mounted last (even if they're not mounted in parallel).

I guess the only way to work around this issue is making sure that the packs do not contain vanilla versions of your modified files, which in turn would mean extracting a pack, replacing the vanilla files and packing it back.

Alternatively, the readme's included with the tool suggest something about “alphabetic order” but don't really expand on why this should matter at all.

EDIT: Another possibility is that the game uses timestamps to order pack files, but this seems to be a very unreliable way to order them...