How to mount multiple views of pack files simultaneously
Basic requirements
This is currently OSX only, but FUSE can be made to work on Linux in 15 minutes of effortOSX and Linux are both supported now.
There is supposedly a Windows port of FUSE in progress - it might not be able to deal with more
complicated filesystems, but this one is very simple, so it's very likely that it could be available
on Windows with one evening of effort and only slightly more clue about Windows programming
than I have.
That is - you are quite likely to see it on Windows at some point. Hell, I already use it on Windows
by fairly contrived means of running Windows inside Parallels and mounting OSX there.
By the way it works just perfectly with Napoleon as well, and all DLCs for both, as far as I know.
Multiple ETW trees
So - have you ever needed to know what ETW filesystem contains in multiple versions?
Well, which modder didn't need that? It's an unbelievable pain in the ass -
to have to unpack whole .pack set for vanilla, unpack it again for your mod,
for old versions of your mod, for your mod with this or that minimod, maybe even for all of the above
with previous CA patch, in case their update is to be blamed for breaking something.
And what if you're maintaining multiple mods? That's going to be pain beyond pains.
And unless you have a spare 2TB drive just for all these combinations or like symlink hackery,
you probably won't even have enough diskspace for that.
You might even be wondering why am I suggesting something as ridiculous -
and that's because this ridiculous thing is already possible!
Prepare PACK filesystem driver
FUSE drivers have two parts - low-level driver inside operating system that doesn't
really know about any particular format, and user-side program handling each mount point.
This mostly means you're very unlikely to have it crash your computer and it doesn't
require special privileges - when driver crashes the only thing that happens is automatic
unmounting of that virtual drive.
So for OSX, you'll need to install MacFUSE and (recommended) restart your computer.
That's the only restart you'll ever need. It's simple .dmg package, nothing difficult.
For Linux fuse might already be installed depending on your distribution, or it's
available as standard package - on Ubuntu it is libfuse-dev. (detailed instructions
for other distributions of Linux to go later if requested)
(Windows instruction to go here later)
Now get pack driver from etwng. You can use git or click on Download link to
get .zip or .tar.gz - whichever way you prefer.
Now go to etwng/pack/fuse directory and run command make
That's it. If you've followed all steps you should have packfs binary ready.
Preparing directory structure and gathering .pack files
Before doing anything more complicated, let's gather ETW and mod .packs -
they're probably on external drives so let's create some convenient symlinks:
$ ln -s '/media/Untitled/Program Files/Steam/steamapps/common/empire total war/data/' ~/.etw
$ ln -s '/media/FREECOM HDD/ETW_everything/mods/' ~/.etwmods
And now let's create some directory structure to keep our mountpoints.
On OSX and Linux mount points look like normal directories;
on Windows they're single letter drives - OSX/Linux way has obvious advantage
when you want to have more than 26 of mounted virtual drives, which is exactly
what we'll be trying to do.
So here's basic directory structure of mountpoints (if you forget this, thes commands
will be repeated later so no worries):
$ mkdir -p ~/.etwfs/packs
$ mkdir -p ~/.etwfs/vanilla
$ mkdir -p ~/.etwfs/mods
$ mkdir -p ~/.etwfs/old
Mounting individual pack files
Let's start with something easier - mounting individual packs.
Usually we want to see entire directory tree, but as it's so easy
let's do that as well.
To mount a pack you need to create a directory just for it (-p option
means to just skip this step if it already exists), and mount it with packfs program.
If it worked, you'll see something like this:
$ mkdir -p ~/.etwfs/packs/
$ ~/etwng/pack/fuse/packfs ~/.etw/ui.pack ~/.etwfs/packs/ui
Loading pack /Users/taw/.etw/ui.pack
$
Now let's take a peek inside:
$ ls ~/.etwfs/packs/ui/*
achievements/ labels.luac
agents.luac loading_screen_pictures/
army.luac optionsmanager.luac
battle ui/ panelmanager.luac
buildings/ portraits/
campaign ui/ recruitment.luac
common ui/ siegeequipment.luac
construction.luac technology_common.luac
coreutils.luac technology_researching.luac
eventicons/ technology_stealing.luac
eventpics/ templates/
file_requesters.luac theatre_map.luac
flags/ units/
frontend ui/ utilities.luac
horizontal_mousewheel_script.luac vertical_mousewheel_script.luac
huds.luac warscapetwin/
Couldn't be easier. Now let's unmount:
$ umount ~/.etwfs/packs/ui
$
That's it, couldn't have been simpler!
Mounting every .pack file
FIXME: Steps described here don't really work in default MacFUSE settings,
which has 24 open filesystems limit. You'd need to compile it manually to get more.
If you get "MacFUSE: failed to open device: Resource busy", that's the reason.
As mounting individual packs is less interesting, just skip it and go to the following step.
Linux fuse has no such problem AFAIK.
We could now run 60 commands like that every time, but I for one
am far too lazy to do that unless I have to. Fortunately I don't, so
I wrote a simple Ruby script that mounts every pack in its own directory.
#!/usr/bin/env ruby
require "pathname"
require "fileutils"
packfs = "/Users/taw/etwng/pack/fuse/packfs"
etw = Pathname("/Users/taw/.etw")
etwfs = Pathname("/Users/taw/.etwfs")
Pathname.glob(etw + "*.pack").each{|pack|
dir = etwfs + "packs" + pack.basename(".pack")
FileUtils.mkdir_p dir
next if dir.mountpoint?
system packfs, pack, dir
}
Just run and enjoy!
Mounting vanilla and mods
Now it's time to mount vanilla:
$ mkdir -p ~/.etwfs/vanilla
$ ~/etwng/pack/fuse/packfs ~/.etw/*.pack ~/.etwfs/vanilla
Loading pack /Users/taw/.etw/boot.pack
Loading pack /Users/taw/.etw/anim.pack
Loading pack /Users/taw/.etw/battlepresets.pack
Loading pack /Users/taw/.etw/battleterrain.pack
Loading pack /Users/taw/.etw/groupformations.pack
Loading pack /Users/taw/.etw/local_en.pack
Loading pack /Users/taw/.etw/main.pack
Loading pack /Users/taw/.etw/models.pack
Loading pack /Users/taw/.etw/seasurfaces.pack
Loading pack /Users/taw/.etw/sound_non_wavefile_data.pack
Loading pack /Users/taw/.etw/sounds.pack
Loading pack /Users/taw/.etw/sounds_animation_triggers.pack
Loading pack /Users/taw/.etw/sounds_campaign.pack
Loading pack /Users/taw/.etw/sounds_music.pack
Loading pack /Users/taw/.etw/sounds_other.pack
Loading pack /Users/taw/.etw/sounds_placeholder.pack
Loading pack /Users/taw/.etw/sounds_sfx.pack
Loading pack /Users/taw/.etw/subtitles.pack
Loading pack /Users/taw/.etw/supertexture.pack
Loading pack /Users/taw/.etw/terrain_templates.pack
Loading pack /Users/taw/.etw/testdata.pack
Loading pack /Users/taw/.etw/ui.pack
Loading pack /Users/taw/.etw/voices.pack
Loading pack /Users/taw/.etw/patch.pack
Loading pack /Users/taw/.etw/patch2.pack
Loading pack /Users/taw/.etw/patch3.pack
Loading pack /Users/taw/.etw/patch4.pack
Loading pack /Users/taw/.etw/patch5.pack
Loading pack /Users/taw/.etw/patch_en.pack
Loading pack /Users/taw/.etw/movies.pack
Loading pack /Users/taw/.etw/patch_media.pack
Loading pack /Users/taw/.etw/ui_movies.pack
$
How simple was that? Let's go right ahead and mount some mods:
$ mkdir -p ~/.etwfs/mods/dm63 ~/.etwfs/mods/aum ~/.etwfs/mods/is21
$~/etwng/pack/fuse/packfs ~/.etw/*.pack ~/.etwmods/dm63/data/*.pack ~/.etwfs/mods/dm63
Loading pack /Users/taw/.etw/boot.pack
Loading pack /Users/taw/.etw/anim.pack
Loading pack /Users/taw/.etw/battlepresets.pack
Loading pack /Users/taw/.etw/battleterrain.pack
Loading pack /Users/taw/.etw/groupformations.pack
Loading pack /Users/taw/.etw/local_en.pack
Loading pack /Users/taw/.etw/main.pack
Loading pack /Users/taw/.etw/models.pack
Loading pack /Users/taw/.etw/seasurfaces.pack
Loading pack /Users/taw/.etw/sound_non_wavefile_data.pack
Loading pack /Users/taw/.etw/sounds.pack
Loading pack /Users/taw/.etw/sounds_animation_triggers.pack
Loading pack /Users/taw/.etw/sounds_campaign.pack
Loading pack /Users/taw/.etw/sounds_music.pack
Loading pack /Users/taw/.etw/sounds_other.pack
Loading pack /Users/taw/.etw/sounds_placeholder.pack
Loading pack /Users/taw/.etw/sounds_sfx.pack
Loading pack /Users/taw/.etw/subtitles.pack
Loading pack /Users/taw/.etw/supertexture.pack
Loading pack /Users/taw/.etw/terrain_templates.pack
Loading pack /Users/taw/.etw/testdata.pack
Loading pack /Users/taw/.etw/ui.pack
Loading pack /Users/taw/.etw/voices.pack
Loading pack /Users/taw/.etw/patch.pack
Loading pack /Users/taw/.etw/patch2.pack
Loading pack /Users/taw/.etw/patch3.pack
Loading pack /Users/taw/.etw/patch4.pack
Loading pack /Users/taw/.etw/patch5.pack
Loading pack /Users/taw/.etw/patch_en.pack
Loading pack /Users/taw/.etwmods/dm63/data/darthmod.pack
Loading pack /Users/taw/.etw/movies.pack
Loading pack /Users/taw/.etw/patch_media.pack
Loading pack /Users/taw/.etw/ui_movies.pack
Loading pack /Users/taw/.etwmods/dm63/data/unit_pack_compatibility_project_v5_8.pack
Loading pack /Users/taw/.etwmods/dm63/data/unit_pack_compatibility_project_v6_0.pack
$~/etwng/pack/fuse/packfs ~/.etw/*.pack ~/.etwmods/aum/data/*.pack ~/.etwfs/mods/aum
Loading pack /Users/taw/.etw/boot.pack
Loading pack /Users/taw/.etw/anim.pack
Loading pack /Users/taw/.etw/battlepresets.pack
Loading pack /Users/taw/.etw/battleterrain.pack
Loading pack /Users/taw/.etw/groupformations.pack
Loading pack /Users/taw/.etw/local_en.pack
Loading pack /Users/taw/.etw/main.pack
Loading pack /Users/taw/.etw/models.pack
Loading pack /Users/taw/.etw/seasurfaces.pack
Loading pack /Users/taw/.etw/sound_non_wavefile_data.pack
Loading pack /Users/taw/.etw/sounds.pack
Loading pack /Users/taw/.etw/sounds_animation_triggers.pack
Loading pack /Users/taw/.etw/sounds_campaign.pack
Loading pack /Users/taw/.etw/sounds_music.pack
Loading pack /Users/taw/.etw/sounds_other.pack
Loading pack /Users/taw/.etw/sounds_placeholder.pack
Loading pack /Users/taw/.etw/sounds_sfx.pack
Loading pack /Users/taw/.etw/subtitles.pack
Loading pack /Users/taw/.etw/supertexture.pack
Loading pack /Users/taw/.etw/terrain_templates.pack
Loading pack /Users/taw/.etw/testdata.pack
Loading pack /Users/taw/.etw/ui.pack
Loading pack /Users/taw/.etw/voices.pack
Loading pack /Users/taw/.etw/patch.pack
Loading pack /Users/taw/.etw/patch2.pack
Loading pack /Users/taw/.etw/patch3.pack
Loading pack /Users/taw/.etw/patch4.pack
Loading pack /Users/taw/.etw/patch5.pack
Loading pack /Users/taw/.etw/patch_en.pack
Loading pack /Users/taw/.etwmods/aum/data/aum_mod.pack
Loading pack /Users/taw/.etw/movies.pack
Loading pack /Users/taw/.etw/patch_media.pack
Loading pack /Users/taw/.etw/ui_movies.pack
$~/etwng/pack/fuse/packfs ~/.etw/*.pack ~/.etwmods/is21/data/*.pack ~/.etwfs/mods/is21
Loading pack /Users/taw/.etw/boot.pack
Loading pack /Users/taw/.etw/anim.pack
Loading pack /Users/taw/.etw/battlepresets.pack
Loading pack /Users/taw/.etw/battleterrain.pack
Loading pack /Users/taw/.etw/groupformations.pack
Loading pack /Users/taw/.etw/local_en.pack
Loading pack /Users/taw/.etw/main.pack
Loading pack /Users/taw/.etw/models.pack
Loading pack /Users/taw/.etw/seasurfaces.pack
Loading pack /Users/taw/.etw/sound_non_wavefile_data.pack
Loading pack /Users/taw/.etw/sounds.pack
Loading pack /Users/taw/.etw/sounds_animation_triggers.pack
Loading pack /Users/taw/.etw/sounds_campaign.pack
Loading pack /Users/taw/.etw/sounds_music.pack
Loading pack /Users/taw/.etw/sounds_other.pack
Loading pack /Users/taw/.etw/sounds_placeholder.pack
Loading pack /Users/taw/.etw/sounds_sfx.pack
Loading pack /Users/taw/.etw/subtitles.pack
Loading pack /Users/taw/.etw/supertexture.pack
Loading pack /Users/taw/.etw/terrain_templates.pack
Loading pack /Users/taw/.etw/testdata.pack
Loading pack /Users/taw/.etw/ui.pack
Loading pack /Users/taw/.etw/voices.pack
Loading pack /Users/taw/.etw/patch.pack
Loading pack /Users/taw/.etw/patch2.pack
Loading pack /Users/taw/.etw/patch3.pack
Loading pack /Users/taw/.etw/patch4.pack
Loading pack /Users/taw/.etw/patch5.pack
Loading pack /Users/taw/.etw/patch_en.pack
Loading pack /Users/taw/.etwmods/is21/data/Imperial Splendour DB.pack
Loading pack /Users/taw/.etwmods/is21/data/Imperial Splendour Media.pack
Loading pack /Users/taw/.etw/movies.pack
Loading pack /Users/taw/.etw/patch_media.pack
Loading pack /Users/taw/.etw/ui_movies.pack
Loading pack /Users/taw/.etwmods/is21/data/patch_is_text.pack
$
Have you noticed order of pack loading? mod packs go before movie packs, and unit pack compatibility are marked as movie packs. Yes, packfs is smart enough to figure it out on its own!
Now let's look what they're like inside with open ~/.etwfs/mods/dm63/db/units_tables/
How awesome is that? Or can PackManager do this? (open ~/.etwfs/mods/dm63/battleterrain/colour_maps/)
I don't think so either.
Old versions of vanilla
The final thing I want to show in this tutorial is how to mount older versions.
Now this isn't quite true, but you can get a good idea of what old versions of ETW
looked like by simply ignoring some patchN.pack files. So what are we waiting for?
Now to avoid listing every .pack on command line there are some
complex shell globs, but don't worry if you don't understand what they mean.
$ mkdir -p ~/.etwfs/old/{orig,patch,patch2,patch3,patch4,patch5}
$ ~/etwng/pack/fuse/packfs ~/.etw/[^p]*.pack ~/.etw/patch{_en,_media}.pack ~/.etwfs/old/orig
$ ~/etwng/pack/fuse/packfs ~/.etw/[^p]*.pack ~/.etw/patch{_en,_media,}.pack ~/.etwfs/old/patch
$ ~/etwng/pack/fuse/packfs ~/.etw/[^p]*.pack ~/.etw/patch{_en,_media,,2}.pack ~/.etwfs/old/patch2
$ ~/etwng/pack/fuse/packfs ~/.etw/[^p]*.pack ~/.etw/patch{_en,_media,,2,3}.pack ~/.etwfs/old/patch3
$ ~/etwng/pack/fuse/packfs ~/.etw/[^p]*.pack ~/.etw/patch{_en,_media,,2,3,4}.pack ~/.etwfs/old/patch4
$ ~/etwng/pack/fuse/packfs ~/.etw/[^p]*.pack ~/.etw/patch{_en,_media,,2,3,4,5}.pack ~/.etwfs/old/patch5
I skipped messages output by these commands.
Of course you can mix different versions of vanilla, multiple mods, mods in different order etc. very easily. The syntax is:
$ /path/to/packfs some.pack another.pack /where/to/mount
Order of packs matters only when they're of the same type,
like with multiple mods (and currently also different patchN
files MUST appear in order, due to packfs limitation that
I hope to fix later)
Doing something useful
Now what is it good for? How about this?
$ ls -l ~/.etwfs/*/*/groupformations.bin
-r--r--r-- 1 root 47414 2010-10-14 21:14 /Users/taw/.etwfs/mods/aum/groupformations.bin
-r--r--r-- 1 root 47414 2010-10-14 21:14 /Users/taw/.etwfs/mods/dm63/groupformations.bin
-r--r--r-- 1 root 47414 2010-10-14 21:14 /Users/taw/.etwfs/mods/is21/groupformations.bin
-r--r--r-- 1 root 63026 2010-10-14 21:59 /Users/taw/.etwfs/old/orig/groupformations.bin
-r--r--r-- 1 root 63982 2010-10-14 22:27 /Users/taw/.etwfs/old/patch2/groupformations.bin
-r--r--r-- 1 root 63982 2010-10-14 22:27 /Users/taw/.etwfs/old/patch3/groupformations.bin
-r--r--r-- 1 root 63982 2010-10-14 22:27 /Users/taw/.etwfs/old/patch4/groupformations.bin
-r--r--r-- 1 root 47414 2010-10-14 21:14 /Users/taw/.etwfs/old/patch5/groupformations.bin
-r--r--r-- 1 root 63982 2010-10-14 22:27 /Users/taw/.etwfs/old/patch/groupformations.bin
$
That was pretty fast, wasn't it?
How long would it take you to find all versions
of groupformations.bin with something like PackManager?
Coming next
I'm pretty sure even Creative Assembly doen't have anything like it!
FUSE driver needs a bit of work, from easiest to most difficult:
Instructions and Makefile flags to make it work on LinuxLinux is now supported as well- Instructions how to increase MacFUSE limit of 24 filesystems
- Handling of pack dependencies, so you can specify patchN in any order you feel like
- And finally - Windows port. I hope I enticed you enough that one of Windows programmers among you will go for it now. If not, BootCamp users rejoice!
Reply With Quote
