Return of XIDX: with bug fixes and updates for M2TW.

**Razor** · March 26, 2012, 11:29 AM

So, how do I get it to work? I can double-click the .exe whatever I want, but nothing happens other than a DOS-window popping up and disappearing. A more elaborate explanation as to how to make it work would be very helpful.

**Emperor of Hell** · March 26, 2012, 11:52 AM

You need to make .bat files with various commands. All info is in the ReadMe.

**Tellos Athenaios** · March 26, 2012, 02:08 PM

Version 1.1 brings:

Bugfixes
Fix memory leak resulting from exit() in the usage info function.

Misc
Change naming conventions for binary blobs extracted from event (EVT) packs. Numbering now starts at zero for every "frame", rather than just at the beginning of the pack. This change would allow other tools to guess the event (bank) that corresponds to a given file name based on the number before the bin suffix.

**Tellos Athenaios** · March 26, 2012, 02:14 PM

Originally Posted by Razor

So, how do I get it to work? I can double-click the .exe whatever I want, but nothing happens other than a DOS-window popping up and disappearing. A more elaborate explanation as to how to make it work would be very helpful.

Originally Posted by Emperor of Hell

You need to make .bat files with various commands. All info is in the ReadMe.

Emperor is basically right, alternatively you can use the commandline manually. For instance, to extract the EVT blobs from events.dat/events.idx into an "output" directory:

Code:

set xidx=D:\xidx\bin\xidx_amd64.exe
cd D:\M2TW\mods\my_mod\data\sounds
mkdir output
cd output
"%xidx%" -xbe ../events

The above code commands first define a handy variable which will refer to the exe to launch. This example assumes you run 64bits Windows, so it picks xidx_amd64.exe.
Then it changes the working directory to the sounds directory of your mod, creates an output directory in that place, and then changes the working directory to the output directory.

Next it calls the XIDX program "%xidx%" using the variable, passes it the command to extract the event packs and add bin file suffixes to the output files. The pack which will be extracted is the events DAT/IDX pair in your mod's sounds directory (that's what the ../events refers to).

**Razor** · March 26, 2012, 02:26 PM

@Emperor of Hell:

How could I have missed that! I've got something working now. Copied and modified one of Vercingetorix's .bat files to extract the packs. Thanks for pointing me to the readme. I missed the important bits.

@Tellos Athenaios: I feel like an idiot. This is quite technical nearly above my capabilities to understand for a humble alpha person like me.

Code:

set xidx=D:\xidx\bin\xidx_amd64.exe

This line is the path to the xidx_amd64.exe

Code:

cd D:\M2TW\mods\my_mod\data\sounds

This line created an .idx pack file in the directory which you specified (create directory?).

Code:

mkdir output
cd output

Can't quite translate this.

Code:

"%xidx%" -xbe ../events

Can't quite understand the "%xidx%" part, even after reading your explanation.

Sorry, just trying to understand this.

**Tellos Athenaios** · March 26, 2012, 03:52 PM

Originally Posted by Razor

@Tellos Athenaios: I feel like an idiot. This is quite technical nearly above my capabilities to understand for a humble alpha person like me.

Code:

set xidx=D:\xidx\bin\xidx_amd64.exe

This line is the path to the xidx_amd64.exe

More precisely it defines a variable called "xidx". The contents (value) of that variable is the path to the xidx exe which you want to use.

Code:

cd D:\M2TW\mods\my_mod\data\sounds

This line created an .idx pack file in the directory which you specified (create directory?).

Code:

mkdir output
cd output

Can't quite translate this.

. No, as explained in the previous post what it does is changing the "current working directory". If a program, say xidx, attempts to read a file called "../events.dat" then what it will actually get from the OS depends on the current working directory. The code above first changes the working directory to "D:\M2TW\mods\my_mod\data\sounds\". Then it runs the mkdir program which is a program that creates directories, so what happens is that a directory called "output" is created inside "D:\M2TW\my_mods\data\sounds". Then, the next cd command switches the working directory to this output folder, so that any files created by XIDX will be "inside" that.

Code:

"%xidx%" -xbe ../events

Can't quite understand the "%xidx%" part, even after reading your explanation.

This line is the one which does all the work, it runs the XIDX program. "%xidx%" bit simply substitutes the value of the xidx variable defined earlier, so you end up running the following from inside that output directory:

Code:

"D:\xidx\bin\xidx_amd64.exe" -xbe ../events

The quotes are necessary in case you have spaces in the path of the XIDX program (and hence, the value of the xidx variable).

**Tellos Athenaios** · April 15, 2012, 03:03 PM

New version available.

Bugfix
Version 1.2 fixes a memory/file descriptor leak when opening archives with the wrong archive type flag. (For example specifying the -a flag for animations packs when trying to list the contents of a sound IDX/DAT pair.)

**Withwnar** · May 14, 2012, 03:32 AM

Nice one. Thank you.

I am unsure of the difference between the x86 and amd64 versions. For unpacking sound files are there some processors on which the x86 version won't work and amd64 should be used instead?

**Tellos Athenaios** · May 14, 2012, 03:18 PM

Unlikely. Basically, if you can run M2TW you will be able to run the x86 version of XIDX.

**Withwnar** · May 14, 2012, 11:39 PM

Great. Thanks.

**Tellos Athenaios** · October 01, 2012, 07:08 PM

New version available:

Bugfixes

Fix non-virtual destructors
Skip packing of EVT blobs which do not conform to the filename conventions used by XIDX.
Fix error messages when packing an EVT blob fails
Improve file name parsing of EVT blobs (no longer limited to files from the current directory).

Feature

Add makefile flag to skip cross compilation (MinGW targets).

Reminder: beware of the data folder
XIDX rewrites paths so they are relative to the root of the mod (or installation if you do not use a mod folder), and it assumes that the "root" data directory is the last "data/" in the path. This is pretty much necessary to make packing work by default on Windows (unless you wish to try you hand at this sort of thing, but it does mean that your last "data/" must be the right "data/". Otherwise the path saved in the IDX file will be an invalid value. Example: "mods\EBII\data\sounds\data\SFX\effect.wav" would be rewritten to "data\SFX\effect.wav" instead of "data\sounds\data\SFX\effect.wav". So if the game tries to load "data\sounds\data\SFX\effect.wav" because that is what you put in your code, it would fail. Easy workaround: do not have data folders inside your data folder, at least not in the parts that you want to pack with XIDX.

This message does not apply to packing events, since the tool does not store filenames for those in generate IDX/DAT packs anyway. Note that for events you will want to sort filenames alphabetically so the filename convention for EVT blobs will guarantee that they appear in the right order. Example command for packing EVT blobs:

Code:

dir /S /A:-D /B /O:N "path/to/EVT/blobs/directory" "*.bin" | xidx -cbef "path/to/mods/my_mod/data/sounds/events"

**Tellos Athenaios** · October 13, 2012, 11:20 AM

Version 1.4 of XIDX is now available and it contains the following updates:

Features

Add support for creating RTW event packs (add support for the RTW file version stamp)
Extended the Readme with a lot more information on how to correctly generate packs, in particular event packs.

Fixes

Fix file version stamps for event packs (though the game seems to ignore them anyway)
Don't forget the Windows users: extend previous bug fix for sniffing out type numbers to paths with backslashes in them as well.

**Tellos Athenaios** · April 06, 2013, 09:58 AM

Version 1.5 of XIDX is now available.

Features

Add -B option for arbitrary suffix manipulation. The -b switch is now equivalent to using `-B bin'

Fixes

Fix entry size in generated animation packs when using suffix manipulation

**makanyane** · March 22, 2014, 06:29 AM

am having problems getting re-packed anims/skeletons to work (M2 crashes on start up)

I think problem is due to

Originally Posted by readme

Please note that when packing XIDX converts absolute paths into relative paths by stripping the component
preceding the last "data" directory in the file names to make the tool easier to use

I'm trying to re-pack stuff that the anim_list had as
mods/Dragonborn/data/animations/
mods/akthross/data/animations/
etc...

unpacking the original (game generated?) anims put the extracted files into those folders - XIDX_x86 couldn't find the files when I left them there, so I moved the files into just data/animations/ and it made the pack.idx /.dat files fine...

however the extracted skeleton files have the mods/akthross/..... .cas path names inside them - and that seems to stay like that through re-packing process

so I'm getting my trace log file attempting to find the first .cas listed in skeletons.dat at mods/akthross/data/animations/MTW2_Spear/MTW2_Spear_Stand_A_idle.cas
which presumably isn't there anymore because the mods folder info was stripped out of the pack files

any ideas? or have I missed something else more fundamental?

**Tellos Athenaios** · March 22, 2014, 05:48 PM

Originally Posted by makanyane

any ideas? or have I missed something else more fundamental?

I thought there was at least one post of mine in which I attempt to clearly outline what is the matter with animations. Turns out, there isn't. So here goes.

XIDX deals with archives in the IDX/DAT file format. These archives are split over two files, one IDX file that contains meta data which describe the contents of the payload DAT file. The payload DAT file is, essentially concatenated contents of a lot of files. XIDX does not really understand any of the various files that go into the payload. It simply reads the data and copies it to the payload DAT file as is. All of the 'cleverness' involves manipulating the IDX meta data.

In the case of animations, however, the game does not use the same file format for animations inside the payload DAT file as whatever is produced by the tools to work with animations. The tools to edit animations focus on working with what I term 'source' files, which is basically the M2TW format of the animation CAS files. The game, on the other hand uses something that looks like RTW CAS animations instead. So before packing them with XIDX you would need to convert your M2TW source animations into the payload format.

Posts of mine which cover the subject, for reference:

And the file format you need to generate, as far as I understand it:

Spoiler Alert, click show to read:

Code:

Many thanks to johnwhile (from TWC) for the research on this format:

CAS file structure is:

+ 0	2byte ushort	number_of_frames
+ 2	2byte ushort	number_of_bone_rotations	// 1 less than number_of_bones in m2tw format, dropped scene root?
+ 4	1byte ubyte	number_of_bone_positions	// 1 less than number_of_bones in m2tw format, dropped scene root?

array of rotation_frame structs of length number_of_frames
array of position_frame structs of length number_of_frames

struct rotation_frame:
    array of quaternion structs of length number_of_bone_rotations

struct position_frame:
    array of coordinate structs of length number_of_bone_positions (bone position)
    it appears M2TW has a bug which means that the first bone has incorrect coordinates for the x and z coordinate (always 0.0)

struct coordinate:
    + 0		32bit float	x
    + 4		32bit float	y
    + 8		32bit float	z

struct quaternion:
    + 0		coordinate struct
    + 12	32bit float	w // angle of rotation around the vector [x,y,z]

struct doublet:
    + 0		32bit float	x 	signed distance
    + 4		32bit float	z	signed distance
    
array of doublet structs of length number_of_frames:
    signed distance over x and z axis between the next frame and the current frame (i.e. next_frame minus current_frame)
    
array of 32bit floats of length number_of_frames:
    distance in translation between last frame and current frame of first bone
    
    if first or last:
        distance between last and first
    else:
        distance between last and first minus sqrt(x^2 + z^2) of current position // really weird. also must round to float agressively because of rounding bugs.
    
array of coordinate structs of length number_of_frames:
    correct bone position for the first bone (see array of position_frame structs) with proper (correct) x and z coordinates.

32bit float	animation_duration ??? (in seconds) // last of the frame time/duration values given in m2tw format. if this is meant to be a total it is almost certainly a bug in M2TW...
32bit float 	distance traveled over x,z axis :: length of the 2d vector described by the next two x & z distance floats
32bit float	distance traveled over x axis (signed) :: x-coord of first bone in last frame minus x-coord of first bone in first frame // bug, would discard distance traveled during first frame?
32bit float 	distance traveled over y axis (signed) :: y-coord of first bone in last frame minus y-coord of first bone in first frame // bug, would discard distance traveled during first frame?
32bit float	distance traveled over z axis (signed) :: z-coord of first bone in last frame minus z-coord of first bone in first frame // bug, would discard distance traveled during first frame?
32bit float 	speed :: the length of the 2d vector described by previous two x & z distance floats divided by animation_duration
32bit float	speed / 10 ???

32bit float 	???
4byte uint       bitmask ???
4byte NULL	(0x0 :: 0) ???

**makanyane** · March 23, 2014, 02:41 AM

Thanks for the info - I think you're overestimating what I'm attempting to do though.

As far as I understand the system/possibilities you have;

Modder/CA made .cas files - these can be worked on & can also be used by the game to re-generate it's own packs. To get game to regenerate own packs you need absolutely everything that's listed in descr_skeleton.txt in that correct .cas format and the .evt files.

Once you've got an engine generated pack set you can extract the anims and skeletons with XIDX - although you don't get .evt files back and the XIDX .cas files aren't the same sort of format that modders can work with & are also not the right sort of format for the game to use to re-generate it's own packs...
...however XIDX can re-pack those back into game readable pack format.

You can't mix and match the two types of stuff - so XIDX won't re-pack modders .cas's and the game won't re-pack XIDX .cas's... that's all fine so far.

The stage I've got to is I have existing game generated packs for the mod as it stands which included a whole heap of stuff I'm having difficulty finding the original modders versions of.
I have a smaller, new, game generated pack set which only contains two new skeletons and anims (produced by deleting everything else from descr_skeleton.txt temporarily)

If I unpack both of those I get XIDX friendly versions of the .cas's for everything, can append the two sets of lists and re-pack.

My issue seems to be that the XIDX re-packing process strips out the mods/my_old_mod/ info from the pack.idx file but not from the skeletons.dat file.
I could re-make the smaller pack by running it from M2/data level, but I can't re-make the larger pack without the mods/my_old_mod stuff, because I don't have the necessary bits, see start of problem...

Game always seems to append the mods/where_it_was_built/ info to the paths in the packs/skeletons it creates if the re-build animations process is run from a mod-folder, this doesn't seem to affect the portability of the files as the existing mod anims with that in have been running quite happily without the referred to mods being present - so I assume it's just a cross reference between the skeletons.dat info and the pack.idx stuff, which is now broken due to it getting stripped from the pack.idx only?

EDIT; also the one big existing anim pack already has references to two different mod paths in it, e.g. mods/modA/data and mods/modB/data,
so at some stage in the past someone has managed to stick two game generated packs together, but I can't see a way of achieving that with the current tool, or in game.
Was there ever a previous version that didn't strip the stuff before /data?

**Tellos Athenaios** · March 23, 2014, 06:04 PM

Originally Posted by makanyane

Thanks for the info - I think you're overestimating what I'm attempting to do though.

As far as I understand the system/possibilities you have;

Modder/CA made .cas files - these can be worked on & can also be used by the game to re-generate it's own packs. To get game to regenerate own packs you need absolutely everything that's listed in descr_skeleton.txt in that correct .cas format and the .evt files.

Yep.

Once you've got an engine generated pack set you can extract the anims and skeletons with XIDX - although you don't get .evt files back

As an aside, the .evt files become part of the generated skeletons when the game regenerates the skeletons.idx/skeletons.dat files.

and the XIDX .cas files aren't the same sort of format that modders can work with & are also not the right sort of format for the game to use to re-generate it's own packs...
...however XIDX can re-pack those back into game readable pack format.

Yes. Please note that the game generates packs with absolute file names in some cases. That means repacking them is required if you want to share your pack (and have it work) with others.

You can't mix and match the two types of stuff - so XIDX won't re-pack modders .cas's and the game won't re-pack XIDX .cas's... that's all fine so far.

More precisely (in case others read this) ... XIDX will not perform the conversion for you so if you try without converting first you will get a pack which contains invalid payload contents as far as the game is concerned.

The stage I've got to is I have existing game generated packs for the mod as it stands which included a whole heap of stuff I'm having difficulty finding the original modders versions of.
I have a smaller, new, game generated pack set which only contains two new skeletons and anims (produced by deleting everything else from descr_skeleton.txt temporarily)

If I unpack both of those I get XIDX friendly versions of the .cas's for everything, can append the two sets of lists and re-pack.

My issue seems to be that the XIDX re-packing process strips out the mods/my_old_mod/ info from the pack.idx file but not from the skeletons.dat file.

I could re-make the smaller pack by running it from M2/data level, but I can't re-make the larger pack without the mods/my_old_mod stuff, because I don't have the necessary bits, see start of problem...

Game always seems to append the mods/where_it_was_built/ info to the paths in the packs/skeletons it creates if the re-build animations process is run from a mod-folder, this doesn't seem to affect the portability of the files as the existing mod anims with that in have been running quite happily without the referred to mods being present - so I assume it's just a cross reference between the skeletons.dat info and the pack.idx stuff, which is now broken due to it getting stripped from the pack.idx only?

What probably happened is a combination of two things:

The game expects that the names of the skeletons are just 'names' inside the archive. It is not prepared for paths, actually. You can get that to work with XIDX 1.5 but it's a bit tricky because you need to pack from within the data\animations directory (and of course you cannot have skeletons inside subdirectories).
Additionally the skeleton files themselves contain references to animations. When the game regenerates them it uses absolute paths for references but I checked with the skeleton pack from a Kingdoms campaign (teutonic) in which all those paths were proper relative paths (same as what goes in the animations archive). That means the references in the skeletons which the game regenerates for you are 'chained' if you will to the exact location which they were generated from as well. You should fix that if you move files around (for instance if you import skeletons from one mod to another) or if you intend to share the resulting skeletons archive with anyone else.

To deal more gracefully with the first issue I have prepared XIDX version 1.6. See next post:

**Tellos Athenaios** · March 23, 2014, 06:05 PM

Version 1.6 of XIDX is now available. It contains:

Features

Add -P and -p options for disabling implicit path mangling.

Bug fixes

Disable scaling for M2TW animation archives.
Write skeleton names instead of paths by default. (May be overriden using -P or -p.)

Misc

Add extra warnings and mention default file paths in --help output.
Add note on correctly packing skeletons, animation CAS file format(s) conversion.

**makanyane** · March 24, 2014, 03:19 AM

Thanks for the quick response and the updated version - am trying that now.

Just a quick note - the new version of xidx_x86.exe now requires "libwinpthread-1.dll" to run on my windows system, have downloaded a version of that .dll and put it in folder with the .exe and it now seems to run fine.

**Tellos Athenaios** · March 24, 2014, 06:44 AM

Originally Posted by makanyane

Thanks for the quick response and the updated version - am trying that now.

Just a quick note - the new version of xidx_x86.exe now requires "libwinpthread-1.dll" to run on my windows system, have downloaded a version of that .dll and put it in folder with the .exe and it now seems to run fine.

I also noticed a couple of other more minor issues, so I've pushed out a new version.

Version 1.6.1 of XIDX is a point release addressing the following issues:

Bug fixes

Fix Windows builds to not complain about libwinpthread-1.dll (-static).
Fix -h option.
Report first of -p or -P as not affecting extracting/listing archives.

Thread: Return of XIDX: with bug fixes and updates for M2TW.

Thread Tools

Search Thread

Display

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Re: Return of XIDX: with bug fixes and updates for M2TW.

Posting Permissions