Return of XIDX: with bug fixes and updates for M2TW.

The Europa Barbarorum Team is proud to announce the return of one of the most venerable modding tools dating back to the early days of RTW: XIDX, with bug fixes and updates for M2TW. The “XIDX” project provides modders with a tool to assemble and disassemble IDX/DAT pairs of files as used in RTW and M2TW (including Kingdoms). Originally developed by Vercingetorix for use with Rome Total War versions of these archives on Windows only, the tool now compiles and runs on Linux as well (and should run on any proper POSIX x86 or AMD64 environment).

The project is based on the source code of the XIDX program as published on the TWC. It has been heavily modified, the comprehensible change log is as follows:

Features:

• Add -m option for building Medieval II versions of supported pack types.
• Add -b option for transparent suffix ('.bin') manipulation (useful with skeletons for avoiding name collisions).
• Add -e option for assembly/disassembly of event packs.

Bug fixes:

• Port/fixes to build on Linux with GCC (tested with various versions >= 4.3.2)
• Builds with GCC-Mingw32 and GCC-Mingw64 (tested with various versions >= 4.4.4)
• Fix lurking segfaults on exit and inside the fork function of the XIDX shell.
• Fix memory allocation, so the tool does not allocate lots of memory it never uses.

Misc:
Strip batch files, MS Visual Studio project file etc., added makefile.

The program is available as both a source and a binary distribution from our FTP server. Both distributions are packaged as 7zip archives. Programs such as 7-zip can be used to extract these contents. (You can get 7-zip from here: http://www.7-zip.org/ or commandline equivalent p7zip from here: http://p7zip.sourceforge.net/ )

The program is made available under the GPL v.2 or later, copy of which is provided in licence.txt of either distribution. Below follows the summary:

Terms of Use
This software is free and there is no warranty what so ever. Use at your own risk.
See licence.txt for a copy of the program licence.

Copyright (C) 2004, 2005 Vercingetorix <vercingetorix11 at gmail dot com>
Copyright (C) 2011 The Europa Barbarorum Team <webmaster at europabarbarorum dot com>

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details at
http://www.fsf.org/licensing/licenses/gpl.txt

Comments, criticism, improvements and bug reports are welcome. Regards,

The Europa Barbarorum Team.

THANK THE LORD!!!!! :D


I can now finally extract animation idxs on Medieval 2! All I needed was the animation files, now I can repack with ease..... I think.

EDIT, nope, maybe I got this function wrong. But it still doesn't extract them right. It extracts them in name only, if you look at it via hex editor, it is showing them internally as a rome total war cas not a medieval 2 one, You can pack them right, but not extract them.

Need to check what's happening there then as it was only updated to support packing explicitly (which consists of writing the right version stamp, basically since packing should not touch the actual contents of the CAS files).

yeah, if you can figure this out I'll rep you so hard, you'll be on the ground.

Okay it turns out that the old code has this nugget burried in its depths:

Code:

---

bool aidxSubFile::PrepareDataOut(){
    bool result;
    scale = 1.0f / scale;
    result = Scale();
    scale = 1.0f / scale;
    return result;
}   
   
bool aidxSubFile::Scale(){
    if( buffer==NULL ) return false;       
   
    if( (scale != 1.0f) && (buffer[4] == 1) ){         
        //if( CheckProblemCas() ) return true;
             
        //float* pelvis = (float*) &buffer[ (numFrames*numBones*16)+5 ];       
        //float* dist = (float*) &pelvis[ numFrames*3 ];
        //float* root = (float*) &dist[ 4*((numFrames-1) / 2) ];
        //float* end = &root[ numFrames*3 ];
        //for( int i = 0; i < 3*numFrames; i++ ){           
        //    pelvis[i] *= scale;
        //    root[i] *= scale;
        //}   
        //for( int i = 0; i < 4*((numFrames-1) / 2); i++ ){
        //    dist[ i ] *= scale;
        //}
        float* start = (float*) &buffer[ (numFrames*numBones*16)+5 ];
        int loop = numFrames*3*2 + (4*((numFrames-1) / 2)) + 8;
       
        for( int i = 0; i < loop; i++ ){
            start[i] *= scale;
        }   
    }   
    return true;
}

---

This code interprets a data block as 32bit floating point data items which are apparently simple 3d coordinates and it scales the distance to the origin of each coordinate. I'd need to delve into some of the CAS importers/exporters for M2TW to see if this is still “correct”. (You wouldn't happen to have a few links, would you?) If this is not your problem then that would imply that the packs actually contain plain old RTW CAS files...

here, one is what it should be and the second one(the one with 2 in the name) is what is being extracted.
http://www.gamefront.com/files/20899323/MTW2_Swordsman_at_mid_c_slashlr

Thanks for the sample. That will be useful for checking where it goes wrong.

If I understand correctly, this tool can import animations of RTW to M2TW?:hmm: Sorry for the stupid question, I didn't understand properly the features it brought here.:whistling

No, it can do what the old XIDX extractor could do, only better.

Quote:

---

Originally Posted by warman222

EDIT, nope, maybe I got this function wrong. But it still doesn't extract them right. It extracts them in name only, if you look at it via hex editor, it is showing them internally as a rome total war cas not a medieval 2 one, You can pack them right, but not extract them.

---

I think I know what the problem is, now, and it is *not* with the XIDX tool.

I've tested round tripping of your CAS file (pack it as animation archive, unpack it, verify the file contents do not change with a SHA256 hash). That works fine, so my understanding of the XIDX code is correct and no modifications are made when packing the file (absent any scaling).

So I checked the Vanilla archives to see if there was an issue there. Lo and behold: the Vanilla animation packs do not contain any such CAS files as the ones you submitted. Instead, they contain (only) RTW CAS data.

The metadata which is contained in these new M2TW CAS files such as bone names is apparently stripped from the CAS file before packing. Presumably this is what the skeleton pack is for (that DAT file does contain strings which look like bone names such as “bone_Rfoot”), but I know of no tool to generate the skeletons (though there has been some research into that subject).

I do not have the necessary understanding of either the two different CAS file formats and the binary stuff in the skeletons pack to write a tool for that. Do you happen to know if there are any CAS exporter/importer tools floating around that come with source code?

I am no expect on this ethier, but I would try some any of the current importer/exporters for 3dsmax, they are in maxscript format and can be opened up easily.


EDIT

I just notice something, if you use extract skeletons function in xidx and open up one of the new files in a hex editior, it has bone data and the list of animation names, look into that aswell.

Update with the following bug fixes:

Bugs

• Fully zero out the buffers before using them, instead of just assigning '\0' to the first byte. This makes valgrind a bit happier (quieter), which is a good thing.
• Fix a bug in sound packing code. A 1 had crept in where a 4 should have been. Ouch.
• Some code cleanup to have less code in a few macro's.

Misc

• Generate statically linked binaries for Windows. This has the benefit of actually working if the user doesn't have the MinGW version of a couple of DLLs installed on his system (which is likely). The system I used for testing of the previous release masked the problem because these DLLs were present...

Care to post another link to the download link the ftp server seems to be defunct.

Both links work for me; which one appears to be broken for you?

None of the 2 work for me whenever I left click the download button a message comes up saying "Enter password for ftp on ftp://europabarbarorum.org"

:whistlingCould you re-upload them here or on fileserve etc. It would be much appreciated.:thumbsup2

Which means whatever your browser is (probably Firefox), it doesn't understand the concept of anonymous FTP.

I'll edit the links, see if that works?

Thanks man!Much rep:thumbsup2:thumbsup2

Minor update, containing a few miscellaneous fixes and decided to call it 1.0:

• Fix bug in generated file names when extracting event packs caused by not dividing the log(position in the pack) by log(10). This ensures that when you list the files (dir, ls, find) they will appear in the same order as they should be placed in the pack because of how lexicographic sorting works. (Simplifies repacking.)
• Fix version reporting.
• Some minor improvements to the readme.