View Poll Results: Do you remember XIDX?

Voters
149. You may not vote on this poll
  • I have used it.

    18 12.08%
  • I have heard of it but never used it.

    13 8.72%
  • What is XIDX?

    118 79.19%
Results 1 to 20 of 20

Thread: Stele 9, another old name: XIDX

Hybrid View

Previous Post Previous Post   Next Post Next Post
  1. #1

    Default Stele 9, another old name: XIDX

    Let us start with a simple roll call: please raise your hand if you remember XIDX, what it is and what it does?

    For those who do not know: XIDX is a program to extract the contents from pairs of Rome Total War IDX/DAT files, and repackage files as IDX/DAT pairs. It was one of the earliest modding tools around, and it was created by a modder nicknamed Vercingetorix. Vercingetorix may no longer spend his days dissecting and reverse engineering file formats created for the Total War engine, but he did something fairly unique in the TW community which ensured his work is still of great value: he released the source code for XIDX under an open source licence. (Of course Vercingetorix is most famous for his work on the CAS file format.)

    Why is his decision to release the source code so important? Well, it turns out that with Medieval II Total War the same types of files were used, again. The tool continued to work beautifully, but there was a catch: the game did not accept packs created by XIDX anymore. What had happened was that the Creative Assembly had made a tiny modification to the file format which amounted to updating the version stamp included in the first few bytes (used by the game to identify the file format).

    Additionally, it also turned out that there were a few problems with XIDX. For one thing, sometimes it crashed mysteriously on Windows Vista or Windows 7. For another, sometimes the names of skeletons in the skeletons IDX/DAT pair clashed with the names of directories used to store the related animations files. For a third, it only ran on Windows. That will probably not be an issue for most of our fans or most of the TW modders out there, but it is a problem for us when we want to use XIDX to build IDX/DAT pairs on our Linux server!

    However, because the source code for XIDX was freely available to us we were able to make changes to the tool and fix bugs (including the crashes). We also updated it to support the Medieval II file formats, added support for a new type of IDX/DAT pair (events.dat, and events.idx), and implemented a scheme for the tool to add/strip file extensions when packing/unpacking files. (That last feature can be used to circumvent the issue with skeleton names mentioned earlier).

    Finally, because the code was available to us under an open source licence, we can now pass on our modified version to you as well, so you may benefit from it.

    Regards,
    Last edited by Tellos Athenaios; October 17, 2011 at 06:10 PM. Reason: Tool is released
    -Tellos Athenaios
    CUF tool - XIDX - PACK tool - SD tool - EVT tool

    ὁ δ᾽ ἠλίθιος ὣσπερ πρόβατον βῆ βῆ λέγων βαδίζει” – Kratinos in Dionysalexandros.

  2. #2

    Default Re: Stele 9, another old name: XIDX

    First
    ok so that was fun, i wasnt around when Vercingetorix was, although i did no he was famous for some pretty exceptional stuff. What exactly does this tool allow you to do? From what i understand it makes editing animations easier?
    sorry if that questions a bit dumb :/
    Last edited by David93; October 17, 2011 at 06:30 PM.

    The Orcs of Gundabad Erin go Bragh FROGS

    When I came back to Dublin I was court marshaled in my absence and sentenced to death in my absence, so I said they could shoot me in my absence"
    Brendan Behan
    The Irish won an Empire
    The Scots ran an Empire
    The English lost an Empire

    "When I told the people of Northern Ireland that I was an atheist, a woman in the audience stood up and said, 'Yes, but is it the God of the Catholics or the God of the Protestants in whom you don't believe?"
    - Quentin Crisp

    There is one weapon that the British cannot take away from us: we can ignore them.
    - Michael Collins

    They have nothing in their whole imperial arsenal that can break the spirit of one Irishman who doesn't want to be broken.
    - Bobby Sands

  3. #3

    Default Re: Stele 9, another old name: XIDX

    Interesting. I must say I appreciate these tiny updates regarding the mod's development on other branches than just gameplay/models/textures. Keep up the good work
    YATS: Manius Aquillius Tuscus, 46, Patrician

  4. #4
    The excited one's Avatar Senator
    Join Date
    Mar 2011
    Location
    australia
    Posts
    1,014

    Default Re: Stele 9, another old name: XIDX

    okkkkk but the problem is that there are people who uses win7 (fanboys) like me its not like everyone of the gamers in here are using something that is not win7 and vista
    war is peace, ignorance is strength, freedom is slavery......
    (george orwell 1984)

  5. #5

    Default Re: Stele 9, another old name: XIDX

    Quote Originally Posted by David93 View Post
    What exactly does this tool allow you to do?
    Both RTW and M2TW use a certain types of “archives” which pack together a bunch of files. They come in various flavours but have in common that they consist of two files: one .idx (dot idx) and one .dat (dot dat) file. These archives are used for sounds, animations, skeletons (meta data for animations) and certain meta data to do with the text files which control sounds.

    Vercingetorix wrote XIDX to unpack the archives into files and vice versa. This is an update which extends what the tool can do for you, and fixes numerous reliability issues at the same time. I'm not sure if you read Stele 8, but this tool is very valuable for us when dealing with issue #15 mentioned there.

    Quote Originally Posted by The excited one
    okkkkk but the problem is that there are people who uses win7 (fanboys) like me its not like everyone of the gamers in here are using something that is not win7 and vista
    I am not sure I follow you, here? The tool should run on Vista or 7 just fine. It's unlikely that the binary distribution does not contain a suitable version of the tool if you run Windows there are two Windows exes provided:

    - For i686 (think: Intel Core2Duo and in general any CPU's from that age.)
    - For AMD64 (think: any common 64bit desktop CPU on the market, needs 64bit Windows though.)

    The Linux executable is for AMD64 and therefore needs AMD64 bit flavour of Linux.
    Last edited by Tellos Athenaios; October 18, 2011 at 06:32 PM.
    -Tellos Athenaios
    CUF tool - XIDX - PACK tool - SD tool - EVT tool

    ὁ δ᾽ ἠλίθιος ὣσπερ πρόβατον βῆ βῆ λέγων βαδίζει” – Kratinos in Dionysalexandros.

  6. #6
    The excited one's Avatar Senator
    Join Date
    Mar 2011
    Location
    australia
    Posts
    1,014

    Default Re: Stele 9, another old name: XIDX

    it works on win7 !? thank god for it i thought this thing only works on XP or even before.............. thanks
    war is peace, ignorance is strength, freedom is slavery......
    (george orwell 1984)

  7. #7
    Slaxx Hatmen's Avatar This isn't the crisis!
    Join Date
    Feb 2005
    Location
    The Living End
    Posts
    3,081

    Default Re: Stele 9, another old name: XIDX

    Hmm. Tried opening the binary .exe files and all I get is a quick flash of the program then it immediately closes. Did I install this right?(I have Windows 7 & an Intel Core i3 processor)
    Under the patronage of Basileos Leandros I

  8. #8
    AnthoniusII's Avatar Μέγαc Δομέστικοc
    Join Date
    Feb 2007
    Location
    Thessalonike Greece
    Posts
    18,974

    Default Re: Stele 9, another old name: XIDX

    Please let me see if i understood this corectly.
    Is this tool to unpack/repack animation and skeleton IDX,DAT packages?
    It solves the problemw that apeared when someone tried to compine in M2TW Caliban's animation pack with new ones?
    Can this tool convert a new animation to a M2TW compatible one?
    EDIT : I tried to contact JMRC in .org in order this moding team to help us with some animations but i had no reply.
    Is this tool an answer to such problems?
    Last edited by AnthoniusII; October 31, 2011 at 02:33 AM.
    TGC in order to continue its development seak one or more desicated scripters to put our campaign scripts mess to an order plus to create new events and create the finall missing factions recruitment system. In return TGC will give permision to those that will help to use its material stepe by step. The result will be a fully released TGC plus many mods that will benefit TGC's material.
    Despite the mod is dead does not mean that anyone can use its material
    read this to avoid misunderstandings.

    IWTE tool master and world txt one like this, needed inorder to release TGC 1.0 official to help TWC to survive.
    Adding MARKA HORSES in your mod and create new varietions of them. Tutorial RESTORED.


  9. #9

    Default Re: Stele 9, another old name: XIDX

    Quote Originally Posted by the celt View Post
    Hmm. Tried opening the binary .exe files and all I get is a quick flash of the program then it immediately closes. Did I install this right?(I have Windows 7 & an Intel Core i3 processor)
    The tool does not feature a GUI. Please refer to the readme file for usage instructions.

    Quote Originally Posted by AnthoniusII View Post
    Please let me see if i understood this corectly.
    Is this tool to unpack/repack animation and skeleton IDX,DAT packages?
    Yes, as well as sound and event IDX/DAT packages.

    It solves the problemw that apeared when someone tried to compine in M2TW Caliban's animation pack with new ones?
    I am not aware of what problem this might be, but I assume that it doesn't.

    Can this tool convert a new animation to a M2TW compatible one?
    This tool does not feature a converter. However it appears that the animation packs themselves contain only RTW style CAS data, and that the other information in M2TW CAS formats is used to generate the skeleton packs instead.

    EDIT : I tried to contact JMRC in .org in order this moding team to help us with some animations but i had no reply.
    Is this tool an answer to such problems?
    No. This tool is an update to XIDX to add support for the M2TW specific modifications to the IDX/DAT file formats themselves, and make it run on a wider range of OS/architecture.

    Please refer to the readme for details and usage instructions.
    -Tellos Athenaios
    CUF tool - XIDX - PACK tool - SD tool - EVT tool

    ὁ δ᾽ ἠλίθιος ὣσπερ πρόβατον βῆ βῆ λέγων βαδίζει” – Kratinos in Dionysalexandros.

  10. #10
    Slaxx Hatmen's Avatar This isn't the crisis!
    Join Date
    Feb 2005
    Location
    The Living End
    Posts
    3,081

    Default Re: Stele 9, another old name: XIDX

    Quote Originally Posted by Tellos Athenaios View Post
    The tool does not feature a GUI. Please refer to the readme file for usage instructions.
    Very funny Tellos. Yes, the DOSS window opens then closes before I can enter any commands. Does it only work for animation IDX files?(I was trying it out on the sound files since they were in the same format I thought.)
    Under the patronage of Basileos Leandros I

  11. #11

    Default Re: Stele 9, another old name: XIDX

    Quote Originally Posted by the celt View Post
    Very funny Tellos. Yes, the DOSS window opens then closes before I can enter any commands. Does it only work for animation IDX files?(I was trying it out on the sound files since they were in the same format I thought.)
    What makes you think I was trying to make fun of you? The readme is quite clear.
    Spoiler Alert, click show to read: 

    ----------------------------------------------
    Installation
    ----------------------------------------------

    If you use a source distribution of the XIDX project you can simply extract the 7zip archive and proceed with make. For more information, see the
    “Building from source” section below.

    If you use a binary release of the XIDX project you can simply extract the 7zip archive which should give you the following versions of the XIDX program:
    - bin/xidx:
    This is an AMD64 ELF executable. It should run on Linux (AMD64).
    - bin/xidx_i686.exe:
    This is an i686 PE executable. It should run on 32bit and 64bit versions of Windows OSes (2k, XP, Vista, 7, Server R2 etc.) and Wine.
    - bin/xidx_amd64.exe:
    This is an AMD64 PE executable. It should run on 64bit versions of Windows OSes (XP, Vista, 7, Server R2 etc.)

    The project does not provide any rule to install it as system administrator, so you may want to select your preferred executable and move it manually to a
    suitable location. It makes sense to ensure that this location is on the search PATH used for looking up programs by name, since XIDX is purely a
    commandline program. (Therefore being on the search PATH will make it easier to invoke the program from arbitrary directories.)

    ----------------------------------------------
    Usage
    ----------------------------------------------

    If you use a platform that is not supported by either of these binaries you can try to build from source instead.

    program usage: [options] [files...]
    command line options:
    -a idx pack(s) are animation packs, by default they are assumed to be sound
    -s idx pack(s) are skeleton packs
    -m idx packs(s) are Medieval II packs, only applies when creating packs.
    -c create an idx pack file
    -x extract idx pack file(s)
    -t list the contents of idx pack file(s)
    -f name of idx pack when creating. (output filename must follow)
    -v verbose output
    -b adds '.bin' suffix to filenames when extracting, removes it when packing
    --help print the help
    --shell gives you a simple 'shell' for entering commands

    You may only specify one of the three "xtc" options. When you create a idx pack the
    filenames can be specified on the command line (after your options). If no files are
    on the command line, it will read filenames from stdin (standard input) with one filename
    per line.

    As a note regarding repacking pack.idx: There are some filenames that have spaces at the end,
    as well as duplicate files in the pack.idx. Unfortunatly we have to deal with this and a listing
    of the data\animations folder will not suffice. I recommend that you save the file list of the
    vanilla pack before you do any editing. ".\xidx -ta >filenames_in_pack_idx.txt" would do the trick.
    When you go to repack the animations you would use this list as the input for filenames.
    i.e. ".\xidx -caf pack.idx <filenames_in_pack_idx.txt".


    Examples:
    1) .\xidx -x music.idx
    2) .\xidx -t music.idx >filenames_in_music_idx.txt
    3) .\xidx -c -f my_music_idx.idx <filenames_in_music_idx.txt

    The first example extracts all the files within music.idx (relative to the current working directory).
    The second example lists all the files within music.idx and saves the output to filenames_in_music_idx.txt
    The third example creates a new idx pack with the name my_music_idx. The files that are to
    be archived are within filenames_in_music_idx.txt which is redirected to stdin.

    In short, the above three commands extracts the music, lists the the files within the pack
    and the repacks them. Logically you would overwrite the vanilla sound files with your own files
    before you run step 3.

    If you wanted to pack all the files in a folder you could do:
    dir .\data\sounds\SFX\* /a:-D /s /b | .\xidx -cf my_sfk.idx
    Where you would change ".\data\sounds\SFX\*" to the folder of your choice.
    -Tellos Athenaios
    CUF tool - XIDX - PACK tool - SD tool - EVT tool

    ὁ δ᾽ ἠλίθιος ὣσπερ πρόβατον βῆ βῆ λέγων βαδίζει” – Kratinos in Dionysalexandros.

  12. #12

    Default Re: Stele 9, another old name: XIDX

    Quote Originally Posted by Tellos Athenaios View Post
    It was one of the earliest modding tools around, and it was created by a modder nicknamed Vercingetorix. Vercingetorix may no longer spend his days dissecting and reverse engineering file formats created for the Total War engine
    Ah.. thems were the days. Did Paul end up in the military?
    PSYCHO V



    "Ask not for whom the bell tolls, it tolls for THEE!" - (John Donne, Meditation 17)

  13. #13
    Shigawire's Avatar VOXIFEX MAXIMVS
    Join Date
    Aug 2003
    Location
    Norway (NORGE), BRØNNØYSUND
    Posts
    3,458

    Default Re: Stele 9, another old name: XIDX

    This is very good work!

    Psycho:
    I think Paul (Verx) started work for Oracle in conjunction with Mastercard or some such financial company. I can't recall the details.
    ------------------------------VOXIFEX MAXIMVS-------------------------------
    ------PROUD PARENT OF THE EUROPA BARBARORUM VOICEMOD-------


    "To know a thing well, know its limits. Only when pushed beyond its tolerances will its true nature be seen." -The Amtal Rule, DUNE

  14. #14

    Default Re: Stele 9, another old name: XIDX

    Quote Originally Posted by PSYCHO V View Post
    Ah.. thems were the days. Did Paul end up in the military?
    As a matter of fact, I don't know: Vercingetorix had already quit modding well before I even started...

    Quote Originally Posted by Shigawire View Post
    This is very good work!
    Cheers. Actually there's work going on to fix the missing link between voicemod source code and binary dumps which XIDX can pack into event packs. That sort of thing turns out not be merely useful for technical point scoring, it has also yielded a few new insights on how voicemod code works and where its limits are.
    -Tellos Athenaios
    CUF tool - XIDX - PACK tool - SD tool - EVT tool

    ὁ δ᾽ ἠλίθιος ὣσπερ πρόβατον βῆ βῆ λέγων βαδίζει” – Kratinos in Dionysalexandros.

  15. #15
    Shigawire's Avatar VOXIFEX MAXIMVS
    Join Date
    Aug 2003
    Location
    Norway (NORGE), BRØNNØYSUND
    Posts
    3,458

    Default Re: Stele 9, another old name: XIDX

    Quote Originally Posted by Tellos Athenaios View Post
    Cheers. Actually there's work going on to fix the missing link between voicemod source code and binary dumps which XIDX can pack into event packs. That sort of thing turns out not be merely useful for technical point scoring, it has also yielded a few new insights on how voicemod code works and where its limits are.
    Oh really? So you've learned something new on the voicemodding front.. that is nice. Generally speaking, what have you learned?
    ------------------------------VOXIFEX MAXIMVS-------------------------------
    ------PROUD PARENT OF THE EUROPA BARBARORUM VOICEMOD-------


    "To know a thing well, know its limits. Only when pushed beyond its tolerances will its true nature be seen." -The Amtal Rule, DUNE

  16. #16

    Default Re: Stele 9, another old name: XIDX

    Quote Originally Posted by Shigawire View Post
    Oh really? So you've learned something new on the voicemodding front.. that is nice. Generally speaking, what have you learned?
    There are bugs in the binary structure, or rather the M2TW parser does not signal an error when two distinct attributes in the code map to the same field in the binary structures. Additionally order of certain code attributes is vital if they are to have any effect at all. This is from my notes about the binary format.
    Code:
    +30 32bit float.
    	if 3d:
    	  mindist, default is 1.0 (0x3f800000). 
    	  according to descr_sounds.txt mindist must be > 1.0
    	if 1d:
    	  0 (0x0), default.
    	bitmask:
    	  If {streamed, ducking} is set and mindist is not specified or 
    	  if {streamed, ducking} follows the mindist declaration, apply: value | {0x1, 0x2}
          +34 32bit float (maxdist or pan)
    	if 3d: 
    	  maxdist, default is 1.0 (0x3f800000) ? 
    	  If pan is set and maxdist is not specified, the value will be pan or
    	  if pan follows the maxdist declaration, the value will be pan instead.
    	  
    	  pan must not be specified if maxdist is.
    	  presumably, must be >= mindist, therefore must be > 1.0
    	if 1d: 
    	  pan. according to descr_sounds.txt must be between 0.0 and 1.0 (inclusive).
    	  maxdist must not be specified.
          +38 32bit float (dry_level or distancepriority)
    	last one of dry_level, distancepriority is used. the two conflict.
          +42 32bit float (wet_level or probradius)
    	last one of wet_level, probradius is used. the two conflict.
          +46 32bit float (effect_level)
    What the above blurb says (among other things) is that if you specify the “streamed” attribute in code (which you need to do in order to tell the engine not to load the sound in main memory at startup), you will find that with 3d positional stuff (e.g. on the battle map) you must specify “streamed” after “mindist”. If you do it the other way around, the parser will appear to forget about your “streamed” attribute.

    The same goes for the “ducking” attribute.
    -Tellos Athenaios
    CUF tool - XIDX - PACK tool - SD tool - EVT tool

    ὁ δ᾽ ἠλίθιος ὣσπερ πρόβατον βῆ βῆ λέγων βαδίζει” – Kratinos in Dionysalexandros.

  17. #17
    The excited one's Avatar Senator
    Join Date
    Mar 2011
    Location
    australia
    Posts
    1,014

    Default Re: Stele 9, another old name: XIDX

    its all greek to me
    war is peace, ignorance is strength, freedom is slavery......
    (george orwell 1984)

  18. #18
    Shigawire's Avatar VOXIFEX MAXIMVS
    Join Date
    Aug 2003
    Location
    Norway (NORGE), BRØNNØYSUND
    Posts
    3,458

    Default Re: Stele 9, another old name: XIDX

    Well Tellos, so you've learned some features of minimum/maximum distance probabilities of the voices - and found some bugs in it. Can you fix the bugs, and can it be used for something in the end?
    ------------------------------VOXIFEX MAXIMVS-------------------------------
    ------PROUD PARENT OF THE EUROPA BARBARORUM VOICEMOD-------


    "To know a thing well, know its limits. Only when pushed beyond its tolerances will its true nature be seen." -The Amtal Rule, DUNE

  19. #19

    Default Re: Stele 9, another old name: XIDX

    Quote Originally Posted by Shigawire View Post
    Well Tellos, so you've learned some features of minimum/maximum distance probabilities of the voices - and found some bugs in it.
    Well those are not really new features, but what we know now is which pairs of attributes cannot be used together. Therefore the tool will be able to check and throw errors when you attempt to declare two different attributes that map to the same field in the binary structure. This is important as it removes ambiguity from code by rejecting ambiguous code.

    Can you fix the bugs, and can it be used for something in the end?
    The bitmask ones can be fixed as it is simply a matter of being more careful than the M2TW parser with these values. But you have to keep in mind that the binary structure itself is fixed as that is what the M2TW engine expects.

    In terms of features the most we have learned tends to involve RTW features that have been confirmed. For instance despite the fact that M2TW doesn't use most of the RTW export_descr_sounds_prebattle triggers but the M2TW parser still recognises them and still generates the necessary binary objects for them. That means in the long term that the EB 1 pre battle speeches could move forward as EB 2 pre battle speeches.
    Last edited by Tellos Athenaios; November 04, 2011 at 11:19 AM.
    -Tellos Athenaios
    CUF tool - XIDX - PACK tool - SD tool - EVT tool

    ὁ δ᾽ ἠλίθιος ὣσπερ πρόβατον βῆ βῆ λέγων βαδίζει” – Kratinos in Dionysalexandros.

  20. #20

    Default Re: Stele 9, another old name: XIDX

    And so forth we will remember Vercingitorix the Modder... All hail and remember vercingetorix the Modder for the duration of human Existince..

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •