How do dilemmas work?

[Ying, Duke of Qin]

This is going to be another one of those Ying-floundering-about-helplessly threads.

Long story short, I've figured out something. All the information pertaining to dilemmas seem to be found in the db/dilemmas_tables in the data.pack and in the patch5. pack. However, those are unfortunately not yet moddable. In addition, most of the event-related tables seem to be locked as well - this includes events_tables, events_view_group_junction, and the related ones as well. They're also all found in patch5.

Scriptorama tells us that we should be able to hook events. So how do dilemmas work, then? Do the dilemmas themselves have anything to do with the events?

[Lord Baal]

PM Vragos about this my friend. I discovered something about this last week but I had not the time to poke it around my self just yet, so he might have made some progress on this field.

[davidlallen]

I am interested to know this also. If you get any information by PM, please update here.

EDIT: I tried adding the dilemmas table format to PFM using this tutorial, but on my first attempt, I am not successful. The syntax appears to be:

dilemmas[TAB]ID,String;
DisplayText,String;
Title,String;
LeftIcon,String;
RightIcon,String;
LeftChoice,String;
RightChoice,String;
Unknown1,Boolean;
Unknown2,Boolean

(Yes, my file has a tab character where I have written [TAB] above.)

But when I enter this into the PFM DBFileTypes file, and then try to read the dilemmas table, PFM gives a fatal assertion complaining the table has invalid format. So I must be off by a little. Most of the data here appears to be strings. For example, the first dilemma in data.pack has choices "Domestic Priority" or "Foreign Priority", but it is unclear how these link to anything else such as the effect it will have.

[Lord Baal]

In my mod it is a way to increase the length span of dilemmas. However further feedback is needed since I think only the mission lengths got affected, and not the effects of the dilemmas/missions.

[Yarkis de Bodemloze]

You can access all dilemma/event related files with Taws converter (if you can get it to run). Doing new ones is pretty straightforward if you have all files at your hand. However, you can't do new effect bundles for events, only use or change existing ones. Dilemmas are events which give you a choice of two follow up events.

[davidlallen]

Thanks for the info. I see that one key file inside TAWS is schema.xml, which "should" contain the same information as the DBFileType files of PFM. However, for many cases the entries are different. To pick one random example, advice_levels, we see the following in schema.xml:

<table name='advice_levels_tables'>
<field name='Key' type='string'/>
<field fkey='advice_threads_tables.Key' name='Thread_Name' type='string'/>
<field name='unknown1' type='int'/>
<field name='unknown2' type='int'/>
<field name='unknown3' type='string'/>
<field name='unknown4' type='string'/>
<field name='unknown5' type='string'/>
<field name='unknown6' type='int'/>
<field name='unknown7' type='int'/>
<field name='unknown8' type='boolean'/>
<field name='unknown9' type='int'/>
<field name='unknown10' type='boolean'/>
<field name='unknown11' type='boolean'/>
<field name='unknown12' type='string'/>
<field name='unknown13' type='optstring'/>
<field name='unknown14' type='blob7'/>
</table>

But here is what is in DBFileTypes_2.txt of PFM 1.6.5:

advice_levels Key,String;
Thread IDRef,String;
Unknown,UInt32;
Unknown,UInt32;
Advice Origin,String;
Advice Type,String;
Advice Content,String;
Unknown,15;
Unknown,String;
Unknown,4;
Unkownn,String;
Advisor IDRef,String;
Unknown,Boolean

The instructions for installing and using TAWS are somewhat intimidating with debates about which version of jruby to use. Most of the people posting on that thread seem to have failed to get it to work at all.

On the other hand, PFM seems easy to use. It would be relatively easy to reformat schema.xml into the DBFileTypes file format. Schema.xml contains a lot of tables missing from PFM, including the dilemma tables I was trying to understand. But, if the data already in PFM contradicts what is in schema.xml, how do I know which is right?

[taw]

Thanks for the info. I see that one key file inside TAWS is schema.xml, which "should" contain the same information as the DBFileType files of PFM. However, for many cases the entries are different. To pick one random example, advice_levels, we see the following in schema.xml:

<table name='advice_levels_tables'>
<field name='Key' type='string'/>
<field fkey='advice_threads_tables.Key' name='Thread_Name' type='string'/>
<field name='unknown1' type='int'/>
<field name='unknown2' type='int'/>
<field name='unknown3' type='string'/>
<field name='unknown4' type='string'/>
<field name='unknown5' type='string'/>
<field name='unknown6' type='int'/>
<field name='unknown7' type='int'/>
<field name='unknown8' type='boolean'/>
<field name='unknown9' type='int'/>
<field name='unknown10' type='boolean'/>
<field name='unknown11' type='boolean'/>
<field name='unknown12' type='string'/>
<field name='unknown13' type='optstring'/>
<field name='unknown14' type='blob7'/>
</table>

But here is what is in DBFileTypes_2.txt of PFM 1.6.5:

advice_levels Key,String;
Thread IDRef,String;
Unknown,UInt32;
Unknown,UInt32;
Advice Origin,String;
Advice Type,String;
Advice Content,String;
Unknown,15;
Unknown,String;
Unknown,4;
Unkownn,String;
Advisor IDRef,String;
Unknown,Boolean

The instructions for installing and using TAWS are somewhat intimidating with debates about which version of jruby to use. Most of the people posting on that thread seem to have failed to get it to work at all.

On the other hand, PFM seems easy to use. It would be relatively easy to reformat schema.xml into the DBFileTypes file format. Schema.xml contains a lot of tables missing from PFM, including the dilemma tables I was trying to understand. But, if the data already in PFM contradicts what is in schema.xml, how do I know which is right?

When you see fields types like " Unknown,15;" / "Unknown,4;" on one side, and "<field name='unknown14' type='blob7'/>" on the other - this means both schemas have only bad partial information available to them.

The first 7 fields seem to be in full agreement, the rest not so much.

[davidlallen]

The first 7 fields seem to be in full agreement, the rest not so much.

Right. And the record size of each is different. So one has to be wrong. I have tried several variations of the DBFileTypes file and I cannot get PFM to recognize the dilemma. I converted your schema for dilemma into PFM, and it could not use that either.

To be honest, the number of other setup files such as jruby, nokogiri, etc is a high "barrier of entry" to using your toolset. Also, it is very command-line driven, so I may play around a little more with PFM before trying to install taws and get up to speed on using it.

Has anybody tried to unify these two partial databases? Each one seems to have some field names the other is missing; but I am concerned that many of the record sizes are plainly different.

[taw]

Right. And the record size of each is different. So one has to be wrong.

My converter is always tested against full real data, so I can tell you with certainty that it converts to tsv and back correctly. It might use stupid data types occasionally (such as treating four 1-byte booleans as one 4-byte int32), but record sizes simply must be correct, or automated testing would detect it.

There are tutorials for my toolset. And someone wrote GUI frontend for it. (you still need to install jruby and nokogiri to use it). If someone wants to make it all more accessible, I'm totally willing to help them, but that's way more work than I have time to do myself.

EDIT: Except db<->tsv converter supports advice_levels_tables only for ETW. So yeah, that's about it. Bad example.

[davidlallen]

<field name='unknown13' type='optstring'/>
<field name='unknown14' type='blob7'/>

I'd like to try automatically converting schema.xml to the PFM DBFileType. For these two quoted lines, what is the corresponding value I should use in the DBFileType file? Is "optstring" any different format from "string"? What does "blob N" correspond to in terms of data size?

[taw]

I'm just fixing this table in my converter as we speak because it's obviously broken (blob7 in schema.xml or 7 in pfm
are not real types, just a way of saying "something goes here").

In any case, ETW uses version 1 of this table, NTW/S2TW uses version 3, so there's no way it would work.
If you wait an hour or so it should be fixed.

[taw]

OK, it's fixed. 100% correct schema for all of ETW/NTW/S2TW is:

  <table name='advice_levels_tables'>
    <field name='Key' type='string'/>
    <field fkey='advice_threads_tables.Key' name='Thread_Name' type='string'/>
    <field name='unknown1' type='int'/>
    <field name='unknown2' type='int'/>
    <field name='unknown3' type='string'/>
    <field name='unknown4' type='string'/>
    <field name='unknown5' type='string'/>
    <field name='unknown6' type='int'/>
    <field name='unknown7' type='int'/>
    <field name='unknown8' type='boolean'/>
    <field name='unknown9' type='int'/>
    <field name='unknown10' type='boolean'/>
    <field name='unknown11' type='boolean'/>
    <field name='unknown12' type='string'/>
    <field name='unknown13' type='optstring'/>
    <field name='unknown14' type='boolean'/>
    <field name='unknown15' type='boolean'/>
    <field name='unknown16' type='boolean'/>
    <field name='unknown17' type='string'/>
    <field name='unknown18' type='string' version_start='3'/>
    <field name='unknown19' type='boolean' version_start='3'/>
  </table>

I don't quite remember what PFM used for optstring, it was string;optional or somesuch, just ctrl-F it.

[davidlallen]

I had picked this table just at random, but thanks for fixing it. Once I understand a little better, I'd like to merge schema.xml with the PFM files. A lot of the fields are labeled in PFM, but "unknown" in your schema. OTOH, your schema has many record types which are missing from PFM. So this project seems like it will be helpful for users of either program. Once I merge the data I can put it out it in either PFM format or schema.xml format.

Do you generate schema.xml by hand, or is it automatically extracted from some of the game files?

I compared the field sizes between PFM and your updated schema. There is an "unknown,15" in PFM which corresponds to your unknown6-11. But after that, you have:

<field name='unknown12' type='string'/>
<field name='unknown13' type='optstring'/>
<field name='unknown14' type='boolean'/>
<field name='unknown15' type='boolean'/>
<field name='unknown16' type='boolean'/>
<field name='unknown17' type='string'/>
<field name='unknown18' type='string' version_start='3'/>
<field name='unknown19' type='boolean' version_start='3'/>

and PFM has:

Unknown,String;
Unknown,4;
Unkownn,String;
Advisor IDRef,String;
Unknown,Boolean

This leaves PFM with string, 4 bytes, string, string, boolean. While you have string, optstring, 3 booleans, string, string, boolean.

Can you explain more about "optstring"? I don't see how it can be one byte. I didn't understand your comment about "just ctrl-F". Among all the DBFile files, I can only see these data types: Single (float), UInt32, Int32, String, Boolean. I have not been able to find any specific documentation of the PFM ""format"", although it is possible there is an old thread somewhere in the NTW, ETW, etc forums I haven't found yet.

[taw]

Optstring field is either byte 0 (NULL) or byte 1 followed by string (2 bytes length, 2 bytes per character).
Yes, there's an optstring there. It is one byte most of the time for this field.

PFM describes them with silly -> notation.

Also I think PFM deals with table schema versions by listing all possible schemas
and guessing somehow, not by looking in headers which version it is.

battlefield_buildings   Building ID,String;                                                                           
Type,String;                                                                                                          
Building 2 ID,String;                                                                                                 
Material,String;                                                                                                      
unknown,UInt32;                                                                                                       
->,Boolean,1;Fort type,String;                                                                                        
->,Boolean,1;Terrain,String     


  <table name='battlefield_buildings_tables'>                                                                         
    <field name='Battlefield_Building_ID' pk='true' type='string'/>                                                   
    <field name='Type' type='string'/>                                                                                
    <field fkey='battlefield_buildings_tables.Battlefield_Building_ID' name='Building_2' type='string'/>              
    <field name='Material' type='string'/>                                                                            
    <field name='Hit_points' type='int'/>                                                                             
    <field name='unknown6' type='int' version_start='3'/>                                                             
    <field name='unknown7' type='int' version_start='3'/>                                                             
    <field name='Fort_type' type='optstring'/>                                                                        
    <field name='Terrain' type='optstring'/>                                                                          
  </table>

[davidlallen]

I was starting to guess that -> meant something special. As far as I can tell (from the outside, not the source code) PFM has this field in the schema, but never displays it in the data. That is, there is no row for it in the right hand panel of PFM. I am not sure yet what happens when you export this type of table to TSV using PFM, then edit, then re-import.

I agree that PFM does not try to take the version number into account. Is it true that TWS2 uses the latest version of every schema? So if I ignore the version_start in your xml, the result will be OK for TWS2? It might not be OK for ETW or NTW, those might use older versions. This is probably the assumption that PFM uses also.

Where does your schema file come from? I should be able to generate a new one with the improved string data from pfm in the next couple of days. But if your schema is auto-generated then this data will be lost.

In your schema, many strings have "pk=true", but not all. First example is:

<table name='advice_levels_tables'>
<field name='Key' type='string'/>

Does it mean something different, if "pk=true" is missing?

[taw]

I was starting to guess that -> meant something special. As far as I can tell (from the outside, not the source code) PFM has this field in the schema, but never displays it in the data. That is, there is no row for it in the right hand panel of PFM. I am not sure yet what happens when you export this type of table to TSV using PFM, then edit, then re-import.

I agree that PFM does not try to take the version number into account. Is it true that TWS2 uses the latest version of every schema? So if I ignore the version_start in your xml, the result will be OK for TWS2? It might not be OK for ETW or NTW, those might use older versions. This is probably the assumption that PFM uses also.

Different patch files in same game often use different version.
And you still need to know what version number to put in db table you create.
I collect db files for testing, and I've seen some mod files with broken headers,
that might be old ETW's PFM before people started using DB Editor which handles
it correctly.

Where does your schema file come from? I should be able to generate a new one with the improved string data from pfm in the next couple of days. But if your schema is auto-generated then this data will be lost.

I have scripts to convert schema.xml <-> DB.xsd.

I can merge schema.xml with updates and no data will be lost.

I thought about merging PFM's data, but they use duplicate column names,
have no version information, so it would be a major manual work to get
it working.

Oh yeah, don't use duplicated column names in schema.xml, not even unknown.

In your schema, many strings have "pk=true", but not all. First example is:

<table name='advice_levels_tables'>
<field name='Key' type='string'/>

Does it mean something different, if "pk=true" is missing?

Some fields are part of the primary key, this are indicated by pk attribute.

If you have table in multiple parts (units_tables/units units_tables/Darthmod_units),
rows with same pk value are overridden, other rows are just added.

What is and what isn't in pk is almost pure guesswork, but it matters, and I've seen a few surprising cases
(iirc research bonuses table doesn't pk anything, so +4 bonus and +6 bonus for the same in mod table
mean +10 total bonus not +6 or something like that).

DB Editor (for ETW) supports this.

[davidlallen]

Thanks for the info. Where does DB.xsd come from?

Looking further into PFM, it seems it does have the ability to manage different versions of a table. For example, DBFileTypes_0.txt contains all the version 0 data; DBFileTypes_1.txt contains all the version 1 data for tables which have a version 1, and so forth. This makes the data collection I need much more complicated, and comparing to make sure that it matches to your version_start data will also be a challenge. But at the end, I can transfer any compatible, useful string data from PFM to schema.xml, and hopefully generate a full set of PFM files which will contain many new tables.

By my count there are 234 table formats which completely match between PFM and schema.xml, 136 where there is schema.xml data but no PFM data at all, and 76 where they contain contradictory formats. So it may be that I can add 136 new formats to PFM. Three or four will be dilemma tables, which was my original goal.

[taw]

About half of DB.xsd came from DB Editor, the other half from my schema autodetector.

Long lists of unknownN come from autodetector, I definitely want names for them.

IIRC correctly I miss about 16 S2TW tables, and PFM has a few of those,
while I have only ETW or ETW/NTW versions.
76 sounds like much. What kind of differences are these?

I can tell you about a few common issues to expect:

• If it's uint vs int, just ignore that, this never actually matters and is pure guesswork most of the time.
• A lot of columns are all 00 00 00 00, so int32 vs float is pure guesswork there. If there are any non-zero cases, autodetector can guess correctly almost all of the time if it's int32 or float. Usually when autodetector doesn't have a clue, neither do humans.
• Sometimes you have byte that's always 00, no idea if it's boolean or optstring. Humans make this mistake all the time, when they look at one table, see a lot of records with 0s and think it's boolean, autodetector can see that 1 record out of 10000 has 01 there and boolean wouldn't fit.
• In big blocks of booleans and small positive ints (like that 15 byte block in advice table PFM had no idea what to do with), both humans and autodetector can miss frame and make some completely wild guesses. That particular one is now correct, but I've occasionally stumbled upon such mis-aligned reading frames and had to fix them manually. Yeah, these would need some fixing. I think PFM is more likely to have those than schema.xml, especially if they took their schema from DB Editor, I fixed quite a few of those already.
• Any blobN / unknown,<number> are obviously wrong. There's some actual columns there.

Well, good luck. If you need any help, just ask.

[taw]

One thing to add - remember to use the most recent schema.xml.
Merging complicated changes in hard, let's avoid that if possible.

[davidlallen]

Further status updates on merging PFM with TAW in this thread:

http://www.twcenter.net/forums/showthread.php?t=470403

(EDIT: I wrote a tutorial for dilemmas, which was my original question:
http://www.twcenter.net/forums/showthread.php?t=475177)