Official CA Q&A thread - XML and XSD modding

[CraigTW]

Hi guys,

Here are some answers to the first batch of questions that were lost when the thread disappeared. I'll get back to you soon with more, and with answers to the rest of the questions in this thread.

Sorry to say that I didn't record the usernames of those who posted when I copied the questions, so you'll have to try to remember which post was yours! If your question hasn't been answered yet, I'll mark it with "will answer soon", or words to that effect.

Initial answers provided by Jack, below:

NTW
About unit classes
I've already been answered that it's not possible to add a unit class. However, we have in NTW 22 classes for land units, and I was thinking to reuse some of them (like elephant).
Question : what is the exact effect of a unit class in game?
I think it only change
- the default abilities from unit_class_to_unit_ability_junctions
- the name of newly built unit (regiment names)
And that's it... So if I tkae Elephant and rename them cuirassier, I don't think it would generate much side effect, would it?

Classes are used in the group formations to work out where units will be placed. They are also used in the AI to help it work out what units are good against others.

About regiment names
To have specific names, it seems you need to edit the startpos, with name allocator for each faction, where you have 22 lists (one for each class), and can put your regiment names.
However, there is a table for unit_regiment_names... But I have the impression it's useless. can you confirm if it's possible to manage the regiment names via the DB?

The info from that table is built into the campaign when the startpos is built. Its where that information is got from. Will ask the campaign team for more info.

About voices
We have an annoying bug, where every time we mod a unit, the voices become messed up. Like select an artillery unit, and it says "Swiss infantry". It is something that we could correct with the DB, or is it really a bug?

For Empire and Napoleon setting the voice field in the units table to an appropriate entry should help with this. Some of it may be hardcoded. More info now added in a separate post, below.

About generals
I couldn't find where you can get the list of generals that could go in the recruitment pool, with their associated unit.

historical_characters table deals with this. Generals unit is decided by culture of the faction and defined in the agent_culture_details table

About effect
I tried to add a new effect to a building
- upkeep_cost_mod_land_all
But it seems it has no effect. Is this effect limited to ministers?
How can we know what effect will work in a town, a settlement or with ministers?

UPDATED with answer from campaign team:

As for effects, there are a number of effects which only work under specific circumstances, ie as tech effects, trait effects, local building effects, factionwide building effects, etc. I cannot begin to list which ones work where, especially in ETW and NTW, as much has changed and been added to the system since then. Trial and error will be necessary; best to mess around with adding effects which are known to work on buildings both locally and factionwide and see what happens.

In the case listed here, it’s most certainly an effect which will only work when applied factionwide, so you need to put it in the building_factionwide_effects_junctions table rather than the building_effects_junction table.

Our version of pfm shows the columns in a different order than your schema lists them for certain tables (a very small few). When your xmls are compiled is the order distorted?

They are outputted with the order they are in our database.

Could you give us a rough list of columns ignored by the game? From previous interaction with you guys I've worked out the unused ones in unit_stats_land, but I'm sure there are more. There are also some (obviously) completely unused tables, such as diseases, disasters, etc. Could you give us a rough list of tables that we may be editing but are unused?

This is a list of ones I know. Will get more info from campaign team.

Unused tables:
· unit_to_gov_type_permissions
· unit_to_gov_type_conversions_jcts

Units
· fitness not used

Projectiles
· bounce_angle not used
· collision_radius not used

Could you download our tools and see if/where we are going wrong? I can supply my Beta version of Pack File Manager that I've been updating with your info if you like, as there are many different versions out there and you may d/l an old version.
You have many xmls/xsds that we've never seen in game packs before (eg genders, regions_and_mercenaries). Were these omitted, or are our tools missing them? I'm guessing omitted, but clarification would be nice

If they do not exist as tables in the pack files, they are not used by the game.

Is message_event_strings partially hardcoded? We've tried adding entries here, and scripting the message events to appear in game, but with no luck. I can assure you the bug wasn't on the scripting side.

New message events require setting up in code, I’m afraid. Best bet (for NTW at least) would be to repurpose the scripted mission message events, of which there are a fair number. There may also be other defunct messages that they can use should these not prove sufficient. They will still have to be triggered through script.

More follows in next post.

[CraigTW]

ETW
Is it possible to split a region into two regions without editing the pathfinding.esf, for example splitting Syria into Cyprus and Syria? In the new region one of the existing buildings will be turned into a settlement, such as the Paphos Iron Mine in Cyprus, so there's no need to add additional buildings.
In the grid_data part of the pathfinding.esf is there a generic code that can be added that will enable modders to add new towns, ports, or resources to the map? A generic code that requires the modder to make some changes to it, such as adding the coordinates of the new towns, ports, or resources, is fine as long as we're told what needs to be changed.

Requires an answer. Talking to campaign team.

Is there a way in NTW to reactivate the restriction of the camera to a limited distance from your units with one of the XML file?

No.

Lordz crucial question for MP
1) Can we mod square in any way via db files to increase the instant bonus given to infantry. When negative, can you please state a price for having your programmers to add a moddable delay to square and introduce a column formation morale bonus?

No, and no price for it. It won’t happen

2) Any chance we can have custom map "farms.farm_manager" file explained as you did for db files?

Have asked the relevant member of the team.

3) are building type fixed or can we add new type in the battlefield_buildings table?

Should be addable I think.

As the developer of a tool to edit CUF fonts I was wondering if a CA developer might be able to answer a few questions. The CUF files are the fonts used by anything from M2TW to even S2TW a good understanding of how they work will help fix all sorts of text rendering issues that crop up once you start modding the CUF files.
On the CUF file format
Quite a few of the CUF “properties” in the first table are poorly understood by me and the one other person I've encountered on the Internet who I know has been looking into this format as well. He/she goes by the handle of just on the TWC. That discussion is over at the TWC, starting with this post to be precise. Below is a summary of what just and I so far found out (it's a few forum posts collated into one, so “I” or “me” might refer to either myself or to just). Could you please clarify what these properties refer to, or correct our misunderstanding of the format where appropriate?
Summary of what we do (not) know so far:
[spoil]
• 4 bytes magic 'CUF0'
• short 0: unknown
• short 1: unknown, seems to always match the height of a capital, but is not used for rendering
• short 2: line height, seems to be two pixels lower than the actual line height.
I think this is used for calculating/allocating a proper height of the entire rendered image (block of text) so that item which extend (far) below the baseline are not cut off.
• short 3: unknown
• short 4: unknown
• short 5: seems to have something to do with baseline, but there is a maximum and a minimum, in one font I tested changing default 7 to 0xF resulting in the text moving down two pixels, while 7 to 1 results in moving up by five pixels.
I think it is the baseline, or at least an equivalent value. Whether signed or unsigned I do not know.
• short 6: also seems related to baseline, changing this matches the changed baseline exactly.
I call it LayoutYOffset. It is a signed short (rather than the unsigned ones that most of the others are). The absolute value is an offset as described earlier, where 0<= offset <= 0x7ff. If the bit corresponding to 0x8000 is set then it is a negative offset, otherwise positive.
• short 7: how wide a space is for justification and text wrapping calculations, the width for rendering a space is as it is specified in the glyph properties and/or kerning.
• short 8: how much less wide a line (or maybe the dynamically sized container with the line) is than the sum of the glyph sizes. If you use a value that is larger than the width of the first word and following space then there is an extra line wrap after the first word and space. In this specific case the space is rendered for justification reasons, normally the space that turns into a line wrap is not rendered/calculated.
I thought it was the horizontal equivalent of #5. At any rate it appears to be a signed short, too; but the above interpretation explains some of the behaviour I saw better.
• short 9: the largest width of any glyph in the font. Anything to the right of this amount of pixels is not rendered.
• short 10: the largest height of any glyph in the font. Anything below this amount of pixels (calculated from the top of that glyph) is not rendered.
• short 11: the number of glyphs
• int 12: total size of the glyph bitmaps
• Array of 65536 shorts/[w]chars. Position in the array indicates UTF-16LE/UCS2 code point, 0xFFFF indicates “not supported”. I call it “chartable” since that's basically what it is. Values correspond to offsets in the following tables.
• for each glyph
o byte 0: starting height of the glyph in pixels above the baseline, 0x80 indicates the glyph doesn't have a height.
o byte 1: Width that should be allocated to the glyph: this is used to advance to the position at which the next glyph should be drawn.
o byte 2: width of the bitmap data of the glyph
o byte 3: height of the bitmap data of the glyph
• for each glyph
o int: offset that indicates where in the bitmap data the data for that glyph is, relative to the start of the bitmap data.
• bitmap data
[/spoil]
ETW expanded on the CUF file format seen in M2TW by introducing kerning tables at the end of the file format.
Empire extension (kerning), as discovered by just on the TWC:[ex]
• short: number of glyphs for which kerning data is available
• short: the amount of glyphs to skip, the first nn glyphs don't have kerning data, some of the last might not either if the number of glyphs with kerning and the skipped glyphs are less than the total amount of glyphs
• for each glyph with kerning
o for each glyph with kerning
 byte: width of the glyph in the outer loop when followed by the glyph in the inner loop, the width is calculated form the left of the glyph.
Tooltip text width
A related question (this part of the engines is simply not well researched) is about the height-for-width swap algorithm used by the engine to render tooltips. That is, when a tooltip appears to be too wide it is made more narrow by increasing the number of lines of text in the tooltip and wrapping the long lines. There does not appear to be a clear maximum width limit, so descriptions cannot be made to force a single width for rendering purposes (e.g. to support indentation for a quote in trait description).
The question then is: do you know whether there are certain specific aspect ratio's being enforced, or more generally how the height-for-width swap algorithm works? Is it similar to what e.g. GTK 3 is doing?
Being able to know what width a piece of text will require is very useful for more advanced modding. It allows us to fix text layout in advance which in turn enables supporting proper indentation based around tabs (not in terms of number of spaces but in terms of pixels, creating an effect similar to that of word processors), as well as prevent/circumvent issues with how ETW/NTW/Medieval II/Kingdoms text layout algorithms do not wrap lines properly for characters from locales that they do not support. (E.g. line wrapping appears to be broken for CJK characters in M2TW Kingdoms at least, which means that if we need to use CJK slots we have to do the line wrapping ourselves or pay the penalty when the engine simply does not render characters which are outside the bounding box as a result.)

Requires an answer. Currently with the UI team.

Hi!
In unit stat naval we have these stats
mothball_cost, this is called hull strength in pfm
reactivate_cost, this is called crew size in pfm.
repair_cost_port, this does not apear in pfm although having value in the xml file
repair_cost_sea, this does not apear in pfm although having value in the xml file
...I will be adding more here.
Can you please tell us what are they, how they work if they do? thank you.

Those are not used.

NTW
At which file AGENTS' voices and icons are related with??

Noone could solve this problem so you are the only hope!

Voices are done via code, have asked UI team about icons.

[alhoon]

This is a list of ones I know. Will get more info from campaign team.

Unused tables:
· unit_to_gov_type_permissions
· unit_to_gov_type_conversions_jcts

Units
· fitness not used

Projectiles
· bounce_angle not used
· collision_radius not used

Clarification:
Are these tables that cannot be used so even if we mod them they won't have any effect at all or they're tables that could be used and modded but they're unused so far?

[T.C.]

Projectiles
· bounce_angle not used

I'm pretty sure I edited this to some effect before. Are you sure about this? many thanks for the answers so far

[T.C.]

Also, is it possible that we could enable in-game logging? Your UI Lua's have lots of output functions but these obviously aren't enabled in the release version of the game. Is there any way we could edit this (a true/false switch?), perhaps via preferences.txt?
Also, is there any other general logging for packs or startpos data that we could enable?

One more question - say I decompiled a UI luac file and had questions about the code. Would you be able to pass it on? I've got a lot of decompiled luacs, and a lot of questions. But one important one

[.Mitch.]

Also, is it possible that we could enable in-game logging? Your UI Lua's have lots of output functions but these obviously aren't enabled in the release version of the game. Is there any way we could edit this (a true/false switch?), perhaps via preferences.txt?
Also, is there any other general logging for packs or startpos data that we could enable?

I too would love to know this.

[AmiR TimuR]

Voices are done via code, have asked UI team about icons.

May you be more specific about this? Which codes or at which file are they located? Will be waiting your answer with a solution of icon problem.

Regards

[Steph]

First, thanks a lot for compiling the questions and answering them.

Classes are used in the group formations to work out where units will be placed. They are also used in the AI to help it work out what units are good against others.

So, if I want to use let say the "Elephant" class and rename it to "Cuirassier" (for instance to get more possibilities for regiment names), would it create side effects? I suppose we could edit the new "group formations" since we now have a table for that. For the "They are also used in the AI to help it work out what units are good against others", is that done through tables in the db? If yes which one?

The info from that table is built into the campaign when the startpos is built. Its where that information is got from. Will ask the campaign team for more info.

Thanks, waiting for more info.

For Empire and Napoleon setting the voice field in the units table to an appropriate entry should help with this. Some of it may be hardcoded. Will check further with the audio team.

Unfortunately, it doesn't work. I sent you a PM about that. To sum up, if you just add one unit to the db, nothing more, than the sounds for the already existing units is messed up. I believe it is probably a kind of bugs.
In patch5.pack, there are more stuff about voices, with XML files and csv files, describing for each units which sound events to play, and for each soud events what actual sound files to play.
I tried changing that also to no effect. I have the impression the info from the db or these sounds XML/CSV files need to be compiled in a way.
I'd really appreciate feedback on that (and probably others would also) since it negates a very nice feature you have in the engine. It's very frustrating to add a unit, and when you select your old artillery that wasn't modded in anyway it says "swiss infantry"

In the case listed here, it’s most certainly an effect which will only work when applied factionwide, so you need to put it in the building_factionwide_effects_junctions table rather than the building_effects_junction table.

Thanks, I'll try that.

[CraigTW]

More info added to these questions from the campaign team...

About regiment names
To have specific names, it seems you need to edit the startpos, with name allocator for each faction, where you have 22 lists (one for each class), and can put your regiment names.
However, there is a table for unit_regiment_names... But I have the impression it's useless. can you confirm if it's possible to manage the regiment names via the DB?

For Empire and Napoleon the table only feeds into esf file generation. To modify names used in game you need to edit the startpos.esf or save game files. The same is currently true for Shogun 2 but this may change in a future patch, although there will be caveats.

About effect
I tried to add a new effect to a building
- upkeep_cost_mod_land_all
But it seems it has no effect. Is this effect limited to ministers?
How can we know what effect will work in a town, a settlement or with ministers?

For performance reasons effects are only considered in specific contexts. Unfortunately since these contexts are in a constant state of flux we are not able to provide general details. In this specific case upkeep_cost_mod_land_all should work as a factionwide building effect (probably).

This is a list of ones I know, campaign team will have more info.

Unused tables:
· unit_to_gov_type_permissions
· unit_to_gov_type_conversions_jcts

Units
· fitness not used

Projectiles
· bounce_angle not used
· collision_radius not used

Correction: unit_to_gov_type_permissions is used in Empire.

Is message_event_strings partially hardcoded? We've tried adding entries here, and scripting the message events to appear in game, but with no luck. I can assure you the bug wasn't on the scripting side.

You can’t add new events, but you can modify existing events and repurpose them.

ETW
Is it possible to split a region into two regions without editing the pathfinding.esf, for example splitting Syria into Cyprus and Syria? In the new region one of the existing buildings will be turned into a settlement, such as the Paphos Iron Mine in Cyprus, so there's no need to add additional buildings.
In the grid_data part of the pathfinding.esf is there a generic code that can be added that will enable modders to add new towns, ports, or resources to the map? A generic code that requires the modder to make some changes to it, such as adding the coordinates of the new towns, ports, or resources, is fine as long as we're told what needs to be changed.

No. It is theoretically possible to edit the raw pathfinding data, but this is not something we have ever done or would ever consider doing.

[Slaxx Hatmen]

Craig, what exactly was the procedure the campaign team went through to create the regions for the campaign maps? Like, I recall you or Jack mentioning that 3D Studio Max was used to actually mold and shape the maps themselves. Was this program also involved in the layout and placement of the regions to any significant point?
I know the other modders here probably know this already but these questions have been puzzling me and I'd like someone from the inside to answer.

[CraigTW]

Craig TW
Most answers for our questions is "No" . What exactly you can help us which modders couldn't modify before

We're trying to help by giving honest answers to the questions. If the answer's no, then sorry, but it's better than a lie surely?

[Primergy]

2) Any chance we can have custom map "farms.farm_manager" file explained as you did for db files?

Mostly it's only uninteresting stuff, like the coordinates of fields (which don't show any effect when changed i think those coordinates are used by the map editor of ca to place field textures like on algeria). So all you can do with the field is activate or deactivate which wall/fence/hedge shows up and which farm_template_tile is loaded (+ the coordinates where it is placed).
It might be that the wall entries at the end need a reference to those farm_instances and wall_instances (if you want to add more then are original used) but till now i didn't reached the point.

[LEGIO_Desaix]

Is the column_bonus in KV_MORALE useless or maybe reserved for group column formation?

Column explanation for kv_morale_ext says

<xsd:element name="ume_encouraged_column_formation" minOccurs="1" od:jetType="longinteger" od:sqlSType="int" od:nonNullable="yes" type="xsd:int"/>
<xsd:element name="special_ability_inspire_unit_bonus" minOccurs="1" od:jetType="longinteger" od:sqlSType="int" od:nonNullable="yes" type="xsd:int"/>
<xsd:element name="special_ability_rally_min_bonus" minOccurs="1" od:jetType="longinteger" od:sqlSType="int" od:nonNullable="yes" type="xsd:int"/>
<xsd:element name="special_ability_rally_max_bonus" minOccurs="1" od:jetType="longinteger" od:sqlSType="int" od:nonNullable="yes" type="xsd:int"/>

I believe it's a typo since those labels are "fields" in the normal kv_morale table.
In this case what is the purpose of the kv_morale_ext table and what are the 4 column headings?

Regarding square unmoddable future...it's a pity but at least a clear answer, not exactly the one we'd have liked to read after pre-release announcements, but at least we stop wondering about any engine future patch.

Thanx.

[CraigTW]

Another quick answer to the above. (I'm posting these as I get answers, so bear with me).

About voices
We have an annoying bug, where every time we mod a unit, the voices become messed up. Like select an artillery unit, and it says "Swiss infantry". It is something that we could correct with the DB, or is it really a bug?

It’s only for Shogun2 that voices are triggered from the database
For Empire/Napoleon you will have to change values in the following file:

Working_data/sounds/banks/sound_bank_unit_voice_events.xml

The sound_event_name referred to in this xml must exist in one of the files in:
Working_data/sounds/events/sound_events*.csv

You will need to be able to run a tool to pack the sounds

[Jack Lusted]

Is the column_bonus in KV_MORALE useless or maybe reserved for group column formation?

It's not used at all.

In this case what is the purpose of the kv_morale_ext table and what are the 4 column headings?

It's not used by the game.

[Steph]

It’s only for Shogun2 that voices are triggered from the database
For Empire/Napoleon you will have to change values in the following file:

Working_data/sounds/banks/sound_bank_unit_voice_events.xml

The sound_event_name referred to in this xml must exist in one of the files in:
Working_data/sounds/events/sound_events*.csv

You will need to be able to run a tool to pack the sounds

That is what I imagined. I try to edit both the XML and the csv files as you suggested, but it didn't change anything. I suppose it's because I didn't pack the sound.
Any chance you could give more info about this sound packing tool? Perhaps... release it?

Also, can you get some additional info from the sound team?
I understand that if I want to add a completly new unit, it won't be listed in the sound_bank_unit_voice_event.xml, and so won't be usable.
But is it normal that the voice for the other existing units, which were not changed, can also be messed up?

[Steph]

It's not used by the game.

Does it mean that every kv_Something_Ext are not used in game? Or just for morale?

[CraigTW]

More answers will be forthcoming tomorrow hopefully. All of the remaining unanswered questions are logged and are being looked over by the team

Please appreciate that the guys involved are giving up quite a bit of their daily time to answer these questions, so we might not be able to get you replies straight away. The UI guys are pretty busy right now, for example. But we'll do our best to reply to everything we can.

[uanime5]

No. It is theoretically possible to edit the raw pathfinding data, but this is not something we have ever done or would ever consider doing.

Thanks for the answer. It seems the pathfinding data is more important for mapping than I realised.

[LEGIO_Desaix]

I know this is not strictly connected to DB fles so I will not complain with no answer.

In MP historical battles xml file, apart from KILL OR ROUT ALL ENEMY, is a "capture location victory condition" possible and, when positive, can we have the correct format. I saw a CAPTURE FORT condition is available in SH2 HB, but what we really need is something like:

<VICTORY CONDITION>
<CAPTURE LOCATION X=50; Y=100 radius=50>
</VICTORY CONDITION>

I guess CAPTURE_POINT type MP battle were not implemented in Empire/NTW but perhaps something can be achieved in historical scenario and since I'm going to release my historical battle&map editor this would benefit all 3 games.

Always on this subject, can we achieve a "fog in the morning and then fair weather for the rest of the battle" effect changing the xyz.environment file governing weather in cutom map? Just to know if it's possible, I then will envistigate and spend the usual "countless hours" trying to get something right. Maybe setting the correct probability of cloud presence?

On a side note it would be really interesting (at least for me) to know why a column graphic was designed, a column bonus line added, a column lua script written and then eventually column was not implemented as proper formation. Since nothing was present on Empire files, it was something expressely thought for NTW. I understand release deadline, but since we had 4 patches after that...