Campaign map modding - a modest proposal

[taw]

I have a serious proposition to everyone involved in campaign map modding research:

How about we stop treating map coordinates in esfs (and possibly other places) as
sometimes floats, and sometimes integers (to be multiplied by 2^-20), but simply
as two low-level ways to encode the same decimal number?

They are no real integers - they are fixed point numbers, with decimal point at 20 bit.

Our tools could deal with mess of having two encodings and we'd only deal with decimal numbers.

The major problem is that there's ridiculously many places where this change needs to happen.
But our tools will need to find and fix all these places if we want to mod campaign map anyway.


I went ahead and made esfxml automatically convert <rec type="vertices"> array
from fixed point to decimal on conversion to xml.

Some care needs to be taken to make sure conversion is exact.
First single precision will not be enough you need double.
Second - when converting back you should use round function (round to nearest integer) not to_i (truncate down to
nearest integer), as conversion to and from string introduces some inaccuracy otherwise.

Ruby conversions are:

• (x * (0.5 ** 20)).to_s <- fixed point to decimal text
• (x.to_f * (2.0 ** 20)).round.to_i <- decimal text to fixed point

Anybody has a problem with various records in esfxml getting transformed from these silly integers to decimal they really are? There will be transition mess before we find them all of course, but the alternative is just too much unnecessary work of manual conversion forever.

[uanime5]

I can name some of the locations of the large map coordinates if this helps. All locations are for the startpos after it has been unpacked using your unpacker.

'cai_interface' folder: in each <rec type='CAI_SITUATED'> and <rec type='CAI_REGION_HLCI'> tags, the <i> tags are coordinates.

'cai_region_slots' folder: in each <rec type='CAI_SITUATED'> tag, the <i> tags are coordinates.

'cai_mobiles' folder: in each <rec type='CAI_SITUATED'> tag, the <i> tags are coordinates.

'cai_settlements' folder: in each <rec type='CAI_SITUATED'> tag, the <i> tags are coordinates.

'cai_trading_posts' folder: in each <rec type='CAI_TRADING_POST'> tag the <i> tags are coordinates.


'region' folder: in each <rec type='BUILDING_MANAGER'> tag the <i> tags with more than one digit are coordinates.

'region_slot' folder: in each <rec type='BUILDING_MANAGER'> tag the <i> tags with more than one digit are coordinates.

[taw]

I can name some of the locations of the large map coordinates if this helps. All locations are for the startpos after it has been unpacked using your unpacker.

'cai_interface' folder: in each <rec type='CAI_SITUATED'> and <rec type='CAI_REGION_HLCI'> tags, the <i> tags are coordinates.

'cai_region_slots' folder: in each <rec type='CAI_SITUATED'> tag, the <i> tags are coordinates.

'cai_mobiles' folder: in each <rec type='CAI_SITUATED'> tag, the <i> tags are coordinates.

'cai_settlements' folder: in each <rec type='CAI_SITUATED'> tag, the <i> tags are coordinates.

'cai_trading_posts' folder: in each <rec type='CAI_TRADING_POST'> tag the <i> tags are coordinates.


'region' folder: in each <rec type='BUILDING_MANAGER'> tag the <i> tags with more than one digit are coordinates.

'region_slot' folder: in each <rec type='BUILDING_MANAGER'> tag the <i> tags with more than one digit are coordinates.

BUILDING_MANAGER doesn't contain any locations iirc,
I assume you meant <i>s just a few entries after BUILDING_MANAGER, right?

Any idea what are extra arguments for these?


I guess cai_trading_post's a= is some sort of theatre id or sea id or region id (values 27/36/42/28),
if it was resuorce it would have only three ids.

Is b by perhaps cai army id?

<cai_trading_post a="27" x="863.715087890625" y="-58.8755569458008" b="0"/>
<cai_trading_post a="27" x="876.923095703125" y="-70.0838851928711" b="5596"/>
<cai_trading_post a="27" x="836.62841796875" y="-66.2535858154297" b="0"/>
<cai_trading_post a="36" x="287.910919189453" y="-113.583038330078" b="0"/>
<cai_trading_post a="36" x="291.287353515625" y="-97.1175003051758" b="0"/>
<cai_trading_post a="36" x="345.533172607422" y="-93.607551574707" b="0"/>
<cai_trading_post a="36" x="322.569641113281" y="-111.794441223145" b="0"/>
<cai_trading_post a="36" x="290.025817871094" y="-81.6731872558594" b="0"/>
<cai_trading_post a="42" x="-295.342224121094" y="-160.54035949707" b="0"/>
<cai_trading_post a="42" x="-321.491638183594" y="-171.015579223633" b="6455"/>
<cai_trading_post a="42" x="-343.119689941406" y="-192.801940917969" b="0"/>
<cai_trading_post a="42" x="-338.189239501953" y="-178.577713012695" b="0"/>
<cai_trading_post a="42" x="-311.444061279297" y="-164.780654907227" b="0"/>
<cai_trading_post a="48" x="27.3437042236328" y="44.4812622070312" b="0"/>
<cai_trading_post a="48" x="40.0620346069336" y="29.7541084289551" b="6489"/>


Zero idea here:

<cai_situated x="912.075012207031" y="-43.2000007629395" a="297" b="27" c="28"/>
<cai_situated x="805.800048828125" y="-51.5" a="297" b="27" c="28"/>
<cai_situated x="-750.0" y="100.0" a="128" b="33" c="21"/>
<cai_situated x="-600.0" y="10.0" a="128" b="33" c="21"/>
<cai_situated x="-350.0" y="370.0" a="998" b="33" c="33"/>
<cai_situated x="-350.0" y="220.0" a="988" b="33" c="33"/>
<cai_situated x="-350.0" y="80.0" a="988" b="33" c="33"/>
<cai_situated x="-170.0" y="400.0" a="203" b="34" c="30"/>
<cai_situated x="-170.0" y="280.0" a="172" b="34" c="30"/>
<cai_situated x="-170.0" y="200.0" a="172" b="34" c="30"/>
<cai_situated x="-147.5" y="150.0" a="172" b="34" c="30"/>
<cai_situated x="445.515014648438" y="61.495002746582" a="150" b="35" c="31"/>
<cai_situated x="669.64501953125" y="68.1150054931641" a="922" b="35" c="31"/>
<cai_situated x="292.5" y="-132.5" a="248" b="36" c="26"/>
<cai_situated x="312.5" y="-62.5" a="248" b="36" c="26"/>
<cai_situated x="-326.375" y="-229.0" a="983" b="42" c="32"/>
<cai_situated x="-272.364990234375" y="-146.600006103516" a="983" b="42" c="32"/>
<cai_situated x="-252.649993896484" y="-153.170013427734" a="983" b="42" c="32"/>
<cai_situated x="-252.649993896484" y="-178.100006103516" a="983" b="42" c="32"/>
<cai_situated x="-344.875" y="-219.75" a="983" b="42" c="32"/>
<cai_situated x="-2.5" y="27.5" a="242" b="48" c="25"/>
<cai_situated x="-2.5" y="-17.6000003814697" a="242" b="48" c="25"/>
<cai_situated x="62.5" y="-32.5" a="242" b="48" c="25"/>

Zero idea here again:

<cai_region_hlci a="54" b="109" c="0" x="39.9997711181641" y="-260.477630615234"/>
<cai_region_hlci a="56" b="31" c="0 1" x="562.499633789062" y="47.9563140869141"/>
<cai_region_hlci a="56" b="110" c="2" x="466.8828125" y="-289.999847412109"/>
<cai_region_hlci a="56" b="111" c="3" x="705.263549804688" y="33.6946716308594"/>
<cai_region_hlci a="56" b="112" c="4" x="713.7294921875" y="34.1324081420898"/>
<cai_region_hlci a="61" b="113" c="0" x="-1212.8662109375" y="-479.999786376953"/>
<cai_region_hlci a="61" b="114" c="1" x="968.406311035156" y="-414.999816894531"/>
<cai_region_hlci a="64" b="35" c="0" x="-120.70320892334" y="232.786819458008"/>
<cai_region_hlci a="64" b="36" c="1" x="-117.967514038086" y="201.053237915039"/>
<cai_region_hlci a="64" b="37" c="2" x="-110.668258666992" y="199.258697509766"/>
<cai_region_hlci a="64" b="38" c="3" x="-99.6234359741211" y="201.858489990234"/>
<cai_region_hlci a="64" b="39" c="4" x="-96.6709060668945" y="206.555374145508"/>
<cai_region_hlci a="64" b="0" c="5" x="332.788696289062" y="-122.129745483398"/>
<cai_region_hlci a="64" b="40" c="6" x="350.576568603516" y="491.350860595703"/>
<cai_region_hlci a="64" b="1" c="7" x="395.185333251953" y="522.413452148438"/>
<cai_region_hlci a="64" b="2" c="8" x="680.998596191406" y="36.4596710205078"/>
<cai_region_hlci a="64" b="3" c="9" x="806.938842773438" y="-56.7416381835938"/>
<cai_region_hlci a="64" b="4" c="10" x="814.473876953125" y="-13.5541496276855"/>
<cai_region_hlci a="1183" b="34" c="3 4 6 7 8 9 11 12 13" x="-619.06689453125" y="312.598419189453"/>
<cai_region_hlci a="1183" b="104" c="5" x="-629.234558105469" y="354.475769042969"/>

[uanime5]

BUILDING_MANAGER doesn't contain any locations iirc,
I assume you meant <i>s just a few entries after BUILDING_MANAGER, right?

Any idea what are extra arguments for these?

Sorry I originally wrote after the <s> tags, then tried to find the nearest rec to make it easier to understand. These <i> tags are definitely in the <rec type='REGION_SLOT'>

I guess cai_trading_post's a= is some sort of theatre id or sea id or region id (values 27/36/42/28),
if it was resuorce it would have only three ids.

Is b by perhaps cai army id?

<cai_trading_post a="27" x="863.715087890625" y="-58.8755569458008" b="0"/>
<cai_trading_post a="27" x="876.923095703125" y="-70.0838851928711" b="5596"/>
<cai_trading_post a="27" x="836.62841796875" y="-66.2535858154297" b="0"/>
<cai_trading_post a="36" x="287.910919189453" y="-113.583038330078" b="0"/>
<cai_trading_post a="36" x="291.287353515625" y="-97.1175003051758" b="0"/>
<cai_trading_post a="36" x="345.533172607422" y="-93.607551574707" b="0"/>
<cai_trading_post a="36" x="322.569641113281" y="-111.794441223145" b="0"/>
<cai_trading_post a="36" x="290.025817871094" y="-81.6731872558594" b="0"/>
<cai_trading_post a="42" x="-295.342224121094" y="-160.54035949707" b="0"/>
<cai_trading_post a="42" x="-321.491638183594" y="-171.015579223633" b="6455"/>
<cai_trading_post a="42" x="-343.119689941406" y="-192.801940917969" b="0"/>
<cai_trading_post a="42" x="-338.189239501953" y="-178.577713012695" b="0"/>
<cai_trading_post a="42" x="-311.444061279297" y="-164.780654907227" b="0"/>
<cai_trading_post a="48" x="27.3437042236328" y="44.4812622070312" b="0"/>
<cai_trading_post a="48" x="40.0620346069336" y="29.7541084289551" b="6489"/>

The Trading Posts are the anchor symbols in the trade theatres. This is why all their coordinates are in the trade theatres.

A is the Theatre ID. These are as follows:

east_indies 27
america 33
europe 34
india 35
madagascar 36
brazil 42
ivory_coast 48

b is a Resource ID. They're all ships as the trade post are in the sea.

5596 is a Netherlands ship.
6455 and 6489 are a Portuguese ships.

Zero idea here:

<cai_situated x="912.075012207031" y="-43.2000007629395" a="297" b="27" c="28"/>
<cai_situated x="805.800048828125" y="-51.5" a="297" b="27" c="28"/>
<cai_situated x="-750.0" y="100.0" a="128" b="33" c="21"/>
<cai_situated x="-600.0" y="10.0" a="128" b="33" c="21"/>
<cai_situated x="-350.0" y="370.0" a="998" b="33" c="33"/>
<cai_situated x="-350.0" y="220.0" a="988" b="33" c="33"/>
<cai_situated x="-350.0" y="80.0" a="988" b="33" c="33"/>
<cai_situated x="-170.0" y="400.0" a="203" b="34" c="30"/>
<cai_situated x="-170.0" y="280.0" a="172" b="34" c="30"/>
<cai_situated x="-170.0" y="200.0" a="172" b="34" c="30"/>
<cai_situated x="-147.5" y="150.0" a="172" b="34" c="30"/>
<cai_situated x="445.515014648438" y="61.495002746582" a="150" b="35" c="31"/>
<cai_situated x="669.64501953125" y="68.1150054931641" a="922" b="35" c="31"/>
<cai_situated x="292.5" y="-132.5" a="248" b="36" c="26"/>
<cai_situated x="312.5" y="-62.5" a="248" b="36" c="26"/>
<cai_situated x="-326.375" y="-229.0" a="983" b="42" c="32"/>
<cai_situated x="-272.364990234375" y="-146.600006103516" a="983" b="42" c="32"/>
<cai_situated x="-252.649993896484" y="-153.170013427734" a="983" b="42" c="32"/>
<cai_situated x="-252.649993896484" y="-178.100006103516" a="983" b="42" c="32"/>
<cai_situated x="-344.875" y="-219.75" a="983" b="42" c="32"/>
<cai_situated x="-2.5" y="27.5" a="242" b="48" c="25"/>
<cai_situated x="-2.5" y="-17.6000003814697" a="242" b="48" c="25"/>
<cai_situated x="62.5" y="-32.5" a="242" b="48" c="25"/>

Which CAI situated is this. I listed 4 different folders this was in, each of which contains different information.

Zero idea here again:

<cai_region_hlci a="54" b="109" c="0" x="39.9997711181641" y="-260.477630615234"/>
<cai_region_hlci a="56" b="31" c="0 1" x="562.499633789062" y="47.9563140869141"/>
<cai_region_hlci a="56" b="110" c="2" x="466.8828125" y="-289.999847412109"/>
<cai_region_hlci a="56" b="111" c="3" x="705.263549804688" y="33.6946716308594"/>
<cai_region_hlci a="56" b="112" c="4" x="713.7294921875" y="34.1324081420898"/>
<cai_region_hlci a="61" b="113" c="0" x="-1212.8662109375" y="-479.999786376953"/>
<cai_region_hlci a="61" b="114" c="1" x="968.406311035156" y="-414.999816894531"/>
<cai_region_hlci a="64" b="35" c="0" x="-120.70320892334" y="232.786819458008"/>
<cai_region_hlci a="64" b="36" c="1" x="-117.967514038086" y="201.053237915039"/>
<cai_region_hlci a="64" b="37" c="2" x="-110.668258666992" y="199.258697509766"/>
<cai_region_hlci a="64" b="38" c="3" x="-99.6234359741211" y="201.858489990234"/>
<cai_region_hlci a="64" b="39" c="4" x="-96.6709060668945" y="206.555374145508"/>
<cai_region_hlci a="64" b="0" c="5" x="332.788696289062" y="-122.129745483398"/>
<cai_region_hlci a="64" b="40" c="6" x="350.576568603516" y="491.350860595703"/>
<cai_region_hlci a="64" b="1" c="7" x="395.185333251953" y="522.413452148438"/>
<cai_region_hlci a="64" b="2" c="8" x="680.998596191406" y="36.4596710205078"/>
<cai_region_hlci a="64" b="3" c="9" x="806.938842773438" y="-56.7416381835938"/>
<cai_region_hlci a="64" b="4" c="10" x="814.473876953125" y="-13.5541496276855"/>
<cai_region_hlci a="1183" b="34" c="3 4 6 7 8 9 11 12 13" x="-619.06689453125" y="312.598419189453"/>
<cai_region_hlci a="1183" b="104" c="5" x="-629.234558105469" y="354.475769042969"/>

a = Region ID
b = Area ID
c = Number of areas in Region, for example region 64 (unexplorable) has 11 areas (0-10)

[taw]

Which CAI situated is this. I listed 4 different folders this was in, each of which contains different information.

They have same schema, so I only made one converter.

Here are some examples by type.

b is actually u4_ary but it seems to always contain exactly one entry.

Many per file:

cai_interface/cai_world-0001.xml
4699| <cai_situated x="503.75" y="237.070007324219" a="540" b="35" c="13"/>
61135| <cai_situated x="-600.0" y="10.0" a="128" b="33" c="21"/>
61168| <cai_situated x="-350.0" y="370.0" a="998" b="33" c="33"/>
61201| <cai_situated x="-350.0" y="220.0" a="988" b="33" c="33"/>
61234| <cai_situated x="-350.0" y="80.0" a="988" b="33" c="33"/>
61267| <cai_situated x="-170.0" y="400.0" a="203" b="34" c="30"/>
61300| <cai_situated x="-170.0" y="280.0" a="172" b="34" c="30"/>

One per file:

cai_settlements/settlement-0085.xml
4| <cai_situated x="608.141967773438" y="144.141967773438" a="938" b="35" c="13"/>
cai_settlements/settlement-0133.xml
4| <cai_situated x="-381.098083496094" y="336.636383056641" a="1249" b="33" c="20"/>
cai_settlements/settlement-0077.xml
4| <cai_situated x="-153.965728759766" y="457.544677734375" a="905" b="34" c="12"/>
cai_settlements/settlement-0115.xml
4| <cai_situated x="-577.5205078125" y="226.714202880859" a="1132" b="33" c="20"/>

One per file:

cai_mobiles/mobile-0092.xml
4| <cai_situated x="263.600006103516" y="390.5" a="833" b="34" c="13"/>
cai_mobiles/mobile-0187.xml
4| <cai_situated x="40.0620346069336" y="29.7541084289551" a="242" b="48" c="25"/>
cai_mobiles/mobile-0228.xml
4| <cai_situated x="118.220001220703" y="225.949996948242" a="553" b="34" c="30"/>
cai_mobiles/mobile-0013.xml
4| <cai_situated x="-18.8999996185303" y="363.420013427734" a="765" b="34" c="11"/>

One or two per file:

cai_region_slots/region_slot-0354.xml
4| <cai_situated x="227.662017822266" y="388.130004882812" a="833" b="34" c="13"/>
cai_region_slots/region_slot-0015.xml
4| <cai_situated x="-68.6836776733398" y="217.595886230469" a="478" b="34" c="13"/>
48| <cai_situated x="-69.8807601928711" y="216.692001342773" a="172" b="34" c="30"/>
cai_region_slots/region_slot-0600.xml
4| <cai_situated x="-556.946044921875" y="156.575744628906" a="1094" b="33" c="18"/>

a = Region ID
b = Area ID
c = Number of areas in Region, for example region 64 (unexplorable) has 11 areas (0-10)

c is u4_ary, it cannot be a number.

See this for example:
<cai_region_hlci a="1183" b="34" c="3 4 6 7 8 9 11 12 13" x="-619.06689453125" y="312.598419189453"/>

Where can I find these region/area ids?
I could make a script to list id by object, just like one ./scripts/list_faction_ids

[uanime5]

They have same schema, so I only made one converter.

Here are some examples by type.

b is actually u4_ary but it seems to always contain exactly one entry.

Spoiler Alert, click show to read: 

Many per file:

cai_interface/cai_world-0001.xml
4699| <cai_situated x="503.75" y="237.070007324219" a="540" b="35" c="13"/>
61135| <cai_situated x="-600.0" y="10.0" a="128" b="33" c="21"/>
61168| <cai_situated x="-350.0" y="370.0" a="998" b="33" c="33"/>
61201| <cai_situated x="-350.0" y="220.0" a="988" b="33" c="33"/>
61234| <cai_situated x="-350.0" y="80.0" a="988" b="33" c="33"/>
61267| <cai_situated x="-170.0" y="400.0" a="203" b="34" c="30"/>
61300| <cai_situated x="-170.0" y="280.0" a="172" b="34" c="30"/>

One per file:

cai_settlements/settlement-0085.xml
4| <cai_situated x="608.141967773438" y="144.141967773438" a="938" b="35" c="13"/>
cai_settlements/settlement-0133.xml
4| <cai_situated x="-381.098083496094" y="336.636383056641" a="1249" b="33" c="20"/>
cai_settlements/settlement-0077.xml
4| <cai_situated x="-153.965728759766" y="457.544677734375" a="905" b="34" c="12"/>
cai_settlements/settlement-0115.xml
4| <cai_situated x="-577.5205078125" y="226.714202880859" a="1132" b="33" c="20"/>

One per file:

cai_mobiles/mobile-0092.xml
4| <cai_situated x="263.600006103516" y="390.5" a="833" b="34" c="13"/>
cai_mobiles/mobile-0187.xml
4| <cai_situated x="40.0620346069336" y="29.7541084289551" a="242" b="48" c="25"/>
cai_mobiles/mobile-0228.xml
4| <cai_situated x="118.220001220703" y="225.949996948242" a="553" b="34" c="30"/>
cai_mobiles/mobile-0013.xml
4| <cai_situated x="-18.8999996185303" y="363.420013427734" a="765" b="34" c="11"/>

One or two per file:

cai_region_slots/region_slot-0354.xml
4| <cai_situated x="227.662017822266" y="388.130004882812" a="833" b="34" c="13"/>
cai_region_slots/region_slot-0015.xml
4| <cai_situated x="-68.6836776733398" y="217.595886230469" a="478" b="34" c="13"/>
48| <cai_situated x="-69.8807601928711" y="216.692001342773" a="172" b="34" c="30"/>
cai_region_slots/region_slot-0600.xml
4| <cai_situated x="-556.946044921875" y="156.575744628906" a="1094" b="33" c="18"/>

Thanks for the examples they made it easier to figure out.

a = Region ID
b = Theatre ID
c = Area ID

c is u4_ary, it cannot be a number.

See this for example:
<cai_region_hlci a="1183" b="34" c="3 4 6 7 8 9 11 12 13" x="-619.06689453125" y="312.598419189453"/>

Where can I find these region/area ids?
I could make a script to list id by object, just like one ./scripts/list_faction_ids

I'm not sure what c is meant to be then. All I know from my spreadsheets is that each region doesn't use the same 'c' number twice and the 'c' numbers correspond to the number of Area IDs a region has.

The Area IDs are mentioned in the HLCIS entries, which is currently part of the cai_world-0001.xml.

They're also part of the regions.esf. In region_data each region has several areas and each of these areas has an Area ID below their coordinates. For example the region 'great_lakes' (1183) has 14 Area IDS, which are 33, 34, 101, 102, 103, 104. Of the 14 areas 9 have the Area ID 34.

Region IDs can be found all the 'CAI_region' folder. Each xml has their Region ID in the <u> tag above the <flt>100.0</flt>.

Both the Region and Area IDs are listed in my 'ETW regions CAI' spreadsheet, as I needed this information to decode the CAI_World_Interface. Would you like me to upload the latest version?

[taw]

cai_situated / cai_region_hlcl / cai_trading_post translations are ready on etwng.

I'll do this thing just after BUILDING_MANAGER soon, partial conversions
are harder than full record conversions the way esf2xml is coded.

[taw]

Two more tags:

<cai_border_patrol_point x="667.338684082031" y="65.3495788574219" a="922"/>

<famous_battle_info x="-56.5" y="370.0" name="Battle of: Dublin" a="1502" b="1502" c="1250" d="no"/>

Also here's now universal tag <v2x> which expands to two <i> <i> entries with conversion from small to big grid.
It's used inside <rec type="LOCOMOTABLE">, and it will be used anywhere else where
it's necessary to do partial conversion.

<v2x x="122.730003356934" y="340.149993896484"/>

Usually, when there are two <i> lines with big numbers next to each other, they are big grid coordinates,
and should be ideally replaced by <v2x> line.

Unfortunately there are enough exceptions that it cannot be done automatically,
and it will need special code writing for every such case.

Also unfortunately, I will need to change esf2xml's architecture somewhat
to be able to handle partial translation of something as complicated as REGION_SLOT :-/

[taw]

I just added <i><i> -> <v2x> conversion for about 12 more places in startpos.esf

I'm sure I missed a few, but if you see a <i><i> pair which looks like big grid coordinates anywhere,
just send me its location, and I can add it quite easily now, even if it's in the middle of something nasty
and complicated.

esfxml is now much better at converting things in the middle of something nasty and complicated now,
so if you have anything else you'd like to see converted like that, tell me as well.

And I can finally add those <!-- xml comments --> easily, so remind me where I was supposed to put them, ok?

[taw]

Every place uanime5 mentioned, and every place I found so far gets now converted -
but I'm sure I've missed a few. If you ever find any unconverted big grid coordinates, report them here (or PM me).