Research of Unknown Areas in Startpos.esf

[husserlTW]

Though we know enough about startpos, there are large areas unexplored which prevent us to improve modding abilities. In CAI area for example there are records partly meaningful, like the sections for factions, regions, slots, etc. But there are also big parts that we cannot handle at all since we know almost nothing like CAI_INTERFACE_MANAGERS , CAI_BDI_POOL , or CAMPAIGN_PATHFINDER.

As you know an element of the game, like a faction, has a record in [...]\CAMAPIGN_MODEL\WORLD\FACTION_ARRAYS and another in [...]\CAMAPIGN_MODEL\CAI_INTERFACE\CAI_WORLD\ CAI_WORLD _FACTIONS. Some stuff from WORLD like Id's are changed in saves and that means in every turn of the game. Those of CAI_WORLD do not. In CAI_WORLD also are included the relations and connections among the game elements and the way AI realize them. The most complex parts are no doubt CAI_INTERFACE_MANAGERS and CAMPAIGN_PATHFINDER.

Some guys in forum have asked to help in this search but that is difficult due the nature if the these files and the available tools. I've started this thread trying to coordinate searching efforts using the same tools available for the same files. This is important if we want to decode records like CAI_INTERFACE_MANAGERS or CAMPAIGN_PATHFINDER. They are full endless lists with a variety of unknown numbers. Existing EsfEditors could not help much to search these areas. EsfEditor 1.5 beta of Erasmus and mainly Taw's esf2xml increase our searching abilities. I've already started exploring CAI_INTERFACE_MANAGERS and I feel as being in a deep dark labyrinth.

This task can have some chance of success if we gather the efforts of the many. So here it is for anybody who likes to take part. Attached you'll find some useful stuff to help you. Using it we will also make sure that we are working on the same startpos file.

The target:

The first thing is to find some normality in those areas, similarities, repeating patterns or/and expressions, etc. For example you will see that there are so many CAI_INTERFACE_MANAGERS as the faction are. In CAI_INTERFACE_MANAGERS\CAI_FACTION_MANAGERS first line is cai faction Id so it is easy to find which one belongs to what faction. In CAI_INTERFACE_MANAGERS\CAI_BDI_POOL are contained the Beliefs, Desires and Intentions of the Ai about the specific faction. If you open the CAI_BDI_POOL\ CAI_BDI_POOL_BELIEFS you'll see that they have a number like Belief ID (first one is 326, then 336 and so on) which are common to all Beliefs in all factions. That similarity probably means that there are certain type of Beliefs (326, 336, etc.)which we should try to decode.

The arsenal

On the attached folder there are:

1. Alpaca's searching tool
2. Erasmus EsfEditor 1.5 beta
3. A starrtpos.esf file
4. A startpos.esf converted in xml
5. Notepad++
6. Python

Alpaca's tool does not need any installation. But you have to install Python before you can uses it. After installing python you are ready to find where numbers are located in an esf file. To do that place your file in Alpaca's folder. You can search any esf file but you have to rename it in startpos.esf. Open a cmd window by entering cmd in START\RUN and point to Alpaca's folder. Alternatively hold shift and right click in an empty (white) area in Alpaca's folder. Let's say you want to search where the number 58283483888 is in startpos. Enter the command:

python main.py 58283483888 myfile.txt

The utility will create a txt call myfile and there you will see all the paths in startpos that this number is met. You can try any number and you can give any txt file name.
Erasmus EsfEditor 1.5 beta is used basically just to see info. It has labels like 1.4.5 and you can navigate fast in startpos when you know the paths from Alpaca's searching utility. Use it to find some info only and not to edit startpos as it has many bugs and is not reliable at all in saving.
Notepad++ is a very good txt editor which handles and xml files, so it is good solution tofor handling Taw's xmls.

How work it out?

Well there is not a certain way of searching, it is a search after all! Personally I follow deferent approaches. I open a xml with CAI_INTERFACE_MANAGERS and I try to track down where the numbers link using Alpaca's utility to get the paths and then EsfEditor. Another way is to compare records and see if there are nay similarity, repeating pattern, etc. It all depends on the mood and the inspirations you have at that moment. It is not an easy task and needs some time to get used startpos lists. But I strongly believe that we can get some light in these dark areas of startpos improving the current modding abilities. Please concentrate on areas that we do not know yet. If you have doubts post a question.

[husserlTW]

http://en.wikipedia.org/wiki/Belief-Desire-Intention_software_model

The Belief-Desire-Intention (BDI) software model (usually referred to simply, but ambiguously, as BDI) is a software model developed for programming intelligent agents. Superficially characterized by the implementation of an agent's beliefs, desires and intentions, it actually uses these concepts to solve a particular problem in agent programming. In essence, it provides a mechanism for separating the activity of selecting a plan (from a plan library) from the execution of currently active plans. Consequently, BDI agents are able to balance the time spent on deliberating about plans (choosing what to do) and executing those plans (doing it). A third activity, creating the plans in the first place (planning), is not within the scope of the model, and is left to the system designer and programmer.

Overview

In order to achieve this separation, the BDI software model implements the principal aspects of Michael Bratman's theory of human practical reasoning (also referred to as Belief-Desire-Intention, or BDI). That is to say, it implements the notions of belief, desire and (in particular) intention, in a manner inspired by Bratman. For Bratman, intention and desire are both pro-attitudes (mental attitudes concerned with action), but intention is distinguished as a conduct-controlling pro-attitude. He identifies commitment as the distinguishing factor between desire and intention, noting that it leads to (1) temporal persistence in plans and (2) further plans being made on the basis of those to which it is already committed. The BDI software model partially addresses these issues. Temporal persistence, in the sense of explicit reference to time, is not explored. The hierarchical nature of plans is more easily implemented: a plan consists of a number of steps, some of which may invoke other plans. The hierarchical definition of plans itself implies a kind of temporal persistence, since the overarching plan remains in effect while subsidiary plans are being executed.
An important aspect of the BDI software model (in terms of its research relevance) is the existence of logical models through which it is possible to define and reason about BDI agents. Research in this area has led, for example, to the axiomatization of some BDI implementations, as well as to formal logical descriptions such as Anand Rao and Michael Georgeff's BDICTL. The latter combines a multiple-modal logic (with modalities representing beliefs, desires and intentions) with the temporal logic CTL*. More recently, Michael Wooldridge has extended BDICTL to define LORA (the Logic Of Rational Agents), by incorporating an action logic. In principle, LORA allows reasoning not only about individual agents, but also about communication and other interaction in a multi-agent system.
The BDI software model is closely associated with intelligent agents, but does not, of itself, ensure all the characteristics associated with such agents. For example, it allows agents to have private beliefs, but does not force them to be private. It also has nothing to say about agent communication. Ultimately, the BDI software model is an attempt to solve a problem that has more to do with plans and planning (the choice and execution thereof) than it has to do with the programming of intelligent agents.
BDI Agents

A BDI agent is a particular type of bounded rational software agent, imbued with particular mental attitudes, viz: Beliefs, Desires and Intentions (BDI).
Architecture

This section defines the idealized architectural components of a BDI system.

• Beliefs: Beliefs represent the informational state of the agent, in other words its beliefs about the world (including itself and other agents). Beliefs can also include inference rules, allowing forward chaining to lead to new beliefs. Using the term belief rather than knowledge recognizes that what an agent believes may not necessarily be true (and in fact may change in the future).
  
  • Beliefset: Beliefs are stored in database (sometimes called a belief base or a belief set), although that is an implementation decision.
  
  
• Desires: Desires represent the motivational state of the agent. They represent objectives or situations that the agent would like to accomplish or bring about. Examples of desires might be: find the best price, go to the party or become rich.
  
  • Goals: A goal is a desire that has been adopted for active pursuit by the agent. Usage of the term goals adds the further restriction that the set of active desires must be consistent. For example, one should not have concurrent goals to go to a party and to stay at home - even though they could both be desirable.
  
  
• Intentions: Intentions represent the deliberative state of the agent - what the agent has chosen to do. Intentions are desires to which the agent has to some extent committed. In implemented systems, this means the agent has begun executing a plan.
  
  • Plans: Plans are sequences of actions (recipes or knowledge areas) that an agent can perform to achieve one or more of its intentions. Plans may include other plans: my plan to go for a drive may include a plan to find my car keys. This reflects that in Bratman's model, plans are initially only partially conceived, with details being filled in as they progress.
  
  
• Events: These are triggers for reactive activity by the agent. An event may update beliefs, trigger plans or modify goals. Events may be generated externally and received by sensors or integrated systems. Additionally, events may be generated internally to trigger decoupled updates or plans of activity.

BDI Interpreter

This section defines an idealized BDI interpreter that provides the basis of the PRS linage of BDI systems[1]:

1. initialize-state
2. repeat
   
   1. options: option-generator(event-queue)
   2. selected-options: deliberate(options)
   3. update-intentions(selected-options)
   4. execute()
   5. get-new-external-events()
   6. drop-unsuccessful-attitudes()
   7. drop-impossible-attitudes()
   
   
3. end repeat

This basic algorithm has been extended in many ways, for instance to support planning ahead[2], automated teamwork[3], and maintenance goals [4].

[husserlTW]

CAI_INTERFACE_MANAGERS\CAI_FACTION_MANAGER\CAI_FACTION_LEARNT_PARAMETERS




1265 = cai faction id (britain)

11 = common in all managers

True = non rebeles, False = rebel faction



Britain has 17 parameters thus itself(?) + 16 other factions.

Pattern 0 0,8 is always the same.

First parameter has a 0 in first line I suppose it has to do with Britain itself.

Second has 1388 -> Huron cai Id and the rest are for factions: France, Pirates, Spain, Cherokee, Ottomans, Morocco, Barbary States, Persia, Knights of St John, Venice, Genoa, Denmark, Maratha, Mughal, Saxony.

Here are all factions that have one or more plus the one with 0 value:



All the rest have only the one with 0 value:





Why are these specific factions?

[husserlTW]

(reserved..)

[husserlTW]

(reserved...)

[Peteriscoolz]

Your diving into the deep end husslerTW,nontheless


.

[husserlTW]

Here it is a first QUIZ in post #3. Why AI choose these specific factions and for what?

All answers are accepted!

[Cursisanafou]

There is only 1 problem with making assumptions into this data... technically there have been turns or possibly just 1 turn made to the game when you actually start the game.

Edit: So it does persist through saves... but the numbers can change.

Observation: Perhaps the 0 represents the general factions... they view every faction the same.

Although truth is the numbers hardly ever change.

[uanime5]

If you ignore Britain and the Netherlands then most of the factions listed are neighbouring factions. For example Austria and Saxony are next to Prussia. 0.8 may be the threat level each of these regions poses to Prussia, so if Prussia was to crush Austria or Austria crushed Prussia the threat level would change. This would allow the AI to better calculate how dangerous each faction is.

Britain and the Netherlands may have more enemy nations because they're naval powers, so they have to be wary of any faction that has access to the sea.

Obviously a faction that hasn't emerged won't have any information regarding how threatening other factions are because it doesn't share a border with anyone. It's also possible that CA didn't add this information to the native factions or the American protectorates because they're not meant to be playable.

It's odd that knights_stjohn, pirates, barbary_states, venice, genoa, and portugal have these entries. Maybe they were meant to be playable factions but were cut at a later date. I can imagine CA wanting a playable pirate faction to improve sales.

Also I found an article about BDI on wikipedia, so it's not something CA just made up.
http://en.wikipedia.org/wiki/Belief-...software_model

[husserlTW]

Very good thinking and info, I'll add it to second post.

[uanime5]

I'll try to simply explain what BDI represents for those who don't understand the wikipedia article.

Belief

Belief: what a faction believes, which can be true or false. For example if Prussia can't see any armies in Austria it may belief that Austria doesn't have any armies and can easily be invaded, though Austria may be hiding it's armies so it can ambush Prussia. In this case Prussia's belief that Austria was defenceless was false.

Belief allows a factions to analyse how valuable a region is, how to respond to other factions, how much of a threat neighbouring regions are, how strong the garrison is, how many reserves are available, how much defence it needs, and what can be built there.

Desire

Desire: what the faction wants. This will depend on the type of campaign you choose as each faction has their own set of victory conditions for each campaign. For example in the long campaign the factions has to conquer 35 regions, including some specific regions. It's possible that desire is changed throughout ETW to reflect the changing priorities of the factions. For example if a factions has 35 regions but is missing 2 of the regions it needs for victory it should desire getting those regions more than getting any other region.

It also controls region and region_slot defence so if a faction loses a region they know they need greater defences to prevent more regions being lost.

Intention

After the faction knows what it wants (desire) and calculates how to get it based on the information it has (belief), it carries out its plan to get it (intention). For example Prussia desires a region Austria controls, believes that the region is undefended, and sends in a small army with the intention of conquering this region.

Intentions affect when a factions chooses to declare war, which upgrades it prioritises, when to promote a general, what to research first, and where to garrison.


The BDI is not static throughout ETW but constantly changes as the AI adapts to new situations. I suspect that the vanilla BDI is larger than the hybrid BDI simply because choosing which faction and type of campaign you're going to play has a large effect on how the AI factions need to respond. So a hybrid startpos may require the vanilla BDI to be added back into it.


husserlTW have you found a region, settlement, building, or building slot that is contained in the BDI of several factions? For example does only Britain's BDI have the region ID 765 (England) or is it also in France and Prussia's BDI? If 765 appears in multiple faction's BDIs then this would explain the high number of BDI entries; every major faction needs to know where every region is and what it contains.

The lack of BDI entries for minor factions is probably to ensure they try to survive but don't try to conquer anywhere. husserlTW do you know whether factions without BDI entries can plan invasion or upgrade building as I'm trying to figure out how the lack of a BDI entry affects what a faction does with a building.

[husserlTW]

Every faction has BDI under it's own CAI_INTERFACE_MANAGER even emergent do. In Beliefs I think there is also the current situation: which regions, cities, towns, etc. the faction has in its disposal. It would be interesting to compare with a save to see how it changes.

I have not found anything like this you are asking yet but I cannot say that I've searched all BDIs. You could search in Taw's xmls with search function of notepad. Do not enter the number 765 but >765 or >765<. You can use my xmls where CAI_INTERFACE_MANAGERS are by faction.

[uanime5]

Thanks for explaining that husserlTW, I didn't realise each faction had it's own BDI entry. That helps explain what they're for.

I searched for the region 765 and found it was in the BDI_pools of all the emergent factions, though it didn't appear in any faction that hadn't emerged. I also searched for its settlement ID (2186), building_slot ID (2179), and a region_slot ID (2188). The settlement ID and region_slot ID were in other factions' BDI pools but the building_slot ID was only in Britain's BDI_pool. This explains the following:

1) As building_slot IDs are only in the owning faction's BDI they only need a small number of BDI entries.

2) As region, settlement, and region_slot IDs in the BDI_pool for every faction they require a large number of BDI_pool entries.


I did compare the BDI_pools from a hybrid startpos to a normal startpos using winmerge in another thread and found many were half to a third of the size and were missing a lot of information. For example the size of the normal bdi_pool_0002 (Britain) is 6.38 MB but the hybrid startpos version is only 2.46 MB. Unsurprisingly the BDI_pools of unemerged factions are about the same size in both version.

I noticed that some of the entries were different but I don't know what this represents.

[husserlTW]

These connections are logical if you think cases like protectorates, trade agrements, etc. But we have to search and find exactly what is about. We gonna have an interesting weekend...

[Cursisanafou]

That could explain why when i make hybrid startpos the game tends to suck. =P

[Geronimo2006]

Is there a way of changing the form of government without it causing the game to crash? I would like to correct some historical errors in governmental system such as Denmark and Portugal being Constitutional Monarchies when historically they were Absolute Monarchies in the timeframe of this game.

[husserlTW]

If it was like this then after 1st turn that the game is based on the save (like hybrid) of previous turn game would suck too, but it is not like this, is it? The fact that hybrid is smaller does not mean that game will follow a certain path every time or it will be easier. Play for example with a hybrid as a minor faction on H or VH and you see what I am talking about, you will loose even if you do not correct the Bonus Values in hybrid and leave it in your favorite.

[husserlTW]

Is there a way of changing the form of government without it causing the game to crash? I would like to correct some historical errors in governmental system such as Denmark and Portugal being Constitutional Monarchies when historically they were Absolute Monarchies in the timeframe of this game.

We've talked about it, didn't we?

http://www.twcenter.net/forums/showthread.php?p=7462121#post7462121

It is an easy change. Just remember that there are 2 places where you'll make the change. If you have difficulties upload your file to fix it.

[Geronimo2006]

We've talked about it, didn't we?

http://www.twcenter.net/forums/showthread.php?p=7462121#post7462121

It is an easy change. Just remember that there are 2 places where you'll make the change. If you have difficulties upload your file to fix it.

You said:

t is an easy modification to make it. You have to make 2 changes: one in FACTION\GOVERNMENT that is usually forgotten and causes CTD and the second in GOV_IMP. That's all.

Did you mean that the first change causes CTD or that forgetting to make the first change causes CTDs?

[husserlTW]

The second one.