Kingdoms Sentinel: Runtime Operations Manager

[Augustus Lucifer]

After looking at what Germanicu5 did with his AI Switcher program, and installing a handy program called DebugView, I think I've found that I can change the files the game runs with depending on the faction chosen. The process might be fairly involved, or it might be fairly easy, but I'm certain that it's plausible.

What I had been hoping for is a way to identify that a player switched factions, without needing any input from them besides their selection of that faction. I found that in the form of faction map and faction unit, which are searched for to display when you click on a faction. This is the exact lines in question that DebugView printed:

00048958 102.62872314 [1844] 16:45:00.046 [system.io] [trace] file open,,auh-newbuild/data/world/maps/campaign/imperial_campaign/map_goryeo.tga,544
00048959 102.62881470 [1844] 16:45:00.046 [system.io] [info] open: found auh-newbuild/data/world/maps/campaign/imperial_campaign/map_goryeo.tga (from: F:\Modding Files\AUH - Test Environment)
00048960 102.62916565 [1844] 16:45:00.046 [system.io] [trace] file open,,data/menu/symbols/FE_faction_units/goryeo.tga,,not found
00048961 102.62925720 [1844] 16:45:00.046 [system.io] [trace] pack open,packs/localized.pack,data/menu/symbols/FE_faction_units/goryeo.tga,,not found
00048962 102.62934875 [1844] 16:45:00.046 [system.io] [trace] pack open,packs/data_1.pack,data/menu/symbols/FE_faction_units/goryeo.tga,,not found
00048963 102.62942505 [1844] 16:45:00.046 [system.io] [warning] open: data/menu/symbols/FE_faction_units/goryeo.tga is missing

Those lines are extremely important, because they appear to only show up when you have your selection on that faction, unlike the buttons which seem to load randomly(or maybe as your cursor scrolls over them?). What I'll need to do is get a program to fluently read the output from DebugView and latch onto 'hot words' like /FE_faction_units/goryeo.tga. If it picks that up it should be able to assume that the current selection is Goryeo and run other processes off that. It will obviously have to loop until the campaign is loaded, but that shouldn't be much of an issue.

You might be wondering why this is a big deal. It's a big deal because it means we don't need to try and trick the game to do certain things for certain factions in-game, and we can set it up separately. To provide just one example, let's look at challenge factions. Challenge factions are what we envision as antagonists that only appear in certain campaigns for the player. For the Mongols this would be other Mongol tribes, for the Tibetans it would be other Monastic orders, for Pagan it might be Mon nationalists. The goal of having these as non-rebel is to make their AI less passive and more driven to oppose the player.

Trying to do this by altering the scenario once the game starts is nearly impossible. You have to spawn armies to conquer cities and set everything up through script, assuming it even works as intended. Other behavior and things that make up that faction can't be altered because there's no scripting commands to do so.

HOWEVER! If we can have a program that works like this one, we can make the files for each campaign different. The Tibetan campaign can have a number of built-in shadow slots, complete with insignia, names, positioning, etc. right from the start. All this involves then is having this handy program that's running in the background switch in new descr_strat and other files depending on the faction selected by the player. Voila, every campaign is noticeably different without the scripting overhead and without putting the burden on the end user to have to switch files in and out or only play one faction.

Aside from the problem of a noob programmer like me trying to write this utility, there's only two issues I foresee at this juncture. The first is that there might need to be a notice like "Leave 5 seconds after choosing faction for files to re-populate.", so the utility has enough time to realize what faction has been selected and move files around. That's not really an issue, and it may not even be necessary because programs tend to work in milliseconds. The second is that saved games work differently than newly started campaigns, because there's no faction select screen. Assuming I can get basic functionality for brand new games working, I'll then take a look at if there's anything to latch onto for saved games.

This has made me a very happy camper. I'm going to probably be attacking some programming eBooks now to see if I can piece this utility together in time for the Tibetan preview.

---

Update: There's also enough information in the log for me to ascertain if a player has chosen to play a Custom Battle or a Campaign game. This should make it possible to also switch out the separate EDU files without forcing the player to restart the game and select which they intend to play. Sadly I can't say the same about saved games right now, since nothing at all loads into the debug log when a save is selected it seems, and the interval between the save being loaded and the files being cached is under a second once 'Load' is pressed, so it could prove more difficult.

---

Update #2: After examining the contents of the .sav file in a Hex editor, I'm fairly confident I could glean enough information from the strings in that file corresponding to buildings or events to ascertain which faction the player is in any given save. Having said that, the .sav file is not loaded on click it seems, but only on pressing load, and then only a mere second before it starts processing, probably not enough time to parse a 1k KB file for a string and switch some files, even if the program were written in C and optimized for speed. I'm not entirely sure that file changes are needed for saved games, since I'm not sure which files it actually loads and which were already loaded into memory. For instance descr_strat should have no bearing on saves since it only modifies the starting scenario.

I'll have to consult someone about what files changes are considered 'save breaking', since these same changes will not need switching to work with a save game since they're pre-loaded into memory via the save. Another thing is some files seem to have no issue being changed during runtime. Germanicu5 AI changer shows that the BAI files can be changed while the campaign game is being played with no issues. If enough files either are pre-loaded or can be changed during runtime(ie. aren't cached), the save 'problem' may not be a problem at all.

---

Attached three files for my own future reference, just log outputs of different things with pauses in between so I could separate out what was logged during what action.

(Please don't mention this outside of the dev forums for now. If it works as intended then I'll try to build the utility so it can be easily adapted for other mod teams and release it as a tool, but I'd prefer no pressure in that respect. If I find I can't do anything with it then I'll mention it in the workshop and see if anyone wants to take up the challenge.)

[Augustus Lucifer]

A minor update to this. After learning more programming skills and scouring the library reference, I found a module to help do some of the necessary management here and have successfully tested it to some extent. The following two lines run kingdoms.exe as a subprocess of my script, using the subprocess module(imported):

    args = ['kingdoms.exe', '@auh.cfg']
    proc = subprocess.Popen(args)

This is a very basic element of what the final program needs to look like(all it does is open it). I need to craft it so that the script runs in the background, can be easily called from a shortcut and perform all operations a BAT normally would, and redirect the I/O from the Kingdoms subprocess to something readable by the script so that it can check for dynamic log writes. It would also help to build it modularly so that the process which calls kingdoms.exe as a subprocess is not attached directly to the process which handles the switching out of files, that way I can use the subprocess manager in other tools and distribute it separately.

Sufficiently over everyone's head? Good!

[Augustus Lucifer]

Thread title changed to "Kingdoms Sentinel: Runtime Operations Manager"; this thread will be used to track progress on development of this tool.

Version 0.1 Milestones

• Enable running of Kingdoms or Medieval 2 EXE by specifying shortcut arguments for version and name of CFG file.
• Persist program in background or tray, utilize .pyw extension so no ugly console is opened. Close program upon exit of Kingdoms/M2.
• Implement tracking of sub-process I/O monitoring to check log writes dynamically.
• Create basic functionality mirroring EDU Switcher, where semantics are: read current EDU in data directory when process starts and before starting sub-process; write exact copy to '/Misc/sp campaign files/'; write copy to '/Misc/custom battle files/' where ownership lines are modified to only include home faction and no_custom attribute is removed; check for the loading of interface files while sub-process is running and use shutil module to move corresponding EDU over depending on if campaign or custom is clicked; set file back to contents of campaign file after exit of sub-process so that we aren't receiving varied EDU commits to repo.
• Distribute working version using cx_Freeze and insure functionality is there on other dev's systems.

Version 0.2 Milestones

• Change to use of INI files for tracking tool settings. See: ConfigParser module.
• Basic experimentation with descr_strat switching based on faction selection on Campaign screen.
• Explore possibilities of freezing sub-process while Sentinel code executes, as well as whether input streams from button presses can be picked up and deciphered.

With any luck I should be able to code v0.1 over the next couple days and include it in the soon-to-come installer, assuming I get the hang of the subprocess module.

[Marcion]

So basically what you're trying to accomplish with this is have these shadow factions appear for particular factions WHEN the player is playing the relevant faction, but not have them if it's the AI? Why can't we have the shadow factions with the AI playing? What exactly is the plan for when the AI is Mongolia, are they going to start united already? I think I can hear players whining about this already...

[Augustus Lucifer]

Yes, the AI will start with united regions. Every game I'm aware of has shown that when it comes to 'rebel grabbing' the player will win out, and then it's just a matter of hitting weak neighbors. Even if we wanted to have the AI need to unite a region we couldn't use shadow factions to do it, because that would necessitate a faction slot for every shadow represented in-game, so it's physically impossible. On the other hand, satisfying people who whine about the early game unification is as simple as writing a give_everything_to_faction script and prompting about it at the beginning.

[Marcion]

Ok, if we do things this way, I think it's important that the player gets a special reward for conquering their respective antagonists. We've talked about it a little already, I think. And so because of that, it would be necessary that the AIs don't eat each other. So if you're the Mongols, the Naimans can't get conquered by anyone but you. If they did it would probably get pretty unbalanced anyway. Can that be done?

[Augustus Lucifer]

It's possible to freeze and un-freeze the AI of a faction as we do with the 'scripting' faction used to hold our 'wonders'. Theoretically we could give the player a set time period to try and unite, and then open up the wider game, that way it's not like two separate games but there's actually a chance you will not be quite united at the end of a given period so there's still a challenge.

The following commands may assist this purpose:

restrict_clickable_area
restrict_clickable_rect
restrict_strat_camera
restrict_strat_radar
set_faction_undiscovered

That will be a matter to figure out later though, because we need a lot more infrastructure in place in terms of units, economy, etc in order to have a serviceable test environment for challenge faction implementation. This tool is currently just concerned with the plausibility of switching out files based on interface interactions, not the eventual application.

[Augustus Lucifer]

Small update to this. Figured out how to check that the subprocess is still open. The following code uses preset executable arguments, runs kingdoms, then checks every 20 seconds it's open and prints a statement to the console if it is(for checking purposes), and finally once it terminates and proc.returncode is set to a value it stops the loop. Then it waits 21-40 seconds to print "End?" and a simple carriage return(<Enter>) will end the program.

import subprocess
import time

def main():
    args = ['kingdoms.exe', '@auh.cfg']
    proc = subprocess.Popen(args)
    term = ''
    while term == '':
        proc.poll()
        if proc.returncode == None:
            print("Program is still open.")
        elif proc.returncode != None:
            term = 1
        time.sleep(20)
    input("End?")

if __name__ == '__main__': main()

This is shoddy code which has a lot of room for improvement, but the important thing is it does what it's supposed to do, so I'm noting it here in case I change or lose the program and for some reason forget how I monitored process termination.

Also attached it as a RAR containing the script. If you place it in the directory containing kingdoms.exe and auh.cfg it should work for you as well, though it doesn't do anything of significance yet. It should be noted that if you don't have Python 3.1 installed on your system and marked as your default Python installation it's as good as garbage to you. In the future when the tool is actually distributed I'll use cx_Freeze to incorporate the library dependencies as DLL files so a Python implementation isn't required.