Research/WIP: UI Editing/Lua Decompiler

[alpaca]

09/07/01:

I will add decompiled UI lua files to the etw-mod-tools SVN repository (thanks LtChambers for letting me use it). You can find any I decompile here:

http://etw-mod-tools.svn.sourceforge...-mod-tools/UI/

---

Well I'm currently trying to find out how the UI in the game works, one of the important steps is bringing the Lua decompiler up to Lua 5.1 and that's what I'm mainly working on at the moment - however we also need to crack the layout files to understand what they're doing.

I know that someone made the family tree mod, and that worked by editing the layout files so maybe someone could point the creators of that mod (if it's still active) to this here thread so I can start continuing their research after I finished the decompiler. I've already been able to decompile a few of the Lua files in seemingly correct fashion and there's a lot of useful stuff in there, as an example here's utilities.lua:

-- Decompiled using luadec 2.0 UNICODE by sztupy (http://winmo.sztupy.hu)
-- Command line was: utilities.luac 

module(..., package.seeall)
local core = require("CoreUtils")
for k,v in pairs(core) do
  if string.sub(k, 1, 1) ~= "_" then
    _G[k] = v
  end
end
CAMERA_TARGET_POS_RIGHT = -1
CAMERA_TARGET_POS_CENTRE = 0
CAMERA_TARGET_POS_LEFT = 1
SPYING_UNIT_DATA_UNKNOWN = 0
SPYING_UNIT_DATA_ICON_KNOWN = 1
SPYING_UNIT_DATA_MEN_KNOWN = 2
SPYING_UNIT_DATA_GUNS_KNOWN = 4
SPYING_UNIT_DATA_XP_KNOWN = 8
SPYING_UNIT_DATA_OWNED = SPYING_UNIT_DATA_ICON_KNOWN + SPYING_UNIT_DATA_MEN_KNOWN + SPYING_UNIT_DATA_GUNS_KNOWN + SPYING_UNIT_DATA_XP_KNOWN
SPYING_DATA_LEVEL_INVALID = -1
SPYING_DATA_LEVEL_PASSIVE = 0
SPYING_DATA_LEVEL_BASIC = 2
SPYING_DATA_LEVEL_ADVANCED = 3
SPYING_DATA_LEVEL_OWNED = 4
CT_PRIMARY_GENERAL = 0
CT_SECONDARY_GENERAL = 1
CT_PRIMARY_ADMIRAL = 2
CT_SECONDARY_ADMIRAL = 3
CT_COMMODORE = 4
CT_BRIGADIER = 5
CT_NAVAL_UNIT = 6
CT_LAND_UNIT = 7
TECHNOLOGY_STATUS_RESEARCHED = 0
TECHNOLOGY_STATUS_BEING_RESEARCHED_OR_STOLEN = 1
TECHNOLOGY_STATUS_BEING_RESEARCHED = 1
TECHNOLOGY_STATUS_BEING_STOLEN = 1
TECHNOLOGY_STATUS_AVAILABLE = 2
TECHNOLOGY_STATUS_AVAILABLE_TO_STEAL_OR_TRADE = 3
TECHNOLOGY_STATUS_UNAVAILABLE = 4
TECHNOLOGY_STATUS_NOT_PRESENT = 5
RPTS_GREYED = 0
RPTS_NORMAL = 1
RPTS_SELECTED = 2
RPTS_HIGHLIGHTED = 3
NUM_RPT_STATES = 4
battle_timeout_limits = {-1, 20, 40, 60}
tech_status_strings = {"TECHNOLOGY_STATUS_RESEARCHED", "TECHNOLOGY_STATUS_BEING_RESEARCHED_OR_STOLEN", "TECHNOLOGY_STATUS_AVAILABLE", "TECHNOLOGY_STATUS_AVAILABLE_TO_STEAL_OR_TRADE", "TECHNOLOGY_STATUS_UNAVAILABLE", "TECHNOLOGY_STATUS_NOT_PRESENT"}
commander_typenames = {"CT_PRIMARY_GENERAL", "CT_SECONDARY_GENERAL", "CT_PRIMARY_ADMIRAL", "CT_SECONDARY_ADMIRAL", "CT_COMMODORE", "CT_BRIGADIER", "CT_NAVAL_UNIT", "CT_LAND_UNIT"}
CreateCharacterCard = function(l_1_0, l_1_1, l_1_2, l_1_3, l_1_4, l_1_5, l_1_6, l_1_7, l_1_8)
  local images = {"{Flags:1}" .. l_1_0.Flag}
  if l_1_0.CardImage ~= nil and #l_1_0.CardImage > 0 then
    images[#images + 1] = "{" .. l_1_1 .. ":1}" .. l_1_0.CardImage
  end
  if l_1_0.ShowAsCharacter == false then
    images[#images + 1] = "{" .. l_1_1 .. ":2}" .. l_1_0.MaskImage
  end
  if l_1_0.Attributes.PrimaryAttributePath ~= nil then
    images[#images + 1] = "{CharacterAttribute:1}" .. l_1_0.Attributes.PrimaryAttributePath
  end
  local card = UIComponent(Component.CreateComponentFromTemplate("CampaignCharacterCard", l_1_1, l_1_2, l_1_3, l_1_4, images))
  card:LuaCall("SetOwnerAndId", UIComponent(l_1_2):Id(), l_1_1, l_1_0.Address)
  l_1_0.ShowAttributes = l_1_7
  l_1_0.ShowFlag = l_1_5
  l_1_0.ShowOuterRing = l_1_6
  card:LuaCall("Initialise", l_1_0, l_1_8)
  return card:Address()
end

CreateArmedCitizenryCard = function(l_2_0, l_2_1, l_2_2, l_2_3, l_2_4)
  local image_list = {}
  local mask_exists = vfs.exists(l_2_0.Icon .. "_mask.tga") == nil
  out.shane("CreateArmedCitizenryCard: unit_record.Icon = " .. tostring(l_2_0.Icon))
  if SPYING_DATA_LEVEL_BASIC < l_2_0.spying_data_level then
    out.shane("Adding icon: " .. l_2_0.Icon .. " to image list")
    image_list[1] = "{" .. l_2_1 .. ":1}" .. l_2_0.Icon .. ".tga"
    if mask_exists == true then
      out.shane("Adding mask icon: " .. l_2_0.Icon .. "_mask.tga" .. " to image list")
      image_list[#image_list + 1] = "{" .. l_2_1 .. ":2}" .. l_2_0.Icon .. "_mask.tga"
    end
  end
  local card = Component.CreateComponentFromTemplate("ArmedCitizenryCard", l_2_1, l_2_2, l_2_3, l_2_4, image_list)
  UIComponent(card):LuaCall("Initialise", l_2_0)
  return card
end

CreateUnitCard = function(l_3_0, l_3_1, l_3_2, l_3_3, l_3_4, l_3_5, l_3_6)
  assert(l_3_2 ~= nil, "Attempting to create a unit card with no parent.")
  l_3_0.primary_commander = l_3_0.CommanderType == CT_PRIMARY_GENERAL or l_3_0.CommanderType == CT_PRIMARY_ADMIRAL
  l_3_0.is_general_or_admiral = l_3_0.primary_commander == true or l_3_0.CommanderType == CT_SECONDARY_GENERAL or l_3_0.CommanderType == CT_SECONDARY_ADMIRAL
  l_3_0.showing_in_character_format = l_3_0.DisplayAsUnit == false and l_3_0.is_general_or_admiral == true
  l_3_0.mask_exists = vfs.exists(l_3_0.Icon .. "_mask.tga") == nil
  l_3_0.selectable = l_3_6
  l_3_0.moveable = l_3_5
  local image_list = {}
  if bit.band(l_3_0.knowledge_mask, SPYING_UNIT_DATA_ICON_KNOWN) ~= 0 then
    if l_3_0.showing_in_character_format == true and string.len(l_3_0.Portrait) > 0 then
      out.shane("Adding portrait: " .. l_3_0.Portrait .. " to image list")
      image_list[1] = "{" .. l_3_1 .. ":1}" .. l_3_0.Portrait
    else
      out.shane("Adding icon: " .. l_3_0.Icon .. " to image list")
      image_list[1] = "{" .. l_3_1 .. ":1}" .. l_3_0.Icon .. ".tga"
    end
    if l_3_0.mask_exists == true and l_3_0.showing_in_character_format == false then
      out.shane("Adding mask icon: " .. l_3_0.Icon .. "_mask.tga" .. " to image list")
      image_list[#image_list + 1] = "{" .. l_3_1 .. ":2}" .. l_3_0.Icon .. "_mask.tga"
    end
  end
  local card = Component.CreateComponentFromTemplate("CampaignUnitCard", l_3_1, l_3_2, l_3_3, l_3_4, image_list)
  UIComponent(card):LuaCall("Initialise", l_3_0)
  return card
end

FitTextToComponent = function(l_4_0, l_4_1, l_4_2)
  local resized = false
  l_4_1 = UIComponent(l_4_0:Find(l_4_1))
  local c_w = l_4_1:Width()
  local font_details = l_4_1:GetStateTextDetails()
  local t_w, t_h, t_l = l_4_1:SetStateText(l_4_2)
  local i = string.length(l_4_2)
  if c_w < t_w + font_details.XOffset or t_l > 1 then
    resized = true
  end
  while c_w < t_w + font_details.XOffset or t_l > 1 do
    local t = string.sub(l_4_2, 1, i) .. "..."
    t_w, t_h, t_l = l_4_1:SetStateText(t)
    i = i - 1
  end
  l_4_1:SetTooltipText(l_4_2, true)
  return resized
end

CreateDiplomacyFlags = function(l_5_0, l_5_1)
  local p = UIComponent(l_5_0)
  p:DestroyChildren()
  local max_w = p:Width()
  local x = 0
  local flag_width = 0
  for k,v in pairs(l_5_1) do
    images = {"{flag:1}" .. v.Flag}
    flag = UIComponent(Component.CreateComponentFromTemplate("faction_flag_small", "flag", l_5_0, x, 0, images))
    flag:SetTooltipText(v.Name, true)
    flag_width = flag:Width()
    x = x + flag_width
  end
  if max_w < x then
    local offset = 1
    local start_x = 0
    local y = 0
    local p = UIComponent(l_5_0)
    while max_w - offset < x and offset < flag_width - 1 do
      start_x, y = p:Position()
      x = 0
      for n = 0, p:ChildCount() - 1 do
        local child = UIComponent(p:Find(n))
        child:MoveTo(start_x + x, y)
        x = x + child:Width() - offset
      end
      offset = offset + 1
    end
  end
end

PositionTooltip = function(l_6_0, l_6_1)
  local this = UIComponent(l_6_0)
  local cx, cy, screenWidth, screenHeight = Component.CursorPosition()
  local x = cx
  local y = cy
  local w, h = this:Dimensions()
  local cursor = Cursor()
  local cursorWidth, cursorHeight = cursor:DistanceToBL()
  if l_6_1 == true then
    y = y - h
  else
    y = y + cursorHeight
  end
  if screenHeight < y + h then
    y = screenHeight - h
  end
  if y < 0 then
    y = 0
  end
  x = x + cursorWidth
  if screenWidth < x + w then
    x = screenWidth - w
  end
  if x < cx + cursorWidth and y < cy + cursorHeight and cy < y + h then
    x = cx - w
  end
  this:MoveTo(x, y)
end

MouseOver = function(l_7_0)
  if l_7_0:Visible() then
    local m_x, m_y = Component.CursorPosition()
    local c_x, c_y = l_7_0:Position()
    return c_x <= m_x and m_x < c_x + l_7_0:Width() and c_y <= m_y and m_y < c_y + l_7_0:Height()
  end
  return false
end

AddLine = function(l_8_0, l_8_1, l_8_2, l_8_3, l_8_4, l_8_5, l_8_6)
  local line_a, line_w
  if l_8_6 == nil then
    l_8_6 = UIComponent(Component.CreateComponentFromTemplate("general_purpose_pixel", "line", l_8_0, l_8_1, l_8_2))
  else
    local px, py = UIComponent(l_8_0):Position()
    l_8_6:MoveTo(px + l_8_1, py + l_8_2)
  end
  local a = l_8_3 - l_8_1
  local o = l_8_4 - l_8_2
  local h = a * a + o * o ^ 0.5
  line_w = h
  line_a = math.asin(o / h)
  l_8_6:Resize(line_w, l_8_6:Height())
  l_8_6:SetImageRotation(0, line_a, -line_w * 0.5, 0)
  l_8_6:SetImageColour(0, l_8_5.r * 255, l_8_5.g * 255, l_8_5.b * 255, 255)
  return l_8_6
end

PropagateShader = function(l_9_0, l_9_1, l_9_2, l_9_3, l_9_4, l_9_5)
  local root = UIComponent(l_9_0)
  for i = 0, root:ChildCount() - 1 do
    PropagateShader(root:Find(i), l_9_1, l_9_2, l_9_3, l_9_4, l_9_5)
  end
  root:ShaderTechniqueSet(l_9_1, true)
  root:ShaderVarsSet(l_9_2, l_9_3, l_9_4, l_9_5, true)
end

OffsetFrom = function(l_10_0, l_10_1)
  local a_x, a_y = l_10_0:Position()
  local b_x, b_y = l_10_1:Position()
  return b_x - a_x, b_y - a_y
end

TableEquality = function(l_11_0, l_11_1)
  for k,v in pairs(l_11_0) do
    if l_11_1[k] == nil or l_11_1[k] ~= v then
      return false
    end
  end
  for i,v in ipairs(l_11_0) do
    if l_11_1[i] == nil or l_11_1[i] ~= v then
      return false
    end
  end
  return true
end

local exportstring = function(l_12_0)
  l_12_0 = string.format("%q", l_12_0)
  l_12_0 = string.gsub(l_12_0, "\\\n", "\\n")
  l_12_0 = string.gsub(l_12_0, "\r", "\\r")
  l_12_0 = string.gsub(l_12_0, string.char(26), "\"..string.char(26)..\"")
  return l_12_0
end

SaveTable = function(l_13_0, l_13_1)
  local charS, charE = "   ", "\n"
  local file, err = nil, nil
  if not l_13_1 then
    file = {write = function(l_1_0, l_1_1)
    l_1_0.str = l_1_0.str .. l_1_1
   end, str = ""}
    charS, charE = "", ""
  elseif l_13_1 == true or l_13_1 == 1 then
    charS, charE, file = "", "", io.tmpfile()
  else
    file, err = io.open(l_13_1, "w")
    if err then
      return _, err
    end
  end
  local tables, lookup = {l_13_0}, {l_13_0 = 1}
  file:write("return {" .. charE)
  for idx,t in ipairs(tables) do
    if l_13_1 and l_13_1 ~= true and l_13_1 ~= 1 then
      file:write("-- Table: {" .. idx .. "}" .. charE)
    end
    file:write("{" .. charE)
    local thandled = {}
    for i,v in ipairs(t) do
      thandled[i] = true
      if type(v) ~= "userdata" then
        if type(v) == "table" then
          if not lookup[v] then
            table.insert(tables, v)
            lookup[v] = #tables
          end
          file:write(charS .. "{" .. lookup[v] .. "}," .. charE)
        else
          if type(v) == "function" then
            file:write(charS .. "loadstring(" .. exportstring(string.dump(v)) .. ")," .. charE)
          else
            if type(v) ~= "string" or not exportstring(v) then
              local value = tostring(v)
            end
            file:write(charS .. value .. "," .. charE)
          end
        end
      end
    end
    for i,v in pairs(t) do
      if not thandled[i] and type(v) ~= "userdata" then
        if type(i) == "table" then
          if not lookup[i] then
            table.insert(tables, i)
            lookup[i] = #tables
          end
          file:write(charS .. "[{" .. lookup[i] .. "}]=")
        else
          if type(i) ~= "string" or not "[" .. exportstring(i) .. "]" then
            local index = string.format("[%d]", i)
          end
          file:write(charS .. index .. "=")
        end
        if type(v) == "table" then
          if not lookup[v] then
            table.insert(tables, v)
            lookup[v] = #tables
          end
          file:write("{" .. lookup[v] .. "}," .. charE)
        else
          if type(v) == "function" then
            file:write("loadstring(" .. exportstring(string.dump(v)) .. ")," .. charE)
          else
            if type(v) ~= "string" or not exportstring(v) then
              local value = tostring(v)
            end
            file:write(value .. "," .. charE)
          end
        end
      end
    end
    file:write("}," .. charE)
  end
  file:write("}")
  if not l_13_1 then
    return file.str .. "--|"
  elseif l_13_1 == true or l_13_1 == 1 then
    file:seek("set")
    return file:read("*a") .. "--|"
  else
    file:close()
    return 1
  end
end

LoadTable = function(l_14_0)
  if string.sub(l_14_0, -3, -1) == "--|" then
    tables, err = loadstring(l_14_0), l_14_0
  else
    tables, err = loadfile(l_14_0), l_14_0
  end
  if err then
    return _, err
  end
  tables = tables()
  for idx = 1, #tables do
    local tolinkv, tolinki = {}, {}
    for i,v in pairs(tables[idx]) do
      if type(v) == "table" and tables[v[1]] then
        table.insert(tolinkv, {i, tables[v[1]]})
      end
      if type(i) == "table" and tables[i[1]] then
        table.insert(tolinki, {i, tables[i[1]]})
      end
    end
    for _,v in ipairs(tolinkv) do
      tables[idx][v[1]] = v[2]
    end
    for _,v in ipairs(tolinki) do
      tables[idx][v[2]], tables[idx][v[1]] = tables[idx][v[1]]
    end
  end
  return tables[1]
end

SortCardsByRank = function(l_15_0, l_15_1, l_15_2, l_15_3)
  local sorted_cards = {}
  local secondary_index = 1
  for s = 1, #l_15_0 do
    if l_15_0[s].CommanderType == l_15_1 then
      sorted_cards[1] = {}
      sorted_cards[1] = l_15_0[s]
      secondary_index = 2
  else
    end
  end
  for s = 1, #l_15_0 do
    if l_15_0[s].CommanderType == l_15_2 or l_15_0[s].CommanderType == l_15_3 then
      sorted_cards[secondary_index] = {}
      sorted_cards[secondary_index] = l_15_0[s]
      secondary_index = secondary_index + 1
    end
  end
  for s = 1, #l_15_0 do
    if l_15_0[s].CommanderType ~= l_15_1 and l_15_0[s].CommanderType ~= l_15_2 and l_15_0[s].CommanderType ~= l_15_3 then
      sorted_cards[secondary_index] = {}
      sorted_cards[secondary_index] = l_15_0[s]
      secondary_index = secondary_index + 1
    end
  end
  return sorted_cards
end

table.load = LoadTable
table.save = SaveTable

Such findings lead me to believe that the UI is completely soft-coded in Lua this time around, so after fully cracking it we should be able to do anything the current UI is able to do (and maybe more)! For example it would be possible to modify the behaviour of the replenish button, or its availability - which was one of my first ideas. But there's far more, of course: We're working on the idea of faction-specific user interfaces like in Kingdoms for example

So, any help in deciphering the layout files (files without extension in the UI folder) while I'm working on the scripts would be highly welcome. So far I know the core UI is "calling" the root of the files which provides the look of the thing and events like mouse down, button pressed and so on. A lot of the UI components (each layout file is one UI component) also provide sub-components, like for example the main campaign UI a veneer_DY component, that are callable by script. The question is what does all the binary data mean?

[HellFell]

Alpaca, thank you for your work. I wonder if it is possible to create a button called "Fire!" that will work like broadside button in naval battles? I mean it was promised in previews and interviews but it is not implemented for land battles. I believe it would be a wonderful addition to the game. What do you think about it, is it possible?

[alpaca]

Alpaca, thank you for your work. I wonder if it is possible to create a button called "Fire!" that will work like broadside button in naval battles? I mean it was promised in previews and interviews but it is not implemented for land battles. I believe it would be a wonderful addition to the game. What do you think about it, is it possible?

I'm inclined to say no as we can't really give new commands to the engine only use existing ones differently.

[just]

You can attach lua code to buttons, so you can add buttons that do anything you can script. I don't remember exactly what is scriptable in battles, but firing might be hard, because I think it needs a target.

[alpaca]

You can attach lua code to buttons, so you can add buttons that do anything you can script. I don't remember exactly what is scriptable in battles, but firing might be hard, because I think it needs a target.

Yes but even if you can tell a unit to attack this will do the same as the attack command when the player right-clicks and not be an instant fire button.

[33orion77]

I belive the guy that made the Royal family tree is GeorgiaPeanuts. His thread is right here: http://www.twcenter.net/forums/showthread.php?t=246530
but I'm affraid he has been inactive for a good two months...
I was also wondering... Do you think it could be possible to create a feature like the papacy in MTW2. The idea would be to create the Holy Roman Empire using this feature ( election with the faction leaders of the german states, influence in the nation's diplomacy etc...) just like to have your thoughs on that...

[alpaca]

I belive the guy that made the Royal family tree is GeorgiaPeanuts. His thread is right here: http://www.twcenter.net/forums/showthread.php?t=246530
but I'm affraid he has been inactive for a good two months...
I was also wondering... Do you think it could be possible to create a feature like the papacy in MTW2. The idea would be to create the Holy Roman Empire using this feature ( election with the faction leaders of the german states, influence in the nation's diplomacy etc...) just like to have your thoughs on that...

Yes and no. I think it might be possible with a lot of work and talent but you couldn't make the AI do anything nor influence bilateral relations.

[Irmo]

Sorry for what may be a stupid question, but how did you get the Luadec to work? I tried downloading it from LuaForge, but all I got were the files, and when I tried compiling it, I got a few error messages. Did you find a compiled version, and if not, what did you use to compile it?

Thanks for all the help.

[alpaca]

Sorry for what may be a stupid question, but how did you get the Luadec to work? I tried downloading it from LuaForge, but all I got were the files, and when I tried compiling it, I got a few error messages. Did you find a compiled version, and if not, what did you use to compile it?

Thanks for all the help.

I'm using (and changing) the source files, to compile I use Visual C++ 2008 Express. However the luadec won't work because there's no 5.1 version yet so my main work is to bring it up-to-date to work with 5.1 - a difficult task for me because the source code is very badly commented and I have to figure out what it does on-the-fly.

[Irmo]

Thanks - I hope you get it working.

[alpaca]

Ok, my first UI editing job: I enlarged the banner size for the main menu. I'm beginning to understand a bit about the layout files, they mainly consist of images that have an ID and are later puzzled together (every image belongs to a container). For example see the logo container below.

First seem to be two integers I don't know about, then the ID of the container, then an ASCII string (prepended by its length) with the container name, then x and y offset of the top left corner relative to the parent, then some flags I don't know about, then again the name of the component (this doesn't have to be the same as the container name), then a -1, some stuff I don't know.

Next up is the ID of the image, the path to the image, width, height, -1, more unknowns followed by another ID for the "NewState" part (which is I think used when you want to write something), again width and height for the text, some text settings I don't know, the font name and size, then the string "normal_t0" followed by another 20 unknown bytes, then the number of children.

Each child entry has the ID followed by x,y,w,h,-1,unknown,horizontal flipping (bool),vertical flipping,something like a child ID (I think) and more stuff I don't know up to events_end

More to come anyways, if someone would like to contribute that'd be highly welcome.

[33orion77]

Thanks but I can't help you further...

[Dewy]

Its about time CA caught up to the latest trends and started using LUA more. hope you get this going

[Sedziwoj]

(...) However the luadec won't work because there's no 5.1 version yet (...)

may be helpful:
http://luadec51.luaforge.net/index.html

and my big request for you - there is no binaries, only source code on project site, so can you compile this source and put binaries here to let others to use it? it will be big, big, big favor to me and probably other of us

[alpaca]

may be helpful:
http://luadec51.luaforge.net/index.html

and my big request for you - there is no binaries, only source code on project site, so can you compile this source and put binaries here to let others to use it? it will be big, big, big favor to me and probably other of us

No, unfortunately not helpful. I'm using it as a base but there's a lot of unhandled stuff, in fact it can decompile few of the important files. And yes, I'll post the executables as soon as they work properly. I'd rather not release it now because the use is very limited and you'd rather break something if you don't know exactly what you're doing.

[Sedziwoj]

any news in topic?

[alpaca]

No, I didn't really have time to mod for the last few weeks and won't have any for another two or so.

[Sedziwoj]

thanks for info, i m really impatient about my friend programmer promised to check it also in free time so if he made some even half working tool i ll place it there

[alpaca]

thanks for info, i m really impatient about my friend programmer promised to check it also in free time so if he made some even half working tool i ll place it there

Well if you have someone to work on it, here's the current version of my sources (VC++ project). I wouldn't bother him overly much though as the luadec project is very badly commented and needs a lot of work getting into it. He might want to check this thing for the VM commands: http://luaforge.net/docman/view.php/...structions.pdf

[Sedziwoj]

thanks