Tutorial Modding game mechanics in DB files

[LtChambers]

Many of the core game mechanics are trapped in a new binary format that CA has started using, some kind of database. Based on some work by alpaca, I've put together a short lesson on how to modify unit stats with a hex editor. This will have to suffice until ETW releases their modding tools or we come up with a way to read and edit the db files natively.

This lesson is not for the faint of heart, hex editing is a tricky business. A very good (unfortunately not free) tool is Hex Workshop which lets you select a range of bytes and interpret those as different data types. You pretty much select a range of numbers in powers of 2 (almost always 1, 2, or 4 bytes) and see what the value is. The strings in these db files are stored as BSTR UTF16, which means that the two bytes before every string are an unsigned short integer indicating the length of the string that follows, and each character in the string is two bytes (in English versions and most other Romantic languages, the second byte of each character will almost always be 00).

The lesson is in the color coding of the long riflemen statistics below. The following excerpt of binary is from the file at:
./data/db/unit_stats_land_tables/unit_stats_land

Bytes highlighted magenta are string lengths.
Bytes highlighted green are part of a UTF-16 string.
Red bytes are ones that are potentially interesting, but I haven't figured out what they are. They could be anything and any type, but usually they will be a single delimiter byte, a 2 byte short integer, a 4 byte float, or a 4 byte long integer.
The 4 cyan bytes are a long int that sets the normal unit size of long rifle men (80).
The 4 blue bytes are a long int that sets the base accuracy of the unit (60).
The 4 bright green bytes are a long int that sets the base reloading skill of the unit (15).
The 4 orange bytes are a long int that sets the amount of ammunition the unit has (15).
The 4 teal bytes are a long int that sets the melee attack of the unit (10).
The 4 brown bytes are a long int that sets the charge bonus of the unit (4).
The 4 gray bytes are a long int that sets the defense of the unit (8).
The 4 purple bytes are a long int that sets the morale of the unit (8).

000076a9h: [COLOR=Magenta]17 00[/COLOR] [COLOR=DarkGreen]61 00 6D 00 65 00 72 00 69 00 63 00 61 00[/COLOR] ; [COLOR=Magenta]..[/COLOR][COLOR=DarkGreen]a.m.e.r.i.c.a.[/COLOR]
000076b9h: [COLOR=DarkGreen]73 00 5F 00 6C 00 6F 00 6E 00 67 00 5F 00 72 00[/COLOR] ; [COLOR=DarkGreen]s._.l.o.n.g._.r.[/COLOR]
000076c9h: [COLOR=DarkGreen]69 00 66 00 6C 00 65 00 5F 00 6D 00 65 00 6E 00[/COLOR] ; [COLOR=DarkGreen]i.f.l.e._.m.e.n.[/COLOR]
000076d9h: [COLOR=Red][COLOR=Cyan]50 00 00 00 [/COLOR]00 00 00 00 00 00 00 00[/COLOR] [COLOR=Magenta]0D 00[/COLOR] [COLOR=DarkGreen]65 00[/COLOR] ; [COLOR=Red][COLOR=Cyan]P...[/COLOR]........[/COLOR][COLOR=Magenta]..[/COLOR][COLOR=DarkGreen]e.[/COLOR]
000076e9h: [COLOR=DarkGreen]75 00 72 00 6F 00 5F 00 6F 00 66 00 66 00 69 00[/COLOR] ; [COLOR=DarkGreen]u.r.o._.o.f.f.i.[/COLOR]
000076f9h: [COLOR=DarkGreen]63 00 65 00 72 00[/COLOR] [COLOR=Red]31 00 00 00[/COLOR] [COLOR=Magenta]12 00[/COLOR] [COLOR=DarkGreen]65 00 75 00[/COLOR] ; [COLOR=DarkGreen]c.e.r.[/COLOR][COLOR=Red]1...[/COLOR][COLOR=Magenta]..[/COLOR][COLOR=DarkGreen]e.u.[/COLOR]
00007709h: [COLOR=DarkGreen]72 00 6F 00 5F 00 6C 00 6F 00 6E 00 67 00 5F 00[/COLOR] ; [COLOR=DarkGreen]r.o._.l.o.n.g._.[/COLOR]
00007719h: [COLOR=DarkGreen]72 00 69 00 66 00 6C 00 65 00 6D 00 61 00 6E 00[/COLOR] ; [COLOR=DarkGreen]r.i.f.l.e.m.a.n.[/COLOR]
00007729h: [COLOR=Magenta]13 00[/COLOR] [COLOR=DarkGreen]69 00 6E 00 66 00 61 00 6E 00 74 00 72 00[/COLOR] ; [COLOR=Magenta]..[/COLOR][COLOR=DarkGreen]i.n.f.a.n.t.r.[/COLOR]
00007739h: [COLOR=DarkGreen]79 00 5F 00 65 00 75 00 72 00 6F 00 5F 00 6C 00[/COLOR] ; [COLOR=DarkGreen]y._.e.u.r.o._.l.[/COLOR]
00007749h: [COLOR=DarkGreen]69 00 67 00 68 00 74 00[/COLOR] [COLOR=Magenta]10 00 [/COLOR][COLOR=DarkGreen]6D 00 61 00 6E 00[/COLOR] ; [COLOR=DarkGreen]i.g.h.t.[/COLOR][COLOR=Magenta]..[/COLOR][COLOR=DarkGreen]m.a.n.[/COLOR]
00007759h: [COLOR=DarkGreen]5F 00 6D 00 75 00 73 00 6B 00 65 00 74 00 5F 00[/COLOR] ; [COLOR=DarkGreen]_.m.u.s.k.e.t._.[/COLOR]
00007769h: [COLOR=DarkGreen]73 00 61 00 62 00 72 00 65 00[/COLOR] [COLOR=Magenta]12 00 [/COLOR][COLOR=DarkGreen]78 00 5F 00[/COLOR] ; [COLOR=DarkGreen]s.a.b.r.e.[/COLOR][COLOR=Magenta]..[/COLOR][COLOR=DarkGreen]x._.[/COLOR]
00007779h: [COLOR=DarkGreen]65 00 75 00 72 00 6F 00 5F 00 72 00 69 00 66 00[/COLOR] ; [COLOR=DarkGreen]e.u.r.o._.r.i.f.[/COLOR]
00007789h: [COLOR=DarkGreen]6C 00 65 00 5F 00 73 00 77 00 6F 00 72 00 64 00[/COLOR] ; [COLOR=DarkGreen]l.e._.s.w.o.r.d.[/COLOR]
00007799h: [COLOR=Red]03 00 00 00[/COLOR] [COLOR=Magenta]07 00[/COLOR] [COLOR=DarkGreen]6C 00 65 00 61 00 74 00 68 00[/COLOR] ; [COLOR=Red]....[/COLOR][COLOR=Magenta]..[/COLOR][COLOR=DarkGreen]l.e.a.t.h.[/COLOR]
000077a9h: [COLOR=DarkGreen]65 00 72 00[/COLOR] [COLOR=Red]01 00 00 00 00 00 00 01 00 30 00 01[/COLOR] ; [COLOR=DarkGreen]e.r.[/COLOR][COLOR=Red].........0..[/COLOR]
000077b9h: [COLOR=Red]00 30 00 00 00 00 00 00 00 01[/COLOR] [COLOR=Magenta]05 00[/COLOR] [COLOR=DarkGreen]72 00 69 00[/COLOR] ; [COLOR=Red].0........[/COLOR][COLOR=Magenta]..[/COLOR][COLOR=DarkGreen]r.i.[/COLOR]
000077c9h: [COLOR=DarkGreen]66 00 6C 00 65 00 [COLOR=Blue]3C 00[/COLOR][/COLOR][COLOR=Red][COLOR=Blue] 00 00 [COLOR=Lime]0F 00[/COLOR][/COLOR][COLOR=Lime] 00 00[/COLOR] 01 00[/COLOR] ; [COLOR=DarkGreen]f.l.e.[/COLOR][COLOR=Red][COLOR=Blue]<...[/COLOR][COLOR=Lime]....[/COLOR].[/COLOR]
000077d9h: [COLOR=Red]30 00[/COLOR] [COLOR=Magenta]09 00[/COLOR] [COLOR=DarkGreen]66 00 6C 00 69 00 6E 00 74 00 6C 00[/COLOR] ; [COLOR=Red]0.[/COLOR][COLOR=Magenta]..[/COLOR][COLOR=DarkGreen]f.l.i.n.t.l.[/COLOR]
000077e9h: [COLOR=DarkGreen]6F 00 63 00 6B 00[/COLOR] [COLOR=Red]01[/COLOR] [COLOR=Magenta]0C 00[/COLOR][COLOR=DarkGreen] 6D 00 75 00 73 00 6B [/COLOR]; [COLOR=DarkGreen]o.c.k..[/COLOR][COLOR=DarkGreen]..m.u.s.k[/COLOR]
000077f9h: [COLOR=DarkGreen]00 65 00 74 00 5F 00 72 00 69 00 66 00 6C 00 65 [/COLOR]; [COLOR=DarkGreen].e.t._.r.i.f.l.e[/COLOR]
00007809h: [COLOR=DarkGreen]00[/COLOR] [COLOR=Red][COLOR=DarkOrange]0F 00 00 00[/COLOR] 01[/COLOR] [COLOR=Magenta]0A 00[/COLOR] [COLOR=DarkGreen]66 00 6F 00 6F 00 74 00[/COLOR] ; [COLOR=Red][COLOR=DarkGreen].[/COLOR][COLOR=DarkOrange]....[/COLOR].[/COLOR][COLOR=Magenta]..[/COLOR][COLOR=DarkGreen]f.o.o.t.[/COLOR]
00007819h: [COLOR=DarkGreen]5F 00 72 00 69 00 66 00 6C 00 65 00[/COLOR] [COLOR=Magenta]05 00[/COLOR] [COLOR=DarkGreen]73 00[/COLOR] ; [COLOR=DarkGreen]_.r.i.f.l.e.[/COLOR][COLOR=Magenta]..[/COLOR][COLOR=DarkGreen]s.[/COLOR]
00007829h: [COLOR=DarkGreen]77 00 6F 00 72 00 64 00[/COLOR] [COLOR=Red][COLOR=Teal]0A 00 00 00[/COLOR] [COLOR=Sienna]04 00 00 00[/COLOR][/COLOR] ; [COLOR=DarkGreen]w.o.r.d.[/COLOR][COLOR=Red][COLOR=Teal]....[/COLOR][COLOR=Sienna]....[/COLOR][/COLOR]
00007839h: [COLOR=Red][COLOR=DimGray]08 00 00 00[/COLOR] 00 00 00 00[/COLOR] [COLOR=Magenta]0A 00[/COLOR] [COLOR=DarkGreen]66 00 6F 00 6F 00[/COLOR] ; [COLOR=Red][COLOR=DimGray]....[/COLOR]....[/COLOR][COLOR=Magenta]..[/COLOR][COLOR=DarkGreen]f.o.o.[/COLOR]
00007849h: [COLOR=DarkGreen]74 00 5F 00 73 00 77 00 6F 00 72 00 64 00 01 00[/COLOR] ; [COLOR=DarkGreen]t._.s.w.o.r.d...[/COLOR]
00007859h: [COLOR=DarkGreen]30 00[/COLOR] [COLOR=Magenta]18 00[/COLOR] [COLOR=DarkGreen]64 00 72 00 69 00 6C 00 6C 00 5F 00[/COLOR] ; [COLOR=DarkGreen]0.[/COLOR][COLOR=Magenta]..[/COLOR][COLOR=DarkGreen]d.r.i.l.l._.[/COLOR]
00007869h: [COLOR=DarkGreen]73 00 65 00 74 00 5F 00 69 00 6E 00 66 00 61 00[/COLOR] ; [COLOR=DarkGreen]s.e.t._.i.n.f.a.[/COLOR]
00007879h: [COLOR=DarkGreen]6E 00 74 00 72 00 79 00 5F 00 6C 00 69 00 67 00[/COLOR] ; [COLOR=DarkGreen]n.t.r.y._.l.i.g.[/COLOR]
00007889h: [COLOR=DarkGreen]68 00 74 00[/COLOR] [COLOR=Magenta]05 00[/COLOR] [COLOR=DarkGreen]65 00 6C 00 69 00 74 00 65 00[/COLOR] ; [COLOR=DarkGreen]h.t.[/COLOR][COLOR=Magenta]..[/COLOR][COLOR=DarkGreen]e.l.i.t.e.[/COLOR]
00007899h: [COLOR=Red][COLOR=Indigo]08 00 00 00[/COLOR] 03 00 00 00 CD CC 8C 3F 00 00 00 40 [/COLOR]; [COLOR=Red][COLOR=Indigo]....[/COLOR]....ÍÌŒ?...@[/COLOR]
000078a9h: [COLOR=Red]CD CC 0C 40 00 00 80 40 00 00 00 00 00 00 00 00[/COLOR] ; [COLOR=Red]ÍÌ.@..€@........[/COLOR]
000078b9h: [COLOR=Red]00 00 20 41 00 00 00 00 00 01 01 00 01 00 00 01[/COLOR] ; [COLOR=Red].. A............[/COLOR]
000078c9h: [COLOR=Red]01 01 01 01 00 00 00 00 48 42 00 00 96 42 00 00[/COLOR] ; [COLOR=Red]........HB..–B..[/COLOR]
000078d9h: [COLOR=Red]C8 42 00 00 00 00 00 00 00 01 01 01 00 00 00 00[/COLOR] ; [COLOR=Red]ÈB..............[/COLOR]

This concludes today's Hex Editing 101 class. I may put together some more lessons on editing other db files.

 

[IndoPruss]

BinaryReader in C# can read those bytes you stated, I'll try now.

 

[Serben]

YEEEEESSSSSS! Finally! You guys own! Keep up the good work and please hurry up and figure out how all this stuff works so we can start editing stuff.


+rep for you.

 

[Iaius Statius Laurentius]

Does the unit stats file also contain information for unit caps on special units, like the Guards units or the unique ships?

The USS Constitution had three sister ships, and it's annoying to be limited to just building one.

 

[stupiddk]

That code looks like the code found in mount and blade. If they release tools like those found in Mount and Blade, there can be crazy mods that change more game mechanics than we could have hoped for. Lets wait for the first real patch next week.

 

[Serben]

So how's it going? Have you, Ltchambers, or alpaca figured out how to make heads or tails of this?

 

[Quixote07]

maybe most moddable TW isn't that far off, when they release the modding tools they said they would.

 

[Lord_Phan]

Can you please check file Startpos.esf in Campaign/Main It's possible that may be where the factions are located. But it's in some type of binary code.

 

[aronnov]

can anyone host a db file? I have the tools in front of me right now where i feel like i could make some progress. just dont have the game within 40 miles of me atm

 

[Serben]

You have the tools? Then post them in an attachment, as soon as possible so we can all start editing! Here are a few of the db files:

 

[Serben]

can anyone host a db file? I have the tools in front of me right now where i feel like i could make some progress. just dont have the game within 40 miles of me atm

And where the hell did you get the tools anyway?

 

[Lord_Phan]

Maybe he's a computer programmer?

 

[Serben]

Well either way, i gave you some db files aronnov, so please upload the tools if you have them.

 

[aronnov]

anyone have luck with BinaryReader in c sharp? I can't get it to read.

oh and thanks for the files serb

 

[Hexcaliber]

I think he means a simple hex editor to examine the files contents, not something more elaborate like a custom editor. ***EDIT**Ack I must of hit reply just as he finished entering his post saying what he was trying to use, lol, nm

 

[Serben]

So what's going on? Do you have the tools or not? If you do, can you please post them.

 

[aronnov]

not any that are working. i'm up there with the TC. the only way to edit these until a tool directly from CA is to edit the hex values with trial and error.

Has anyone had a direct response from CA?

 

[sage2]

Basically, it's up to hex editing float values. For very simple things, like changing a simple value, this is a PITA. For more complex things, like rebalancing units, it's very tough / close to impossible unless you're autistic, caffeinated and unemployed. Finally, anything more then that, such as appending a string to a unit, it's a no go, since the binary will then not load.

I use this website for converting numbers into floats: http://babbage.cs.qc.cuny.edu/IEEE-754/Decimal.html (although most hex eds should do the conversion for you). Don't forget that some things might be stored as integers (e.g. just normal base 16). Also, don't forget that x86 reads backwards, so, the number 51 (424C0000) would actually be represented in hex as 00 00 4C 42, and -12.1 (C141999A) would be 9A 99 41 C1. Make sure you don't use an 'insert' function with your hex editor; use replace. Insert will almost certainly fail.

I use the XVI32 free hex editor, but they're all pretty much the same.

If all this is utter Greek to you, then you will have a pretty steap learning curve.

Personally, I find it utterly, and immensely retarded that CA has gone to a proprietary data format. I mean... what the hell? All it does it make any serious modding extremely time consuming, or require the community to write it's own tools that will format and write the data files. I would presume it's so they can make more money selling units to the community, or something. But, seriously lame. These ease of making changes to unit balancing or buildings or provinces or campaign start dates or whatever was one of the really nice things about previous TWs.

Sage

 

[Serben]

Well i studied vb, c++ and java in high school, as well as the hexadecimal system (which we used for digital electronics). Only problem is i've forgotten most of it. But i'm sure i still remember enough to do some easy edits like changing the time it takes to build buildings and units, which is really the only thing i care about. That and making units bigger. I've got all weekend, so i'm sure i can throw something together. I was just hoping someone had the tools so i wouldn't have to do it the hard way. But if someone comes up with something, please post all your progress in this thread. That way we can collect all our combined findings in this thread, which should make things easier.

 

[aronnov]

Well i studied vb, c++ and java in high school, as well as the hexadecimal system (which we used for digital electronics). Only problem is i've forgotten most of it. But i'm sure i still remember enough to do some easy edits like changing the time it takes to build buildings and units, which is really the only thing i care about. That and making units bigger. I've got all weekend, so i'm sure i can throw something together. I was just hoping someone had the tools so i wouldn't have to do it the hard way. But if someone comes up with something, please post all your progress in this thread. That way we can collect all our combined findings in this thread, which should make things easier.

that's what i'm basically using. I'm just going to change the values and see what happens. do we have to repack the files for the game to recognize the changes?

 

[LtChambers]

Did you guys read the color coded example? I showed you which bytes to change for changing various unit stats. In this case, for the long riflemen, but all the units are in that unit_stats_land db file.

 

[Serben]

Well, i'm stuck. I opened up the db file in the "building_levels_tables" folder with a hex editor, and found all the buildings in there, along with what i thought were the times to build them. I scrolled down until i found the weavers cottage upgrade, and the first four numbers right after its string were "00 03" in hex. And since it takes 3 turns to build the weavers cottage in the game i tried changing it to 01 and saved it. Then i renamed my main.pack file to main so that the game wouldn't recognize it, and fired up the game. But in the game it still takes 3 turns to build the weaver's cottage. So either that number i found isn't the time to build the upgrade, or the game somehow overrides it.

 

[compound]

. mispost...

 

[compound]

building levels looks like it describes what chain the building belongs to and what number in that chain it is.

try the building chains table, the integer after it there looks more like it


edit: after looking through that, it looks like it has, how many turns to research too

 

[LtChambers]

Serben, I looked at building_levels and you're close. I'll color code an example:

Magenta: string lengths
Green: UTF-16 strings
Red: could be important, not certain
Cyan: long int for build time for settlement fortifications (4)
Blue: long int for cost of settlement fortifications (5000)

000049bfh: [COLOR=Magenta]19 00[/COLOR] [COLOR=DarkGreen]73 00 65 00 74 00 74 00 6C 00 65 00 6D 00[/COLOR] ; [COLOR=Magenta]..[/COLOR][COLOR=DarkGreen]s.e.t.t.l.e.m.[/COLOR]
000049cfh: [COLOR=DarkGreen]65 00 6E 00 74 00 5F 00 66 00 6F 00 72 00 74 00[/COLOR] ; [COLOR=DarkGreen]e.n.t._.f.o.r.t.[/COLOR]
000049dfh: [COLOR=DarkGreen]69 00 66 00 69 00 63 00 61 00 74 00 69 00 6F 00[/COLOR] ; [COLOR=DarkGreen]i.f.i.c.a.t.i.o.[/COLOR]
000049efh: [COLOR=DarkGreen]6E 00 73 00[/COLOR] [COLOR=Magenta]19 00[/COLOR][COLOR=DarkGreen] 73 00 65 00 74 00 74 00 6C 00[/COLOR] ; [COLOR=DarkGreen]n.s.[/COLOR][COLOR=Magenta]..[/COLOR][COLOR=DarkGreen]s.e.t.t.l.[/COLOR]
000049ffh: [COLOR=DarkGreen]65 00 6D 00 65 00 6E 00 74 00 5F 00 66 00 6F 00[/COLOR] ; [COLOR=DarkGreen]e.m.e.n.t._.f.o.[/COLOR]
00004a0fh: [COLOR=DarkGreen]72 00 74 00 69 00 66 00 69 00 63 00 61 00 74 00[/COLOR] ; [COLOR=DarkGreen]r.t.i.f.i.c.a.t.[/COLOR]
00004a1fh: [COLOR=DarkGreen]69 00 6F 00 6E 00 73 00[/COLOR] [COLOR=Red]00 00 00 00 00 00[/COLOR] [COLOR=Cyan]04 00[/COLOR] ; [COLOR=DarkGreen]i.o.n.s.[/COLOR][COLOR=Red]......[/COLOR][COLOR=Cyan]..[/COLOR]
00004a2fh: [COLOR=Cyan]00 00[/COLOR] [COLOR=Blue]88 13 00 00[/COLOR] [COLOR=Red]00 00 00 00 00 00 00 00 00 00[/COLOR] ; [COLOR=Cyan]..[/COLOR][COLOR=Blue]ˆ...[/COLOR][COLOR=Red]..........[/COLOR]

Other than editing these values, you need to make sure your db files are actually being used. You'll have to extract the entirety of main.pack and then move main.pack out of the data directory.

 

[Agostinos]

I've unpackes main.pack and renamed it to main.pack_unpack and tried to edit some unitstats like in post #1. But nothing happens Can someone help me?

 

[LtChambers]

Don't rename it. Move it to another directory. It's easier and more likely to work (the game may actually find main.pack because it's a prefix of the renamed string!). Also, it appears patch.pack overwrites the db directory as well. You'll need to extract patch.pack over main.pack and then move patch.pack out of the data directory as well.

 

[Agostinos]

Thank you for the fast reply. Moved main.pack to the desktop and still no changes. I suspect patch.pack does block the changes but when i extract the files in there and move pacth.pack on the desktop the game will crash

 

[Serben]

I can't move patch.pack out of the data directory, even if i've unpacked it first. If i do, the game crashes on startup, right when the logos and intro videos start playing. Which means it's impossible to edit anything since i'm assuming the files inside patch.pack override any changes i make on the unpacked db files.

 

[LtChambers]

Yes, that's a serious issue, but I'm fairly sure why it's happening. The patch file overrides files from many of the other packs, but when you extract the patch file and move it, the extracted files get overridden by those from the packs which hold the unpatched files. The only solution is to extract every pack that the patch file overrides and then extract the patch onto that and then move all the packs out. That's a major commitment of disk space though. This solution is in lieu of a "--file-io-first" flag which would be too easy. :/