Modding game mechanics in DB files

[LtChambers]

Many of the core game mechanics are trapped in a new binary format that CA has started using, some kind of database. Based on some work by alpaca, I've put together a short lesson on how to modify unit stats with a hex editor. This will have to suffice until ETW releases their modding tools or we come up with a way to read and edit the db files natively.

This lesson is not for the faint of heart, hex editing is a tricky business. A very good (unfortunately not free) tool is Hex Workshop which lets you select a range of bytes and interpret those as different data types. You pretty much select a range of numbers in powers of 2 (almost always 1, 2, or 4 bytes) and see what the value is. The strings in these db files are stored as BSTR UTF16, which means that the two bytes before every string are an unsigned short integer indicating the length of the string that follows, and each character in the string is two bytes (in English versions and most other Romantic languages, the second byte of each character will almost always be 00).

The lesson is in the color coding of the long riflemen statistics below. The following excerpt of binary is from the file at:
./data/db/unit_stats_land_tables/unit_stats_land

Bytes highlighted magenta are string lengths.
Bytes highlighted green are part of a UTF-16 string.
Red bytes are ones that are potentially interesting, but I haven't figured out what they are. They could be anything and any type, but usually they will be a single delimiter byte, a 2 byte short integer, a 4 byte float, or a 4 byte long integer.
The 4 cyan bytes are a long int that sets the normal unit size of long rifle men (80).
The 4 blue bytes are a long int that sets the base accuracy of the unit (60).
The 4 bright green bytes are a long int that sets the base reloading skill of the unit (15).
The 4 orange bytes are a long int that sets the amount of ammunition the unit has (15).
The 4 teal bytes are a long int that sets the melee attack of the unit (10).
The 4 brown bytes are a long int that sets the charge bonus of the unit (4).
The 4 gray bytes are a long int that sets the defense of the unit (8).
The 4 purple bytes are a long int that sets the morale of the unit (8).

000076a9h: 17 00 61 00 6D 00 65 00 72 00 69 00 63 00 61 00 ; ..a.m.e.r.i.c.a.
000076b9h: 73 00 5F 00 6C 00 6F 00 6E 00 67 00 5F 00 72 00 ; s._.l.o.n.g._.r.
000076c9h: 69 00 66 00 6C 00 65 00 5F 00 6D 00 65 00 6E 00 ; i.f.l.e._.m.e.n.
000076d9h: 50 00 00 00 00 00 00 00 00 00 00 00 0D 00 65 00 ; P.............e.
000076e9h: 75 00 72 00 6F 00 5F 00 6F 00 66 00 66 00 69 00 ; u.r.o._.o.f.f.i.
000076f9h: 63 00 65 00 72 00 31 00 00 00 12 00 65 00 75 00 ; c.e.r.1.....e.u.
00007709h: 72 00 6F 00 5F 00 6C 00 6F 00 6E 00 67 00 5F 00 ; r.o._.l.o.n.g._.
00007719h: 72 00 69 00 66 00 6C 00 65 00 6D 00 61 00 6E 00 ; r.i.f.l.e.m.a.n.
00007729h: 13 00 69 00 6E 00 66 00 61 00 6E 00 74 00 72 00 ; ..i.n.f.a.n.t.r.
00007739h: 79 00 5F 00 65 00 75 00 72 00 6F 00 5F 00 6C 00 ; y._.e.u.r.o._.l.
00007749h: 69 00 67 00 68 00 74 00 10 00 6D 00 61 00 6E 00 ; i.g.h.t...m.a.n.
00007759h: 5F 00 6D 00 75 00 73 00 6B 00 65 00 74 00 5F 00 ; _.m.u.s.k.e.t._.
00007769h: 73 00 61 00 62 00 72 00 65 00 12 00 78 00 5F 00 ; s.a.b.r.e...x._.
00007779h: 65 00 75 00 72 00 6F 00 5F 00 72 00 69 00 66 00 ; e.u.r.o._.r.i.f.
00007789h: 6C 00 65 00 5F 00 73 00 77 00 6F 00 72 00 64 00 ; l.e._.s.w.o.r.d.
00007799h: 03 00 00 00 07 00 6C 00 65 00 61 00 74 00 68 00 ; ......l.e.a.t.h.
000077a9h: 65 00 72 00 01 00 00 00 00 00 00 01 00 30 00 01 ; e.r..........0..
000077b9h: 00 30 00 00 00 00 00 00 00 01 05 00 72 00 69 00 ; .0..........r.i.
000077c9h: 66 00 6C 00 65 00 3C 00 00 00 0F 00 00 00 01 00 ; f.l.e.<........
000077d9h: 30 00 09 00 66 00 6C 00 69 00 6E 00 74 00 6C 00 ; 0...f.l.i.n.t.l.
000077e9h: 6F 00 63 00 6B 00 01 0C 00 6D 00 75 00 73 00 6B ; o.c.k....m.u.s.k
000077f9h: 00 65 00 74 00 5F 00 72 00 69 00 66 00 6C 00 65 ; .e.t._.r.i.f.l.e
00007809h: 00 0F 00 00 00 01 0A 00 66 00 6F 00 6F 00 74 00 ; ........f.o.o.t.
00007819h: 5F 00 72 00 69 00 66 00 6C 00 65 00 05 00 73 00 ; _.r.i.f.l.e...s.
00007829h: 77 00 6F 00 72 00 64 00 0A 00 00 00 04 00 00 00 ; w.o.r.d.........
00007839h: 08 00 00 00 00 00 00 00 0A 00 66 00 6F 00 6F 00 ; ..........f.o.o.
00007849h: 74 00 5F 00 73 00 77 00 6F 00 72 00 64 00 01 00 ; t._.s.w.o.r.d...
00007859h: 30 00 18 00 64 00 72 00 69 00 6C 00 6C 00 5F 00 ; 0...d.r.i.l.l._.
00007869h: 73 00 65 00 74 00 5F 00 69 00 6E 00 66 00 61 00 ; s.e.t._.i.n.f.a.
00007879h: 6E 00 74 00 72 00 79 00 5F 00 6C 00 69 00 67 00 ; n.t.r.y._.l.i.g.
00007889h: 68 00 74 00 05 00 65 00 6C 00 69 00 74 00 65 00 ; h.t...e.l.i.t.e.
00007899h: 08 00 00 00 03 00 00 00 CD CC 8C 3F 00 00 00 40 ; ........ÍÌŒ?...@
000078a9h: CD CC 0C 40 00 00 80 40 00 00 00 00 00 00 00 00 ; ÍÌ.@..€@........
000078b9h: 00 00 20 41 00 00 00 00 00 01 01 00 01 00 00 01 ; .. A............
000078c9h: 01 01 01 01 00 00 00 00 48 42 00 00 96 42 00 00 ; ........HB..–B..
000078d9h: C8 42 00 00 00 00 00 00 00 01 01 01 00 00 00 00 ; ÈB..............

This concludes today's Hex Editing 101 class. I may put together some more lessons on editing other db files.

[IndoPruss]

BinaryReader in C# can read those bytes you stated, I'll try now.

[Serben]

YEEEEESSSSSS! Finally! You guys own! Keep up the good work and please hurry up and figure out how all this stuff works so we can start editing stuff.


+rep for you.

[Iaius Statius Laurentius]

Does the unit stats file also contain information for unit caps on special units, like the Guards units or the unique ships?

The USS Constitution had three sister ships, and it's annoying to be limited to just building one.

[stupiddk]

That code looks like the code found in mount and blade. If they release tools like those found in Mount and Blade, there can be crazy mods that change more game mechanics than we could have hoped for. Lets wait for the first real patch next week.

[Serben]

So how's it going? Have you, Ltchambers, or alpaca figured out how to make heads or tails of this?

[Quixote07]

maybe most moddable TW isn't that far off, when they release the modding tools they said they would.

[Lord_Phan]

Can you please check file Startpos.esf in Campaign/Main It's possible that may be where the factions are located. But it's in some type of binary code.

[aronnov]

can anyone host a db file? I have the tools in front of me right now where i feel like i could make some progress. just dont have the game within 40 miles of me atm

[Serben]

You have the tools? Then post them in an attachment, as soon as possible so we can all start editing! Here are a few of the db files:

[Serben]

can anyone host a db file? I have the tools in front of me right now where i feel like i could make some progress. just dont have the game within 40 miles of me atm

And where the hell did you get the tools anyway?

[Lord_Phan]

Maybe he's a computer programmer?

[Serben]

Well either way, i gave you some db files aronnov, so please upload the tools if you have them.

[aronnov]

anyone have luck with BinaryReader in c sharp? I can't get it to read.

oh and thanks for the files serb

[Hexcaliber]

I think he means a simple hex editor to examine the files contents, not something more elaborate like a custom editor. ***EDIT**Ack I must of hit reply just as he finished entering his post saying what he was trying to use, lol, nm

[Serben]

So what's going on? Do you have the tools or not? If you do, can you please post them.

[aronnov]

not any that are working. i'm up there with the TC. the only way to edit these until a tool directly from CA is to edit the hex values with trial and error.

Has anyone had a direct response from CA?

[sage2]

Basically, it's up to hex editing float values. For very simple things, like changing a simple value, this is a PITA. For more complex things, like rebalancing units, it's very tough / close to impossible unless you're autistic, caffeinated and unemployed. Finally, anything more then that, such as appending a string to a unit, it's a no go, since the binary will then not load.

I use this website for converting numbers into floats: http://babbage.cs.qc.cuny.edu/IEEE-754/Decimal.html (although most hex eds should do the conversion for you). Don't forget that some things might be stored as integers (e.g. just normal base 16). Also, don't forget that x86 reads backwards, so, the number 51 (424C0000) would actually be represented in hex as 00 00 4C 42, and -12.1 (C141999A) would be 9A 99 41 C1. Make sure you don't use an 'insert' function with your hex editor; use replace. Insert will almost certainly fail.

I use the XVI32 free hex editor, but they're all pretty much the same.

If all this is utter Greek to you, then you will have a pretty steap learning curve.

Personally, I find it utterly, and immensely retarded that CA has gone to a proprietary data format. I mean... what the hell? All it does it make any serious modding extremely time consuming, or require the community to write it's own tools that will format and write the data files. I would presume it's so they can make more money selling units to the community, or something. But, seriously lame. These ease of making changes to unit balancing or buildings or provinces or campaign start dates or whatever was one of the really nice things about previous TWs.

Sage

[Serben]

Well i studied vb, c++ and java in high school, as well as the hexadecimal system (which we used for digital electronics). Only problem is i've forgotten most of it. But i'm sure i still remember enough to do some easy edits like changing the time it takes to build buildings and units, which is really the only thing i care about. That and making units bigger. I've got all weekend, so i'm sure i can throw something together. I was just hoping someone had the tools so i wouldn't have to do it the hard way. But if someone comes up with something, please post all your progress in this thread. That way we can collect all our combined findings in this thread, which should make things easier.

[aronnov]

Well i studied vb, c++ and java in high school, as well as the hexadecimal system (which we used for digital electronics). Only problem is i've forgotten most of it. But i'm sure i still remember enough to do some easy edits like changing the time it takes to build buildings and units, which is really the only thing i care about. That and making units bigger. I've got all weekend, so i'm sure i can throw something together. I was just hoping someone had the tools so i wouldn't have to do it the hard way. But if someone comes up with something, please post all your progress in this thread. That way we can collect all our combined findings in this thread, which should make things easier.

that's what i'm basically using. I'm just going to change the values and see what happens. do we have to repack the files for the game to recognize the changes?