Using the DecodeTool for unknown DB Tables (PFM)

[daniu]

The new version for PFM (1.7.10) contains a tool helping with the analysis of unknown database tables.
This thread gives a short introduction into how it can be used.

Starting up

The DecodeTool can either be started as a standalone application (DecodeTool.exe contained in the PFM zip file) or from within the PFM. It was initially written for standalone so it works a bit better this way.

In standalone mode, you can open a DB file you have previously exported from the PFM.
It guesses the file's DB type from the directory it is contained in (the export does create this directory). It also looks in the directory above ("db" created by the export) for DBFileType files which may contain a definition for this db file and loads that so you have something to start from.

In the integrated mode, the tool is opened with the data from the db file that is currently being tried to open; it does not (yet) know the db type and does not load an initial type.
You can always manually load a DBFileType file via the Definitions->Load menu entry.

Display

Once you have opened a file, you can see its raw data on the left part of the window (encoded in hex).
On the top right, there are two lists; one containing the type for each column, one containing the corresponding data.

Then there are navigation buttons enabling you to move to the first entry ("<<"), single step forward and back (">" and "<", respectively) and one moving to the first entry containing a field causing a software error ("problem").

Below that, there are five preview panels, one for each available data type (string, int, bool, decimal and optional string); these show you the value if the corresponding type is chosen and allow you to actually choose it ("use").
Selecting the "use" button will add the selected type to the selected position in the type list.
There is also a "Delete" button to remove entries, and a "Show" to display the resulting DBFileType entry.

Then there are two informational lables:
one containing the typename of the db currently edited (only correct in standalone mode);
one containing information on how many entries are expected in the currently opened file, how many of those were successfully read and how much of the available data was processed.

Below that, there is an entry field for the header length of the current data file.

The Guessing Game

Here is how you actually analyse a data file:
1. Open the file
Straightforward; select it from the open dialog or skip this if opened in integrated mode.

2. Enter Header length
To do this, you need to look at the hex code on the left.
DB files can possibly contain several header entries; the tool does not detect this automatically yet so you have to add up some items.
Start with 5 (1 byte marker and 4 byte entry count).
a) If you see a byte combination "fd fe fc ff", add 78 (ID string marker and length).
b) If you see a byte combination "fc fd fe ff", add 8. This may appear after the other combination.
This means you can have either 5, 13, 83 or 91 as a result (none of a/b, only b, only a, or both a and b).
After you click "Set", the bytes for the area you have entered will be marked in red.
Note that when you load another file, you have to enter the header length again.

3. DB file version number
This depends on the header again.
If you have a 5 byte header, the version is 0.
If you have an "fd fe fc ff" but no "fc fd fe ff" entry, version is 1.
If you have an "fc fd fe ff" entry, the version is contained in the first byte behind the entry (where 0a means 10, 0b means 11 and so on).

4. Start guessing types
Now you can start looking at the values in the preview fields. The preview will show the values for the next entry after all types you entered have been applied.
You can also select one of the items in the value list to show the previews for that field again (in case you think the type there might be wrong).
Choose "use" for the value that makes most sense to you, "Delete" to delete either the selected entry or the last in the list.

Often enough, only one or two of the values will make sense; sometimes you have to choose between two possible ones.
The type you select will be inserted at the currently selected position in the type list, or at the end when none is selected.

What is really easy to get wrong is the choice between boolean and optstring values.
Optstrings consist of a boolean followed by a string, but the string part is only used if the boolean is set to "true"; therefore, a "0" can either mean a boolean value of "False" or an optional string that does not contain a value.
To switch between the two, I added the menu entry in More ("Bool<->Optstring"). Select the boolean or optstring in the type list and select it to toggle between the two.

What you need to have in the end is an entry in the information section which has read all entries and all data:
"Trying to read X fields: X read, B/B bytes". This means you have created a useful definition (except maybe for ambiguous types).
You can now click the "Show" button which will open a window showing the DBFile entry for your definition.
In standalone mode, you will need to add the real type name to the front of the third (i.e., non-empty) line - make sure you keep the tab between type and first entry.
Paste it into the DBFileTypes_<version>.txt file in your DBFileTypes_user directory and restart PFM to check if everything went fine.

There are still lots of improvements that could be done to the tool, but since it works reasonably well, it's not really worth the effort to put too much time into them.