Aurum's Mods

Youtube Github Discord Docs Blog

Flatbuffer format

Pokémon Ranger uses flatbuffers to store data of any kind, including graphics, Pokémon and object settings, Browser information, map data and much more. In general, flatbuffers are binary buffers containing nested objects and structures that are organized using offsets so that the data can be accessed like any pointer-based data structure*. In other words, the data is directly loaded into memory and does not need further deserialization. As usual with Nintendo DS games, these are stored in little-endian byte order. Flatbuffers consist of one or more labeled data sections. Data is accessed by searching for a section's label in the flatbuffer container.

Curiously, some flatbuffer files are called CSV whereas data is not stored in plain-text. This is likely a leftover from conversion processes.

Structure

These files all share the same structures:

  1. 0x20 bytes header
  2. Data block, consisting of one or more different sections
  3. Pointer fix list
  4. List of labeled section offsets to access section data
  5. List of specific labeled entries inside the data block

The header is 0x20 bytes and consists of these fields:

Offset Type Name Description
0x00 int fileSize Total file size
0x04 int dataSize Size of data block
0x08 int numPointers Size of pointers list
0x0C int numSections Number of sections in data block
0x10 int numLabels Number of labeled entries in data block
0x14 int unk14 Unknown. Used values are 0x00 and 0x4C313030 (L100)
0x18 pad[8] pad18 Padding / unused

Using the information in the header we can calculate all offsets for structure segments. I'll use pseudo-code for this as well as the field names described above:

Name Formula Description
offData = 0x20 Offset to data block
offPointers = offData + dataSize Offset to pointer fix list
offSections = offPointers + numPointers * 0x4 Offset to list of labeled sections
offLabels = offSections + numSections * 0x8 Offset to list of labeled entries
offStrings = offLabels + numLabels * 0x8 Offset to label string pool

Parsing pointer fix list

This is simple to parse. Beginning at offPointers is a list or array of int offsets to various data pointers in the data block. The game iterates over this list and goes to each specified location and converts the value at that position to a pointer. Therefore, it is necessary to fill this list with every offset that needs to be converted to a pointer when loaded by the game. Otherwise, the game will likely crash if it tries to dereference invalid data due to unfixed pointers.

Parsing labeled sections & entries

The section labels are found at offSections. There are numSections entries, each of which is 0x08 bytes. The strings appear to be encoded in ASCII and are null-terminated. The entry structure is as follows:

Offset Type Name Description
0x00 int offData Offset into data block
0x04 int offString Offset into string pool

This structure is the same for labeled entries, however, these are found at offLabels and there are numLabels of these entries.

Data sections

A flatbuffer may consist of more than one data section. It is important to note that each section has to be aligned to four bytes! The different section formats are specified on the corresponding data file documentations.

Accessing data

In general, the loading process is very primitive:

  1. Load flatbuffer into memory
  2. Parse header and calculate offsets
  3. Parse labeled sections / entries and convert their offsets to pointers
  4. Go through pointer fix list to convert offsets to pointers

Then, the game retrieves data sections by passing over a string containing the section name. Again, the way how the game treats the data depends on the context and section type.