Unlocking more memory in a PS3 game

A little known fact about the PlayStation 3 is that throughout its lifespan, Sony gradually freed up more system memory for developers through firmware optimizations. At launch, the console’s operating system occupied 64MB of the system’s 512MB total memory, which used a 256MB RAM/VRAM split memory configuration (32MB of system overhead in each). By March 2008 with the release of firmware 2.20, the VRAM usage had dropped to just 7MB - an impressive 80% reduction in memory usage!

You might be asking yourself why this matters at all. You would assume that the majority of games already take advantage of this extra memory considering that firmware 2.20 released only 17 months after the launch of the PlayStation 3. That’s a reasonable conclusion to come to; however, I know of at least one game that doesn’t. That game is Metal Gear Solid 4, and by extension Metal Gear Online 2.

So what does this mean? Well, it turns out Metal Gear Online 2 is still actively played by a die-hard community all of these years later, and due to reverse engineering efforts, the game is somewhat moddable. This means new content such as cosmetics or maps can be added to the game, but as you might’ve guessed the game was already pushing it dangerously close to the memory limits. At the worst case there may have only been a couple MBs of VRAM available, which varied depending on the circumstances such as what map is being played and by how many players.

At some point while adding new content to the game, players started experiencing crashes. I determined that the crashes were coming from areas of the code responsible for allocating textures, so I decided to dig a bit deeper. While examining the source code of RPCS3, the most widely-used PS3 emulator, I came across the following function.

/*
 * Get usable local memory size for a specific game SDK version
 * Example: For 0x00446000 (FW 4.46) we get a localSize of 0x0F900000 (249MB)
 */
u32 gcmGetLocalMemorySize(u32 sdk_version)
{
    if (sdk_version >= 0x00220000)
    {
        return 0x0F900000; // 249MB
    }
    if (sdk_version >= 0x00200000)
    {
        return 0x0F200000; // 242MB
    }
    if (sdk_version >= 0x00190000)
    {
        return 0x0EA00000; // 234MB
    }
    if (sdk_version >= 0x00180000)
    {
        return 0x0E800000; // 232MB
    }
    return 0x0E000000; // 224MB
}

This revealed that different SDK versions allowed games to utilize more or less VRAM, which I had not known previously. I began to wonder if MGS4/MGO2 were actually using the full amount, and if not then could we expand it to allow adding more content to the game without crashing. MGS4 released a few months after firmware 2.20 and the latest update for MGO2 was in 2012, so it seemed likely that the developers would have made use of the latest SDK but I decided to look anyway.

Opening up the binary for MGO2 in Ghidra, you will find the following string:

"GCC: (GNU) 4.1.1 (SDK240, $Rev: 2499 $)"

A bit discouraging as SDK240 means the game was compiled targeting the 2.40 firmware. While this means that the game can use maximum amount of memory available, it would seemingly imply that the developers were already likely doing so. Despite that, I didn’t actually have any verification that was the case so naturally I continued to investigate. Cross referencing the string actually brings us to an interesting place in the binary.

*piVar1 = (int)("GCC: (GNU) 4.1.1 (SDK240, $Rev: 2499 $)" + iVar14 + 2);
*(undefined4 *)("GCC: (GNU) 4.1.1 (SDK240, $Rev: 2499 $)" + iVar14 + 2) = 0;
iVar14 = *piVar1;
puVar10 = (undefined4 *)(iVar14 + 0xa000);
piVar1[4] = (int)puVar10;
*(undefined4 *)(iVar14 + 0xa008) = 0x80040000;
*puVar10 = 0;
*(undefined4 *)(iVar14 + 0xa004) = 0;
cellGcmGetConfiguration
          (cVar7 + '\f',uVar11,cVar13,uVar16,in_r7,in_r8,(char)puVar10,in_r10,
            *(undefined *)((int)puVar5 + 0xb),*(undefined *)((int)puVar5 + 0xf),puVar5[5]);

This is barely legible decompiled code, but the important thing is we can see that we’re in an area responsible for configuring the PS3’s graphics system (cellGcm is the graphics library). In the very same function we can see bunch of relatively large hardcoded constants littered throughout, all being used in the same way.

// simplified the code for easier reading
piVar1[0x19] = 0x1900000;
piVar1[0x4059] = 0x600000;
piVar1[0x404a] = 0x100000;
piVar1[0x28] = 0xaf00000;
// etc...

// we can later see that all of the values are 
// eventually being added up and stored somewhere in memory
*(int *)(iVar22 + 4) = piVar1[0x19] + piVar1[0x4059] + piVar1[0x404a] + piVar1[0x28] // etc..

When looking at that pointer in memory, we get a value suspiciously close to the ones we saw in the source code of RPCS3.

0xE200000 (236,978,176)

While monitoring this memory address during gameplay, you can see a value close by increasing or decreasing between loading stages. Setting a breakpoint on the code that loads textures, we can see it increasing after each break meaning this appears to be some sort of structure that keeps track of memory usage. The number we found before is presumably the total amount of allowed VRAM that the game can allocate.

236MB is an interesting number, as it’s higher than the allowed 234MB from firmware 1.90 but lower than the 242MB allowed from 2.00. In any case, the largest constant we saw in the previous code was 0xAF00000 which comes out to ~183MB. We can assume this is probably for textures as they take up the majority of VRAM in games. The only way to know for sure is to patch the binary and try to force another crash that previously happened when loading too many textures.

Surprise surprise, after testing the patched binary, we no longer crash!

I’m still not entirely sure why the game was only utilizing 236MB of memory despite that not actually being one of the hardcoded values in the SDK as well as receiving updates for years after the limit was raised to 249MB, but my patch worked and has been in use by the community for years at this point.

This patch allowed the Metal Gear Online community to add dozens of new items to the game without worrying about crashing, and given the relatively modest texture sizes common during the PS3 era, you can add a surprising amount of content in the extra 13MB that was patched in.