Real Gameplay Saves Revealed a Softlock the Tests Had Missed

Replacing injected test state with saves captured during actual gameplay exposed a Godot softlock that existing tests had hidden. The same approach also confirmed that save migration code under development could load real saves in an older format.

Key Takeaways#

• Capture saves during normal gameplay and use states players can actually reach as test preconditions.
• Load every captured fixture through the production loader and check values through the current game API.
• Remove test-only reset logic that could hide missing recovery behavior in the game.
• Count the remaining state-injection calls and require that count to keep decreasing.

Why State Injection Was Not Enough#

The web game's healing test warped to a checkpoint, set health to 5, and then invoked the healing action. This verified healing from an artificial state. It did not check whether a player could reach that checkpoint with the same inventory, battle outcomes, and progress.

Across 14 spec files, there were 18 state-changing calls, including forced victories and defeats, warps, and inventory injection. Each made tests easier to write, but none proved that normal gameplay could produce the starting state.

The replacement rule was simple: fix seeds, choices, and inputs as needed, but let normal game code produce the resulting state.

Capture, Load, and Verify#

This approach was applied to a turn-based web game, a Godot game, and a reusable web game template. The web projects followed these steps:

1. Have a bot play using actions available to players. 2. Export all localStorage contents at suitable checkpoints. 3. Save each capture to fixtures/saves/<app-version>/<name>.json. 4. Load every fixture through the production loader and use some as starting states for E2E tests.

Each fixture contains the raw save string and a manifest describing the version, capture command, scenario, and expected values. After loading, expected values are retrieved through the current game API. This keeps verification intact even when internal save field names change.

The healing test now starts from a captured checkpoint save. Before healing, it checks that the player is at the checkpoint and has less than maximum health. If loading fails and silently starts a new game, the precondition check fails. The defeat test needs no fixture: the bot skips turns until enemy attacks cause a normal defeat.

Review Expected Values#

The capture script must not automatically turn the values it reads into expected values. That would lock in incorrect states just as readily as correct ones.

New captures leave invariants set to null. The regression test deliberately fails and prints the values returned after loading. A reviewer checks those values against the screen at capture time and the raw save data before recording them as expected values. The test also fails if there are zero fixtures, preventing an empty directory from producing an apparent success.

This enabled a concrete migration check. A v3 save captured from the committed version of the web game was loaded in a separate worktree containing the v4 migration code under development. Both fixture tests passed. The loader accepted the old keys, migrated the save, and preserved the values under test. This confirmed compatibility with those fixtures, not with every older save.

The Bot Encountered a Real Softlock#

The Godot capture bot was supposed to record the autosave after the first milestone. It made 60 attempts on each of three seeds but never reached it. After the first defeat, party members remained at zero health. Subsequent attempts ended almost immediately, and players had no way to heal.

The existing balance tests had missed this because Bot.simulate called restore() before every attempt. The test bot was repairing state on the game's behalf. The capture bot followed the game's behavior and got stuck exactly where a player would.

The fix restored everyone's health when returning from an attempt. Because saves occur between attempts, loading uses the same recovery logic. This also covers older saves containing characters at zero health. Two tests verified recovery after defeat and recovery when loading such saves; both failed when the fix was removed. The bot's internal restore() call was also removed.

Keep Shortcuts from Returning#

On the web side, only the healing and defeat setups were replaced, leaving 15 state-injection calls. A counting test now enforces that baseline. It fails if the count increases, and also if it decreases until the baseline is lowered, preserving each reduction.

Capture quality still depends on the bot. Random inputs for smoke testing can produce saves, but reaching specific checkpoints requires game-specific decisions. Raw saves are stored as single-line JSON strings and are difficult to review, so fixture manifests need to describe the scenario and capture procedure.

Test storage also needs to be separate. During this work, existing Godot tests were found to be creating and deleting slots in the production user:// location. Test saves are now stored under user://verify/tests/.

Next Step#

Replace one E2E test setup that uses state injection with a save captured during normal gameplay, and make the test fail if loading it does not restore the intended starting state.

Source: https://zenn.dev/gzer0/articles/real-play-save-fixtures-for-ai-tests