A Replay Harness That Proves Determinism in CI
CandyRush is built on a deterministic simulation: the same seed and the same inputs must produce the same game, tick for tick. That property is what later made server-verified weekly scores possible. In early May 2026 it was still mostly a belief, backed by many small unit tests of individual classes.
On May 8 I closed that gap with a replay harness that runs in the normal test suites. What it does, in one table:
| Layer | Scenario | Length | Digest | Runtime |
|---|---|---|---|---|
| EditMode foundation | Seeded RNG, fixed-point math, loop scheduling, synthetic input stream | 3,600 ticks (60 s) | 61 hashes, one per second | under a second |
| PlayMode entity state | Real game scene, full dependency injection, stage 1, one ability | 1,800 ticks (30 s) | 30 hashes, one per second | one scene boot, then the sim at 10x speed |
Each layer hashes the simulation state once per simulated second and compares the stream against a fixture committed to the repository. Any difference at any second fails the test and prints the first ticks that diverged.
The harness landed as one commit: 21 files, 1,054 lines added. It also found a real determinism leak before it was merged.
Where it came from
The harness was a follow-up. The day before, PR #254 audited every projectile
class for determinism violations. It deleted five dead projectile classes and
ported the nine live ones to tick-based timing inside Simulate: no more
easing-library callbacks deciding when a sword's hitbox closes, no more float
accumulators for elapsed time. That PR was 21 files, +879/-740.
Its acceptance criteria included an end-to-end replay test, which did not fit inside it. So I wrote a follow-up issue with explicit acceptance criteria: a recorded input stream and seed, playback with no live input, a state digest at fixed tick intervals suitable for byte comparison, a reference run on Windows verified on at least one other OS, and a CI gate.
An agent implemented it against that issue. I reviewed the design and the diff, and the issue closed the same day. The digest, the Windows reference and the CI gate were delivered. Two things were delivered differently or not at all, and I come back to both below: the input is defined in code rather than recorded to a file, and no second OS has been verified.
How a digest is built
A digest is a list of (tick, hash) pairs. The PlayMode layer computes each
hash from the live scene:
ulong hash = 17;
hash = hash * 31 + tick;
// Player position: FixedVector2 raw values are bit-identical across platforms.
var pos = player.FixedPosition;
hash = hash * 31 + (ulong)pos.X.RawValue;
hash = hash * 31 + (ulong)pos.Y.RawValue;
// Then: player projectiles, enemy projectiles, enemies (id, position, HP),
// each list sorted into a stable order and prefixed with its count.
Three choices matter here.
Hash raw fixed-point values, not floats. The simulation stores positions as
Fixed64, a 64-bit fixed-point type. Its raw integer is the actual state. Two
machines either agree on it bit for bit or they don't; there is no epsilon to
tune. The one exception is enemy HP, still a float at the time, which is
rounded to thousandths before hashing.
Hash counts as well as contents. An extra projectile with no other effect still changes the hash.
Sort before hashing. Unity's FindObjectsByType returns objects in an
undefined order. Every list is sorted into a deterministic order first, and
that rule is what exposed the bug.
The scenario is fixed in code rather than recorded from a human session: seed
12345, stage 1, the ThunderRing ability, and an idle input sequence for 1,800
ticks. The EditMode layer generates a seeded synthetic input stream instead
(30% density over 3,600 ticks), so player input is exercised at the lowest
layer. The committed fixture is the recording: a small text file with a header
describing the scenario, then one hash per second.
# Candy Rush replay determinism baseline
schema=1
scenario=playmode_entity_state_v1
os=windows
snapshot_count=30
seed=12345
ticks=1800
snapshot_interval=60
ability=ThunderRing
---
60:C6202871EAEDBD4B
120:C8229854DEC63708
...
The leak it found while it was being built
The first version sorted projectiles by EntityId, the same rule used
everywhere else in the simulation. For projectiles, that rule turned out to be
empty.
The cause: both projectile classes exposed SetEntityId, but nothing ever
called it. Every projectile in the game had EntityId = 0. Sorting a list
where every key is zero is a no-op, so the hash quietly fell back to
FindObjectsByType's undefined order.
This was a latent problem, not just a test problem. Any simulation code that trusted "sorted by EntityId" to mean "deterministic order" for projectiles was wrong. The harness now sorts projectiles by their raw fixed-point position, which is real deterministic state. The missing IDs were tracked as their own issue, closed by the same commit.
Unit tests of individual classes did not catch this. They test one object at a time, and with one object, order never matters.
Three tests per layer
Each layer runs the same three checks:
- Same OS, run to run. Generate the digest twice in one process and require identical hashes. This catches side channels like wall-clock time, unordered dictionaries and Unity callback order, without any fixture.
- Matches the committed self baseline. Compare a fresh digest with this OS's committed fixture. This is the regression gate: any change that alters the game's behaviour fails here, even if it is still deterministic.
- Matches committed peer baselines. Compare a fresh digest with every other OS's fixture. This is the cross-platform gate.
Two pieces of process sit around them. A missing fixture bootstraps: the test writes it and asks you to commit it. A deliberate behaviour change uses an explicit regenerate test that never runs in a normal pass, so re-blessing a fixture is always a named, reviewable act rather than a side effect. Fixtures also carry a schema version; a peer with a different schema is skipped rather than failed, so a format change does not need simultaneous commits from every OS.
In CI the harness runs in the regular EditMode and PlayMode steps on the self-hosted Windows runner, and a failed test fails the build.
What it proves today, stated plainly
The cross-platform design is in place. The cross-platform evidence is not.
I checked the history: every replay fixture ever committed has an os=windows
tag. As of November 2026 there are five of them (the original two, plus props,
boomerang and stage-segment scenarios added later), and no macOS, Linux or
mobile fixture has ever been committed. So the peer test has never compared
two operating systems. That was the unmet acceptance criterion.
What the harness does prove, on every run:
- the simulation is deterministic run to run on Windows, at the entity level;
- the game's behaviour has not changed since the fixture was last blessed.
The second guarantee turned out to be the one I used most. Between September and November, the Windows fixtures were re-blessed in many separate commits, each named for the intentional simulation change that caused it: a seed-mixing fix, a flow-field rewrite, wave variance, an enemy-density performance pass. None of those changes could have slipped a behaviour change in silently.
The cross-platform claims the game does make rest on other evidence: fixed-point math in the simulation and, later, server-side re-simulation of submitted runs. Those are separate stories.
Two things that went wrong later, and the fixes
An inconclusive result is not a pass. The peer test reported
Inconclusive when there was no other OS to compare with. That looked
harmless, but Unity's batch test runner maps Inconclusive to a failing exit
code. On a Windows-only setup, a suite with zero failures still exited
non-zero. In June the peer test was changed to pass with an explanatory
message when no peers exist.
A fixture can be committed stale. In June both PlayMode self-baseline tests went red. Before regenerating anything, the agent fixing it verified that fresh hashes were identical across separate Unity processes and that no simulation code had changed since the fixtures were committed: they had been committed stale during earlier work. Only then were they regenerated through the explicit regenerate test. Regenerating first would have destroyed the evidence of whether the simulation had actually drifted.
What generalizes
Hash the state you actually store. Fixed-point raw values made the comparison exact. Floats need a tolerance (the HP rounding is one), and a tolerance is a place for bugs to hide.
Every list you hash needs a total order. "Sorted by ID" is only an order if the IDs are unique. Count the distinct keys before you trust a sort.
Make re-baselining an explicit act. A golden-file test that rewrites its own file on every run proves nothing. A separate, opt-in regenerate step keeps every baseline change visible in review.
Name what is not yet proven. The harness is cross-platform by design and single-platform in evidence. Saying so is what keeps a "determinism verified" badge honest.