500 Enemies: From 27 to 42 FPS in a Deterministic Simulation
On October 26, five days before CandyRush launched on Steam, a playtest of arc 3 at depth 5 in the Unity editor was close to unplayable: 15 to 30 FPS. By the end of the day the same stage measured 42.4 FPS at peak, and the replay digests were identical whether the new code ran on 1 worker thread or 23.
All figures below are from the arc 3 "Horde" segment at depth 5, at its 500-enemy cap:
| Before | After | |
|---|---|---|
| FPS at peak (median) | 27.4 | 42.4 |
| Frame time | 36.4 ms | 23.5 ms |
| Simulation ticks per frame | 2.14 | 1.42 |
| Enemy neighbour work per tick | ~3.0 ms | 0.86 ms |
| Hazard props per tick | 1.80 ms | 0.57 ms |
The hazard-prop row comes from a follow-up capture of the final code, which peaked at 41.4 FPS, within run-to-run noise of 42.4.
These are Unity editor captures, on a Ryzen 9 7900X with an RTX 4090 and a 1599x894 Game view, with Burst compilation turned on in the editor for the "after" row. In my experience, deployed player builds typically run 2-3x higher FPS than the editor; that is my experience, not something measured for this post. Run-to-run noise on the same case was about plus or minus 1.5 ms per frame (41.4, 42.4 and 43.1 FPS across three runs).
The work shipped as two pull requests merged on October 26: UTools #179, then CandyRush #1079. I directed an AI coding agent through it: I set the constraints and the questions, and reviewed each measurement before the next change. This post is about how the decisions were made.
The constraint
CandyRush is a deterministic, fixed-timestep simulation. Every weekly
leaderboard score is verified by replaying the player's recorded inputs on a server, and
that only works if the same inputs produce the same state, bit for bit, on
Windows, macOS, Linux and the web. The simulation uses fixed-point math
(Fixed64), not floats, for exactly that reason.
That rules out the usual quick wins. A float fast path had already been tried: enemy separation used float math until August 9, when it was switched off because web and desktop replays diverged. I raised relaxing determinism outside the weekly leaderboard run as an option. The written analysis recommended against a dual-mode simulation, and what shipped keeps one deterministic simulation and relaxes only presentation.
Step one: make the problem reproducible in 90 seconds
A 15-minute playthrough is not a measurement. The first deliverable was a
PlayMode capture test, SegmentDensityCaptureTests, that builds a real run at
a chosen depth from the shipped segment data and records one row per second:
frame time, the simulation's share of it, ticks per frame, live enemy count and
crowd overlap. It also reads Unity's own profiler markers for an engine
breakdown, and supports A/B variants (physics off, animators off, audio off,
and others).
The test reproduced the report: Horde at depth 5 fills to the 500-enemy cap in
about 35 seconds and then holds at 25 to 29 FPS. A run takes about 90 seconds
instead of 15 minutes. It is marked [Explicit] so it never runs in CI.
The baseline also said where to look:
| At peak, depth 5 | Enemies | FPS | Frame ms |
|---|---|---|---|
| Horde | 500 | 27.4 | 36.4 |
| Combat | 309 | 60.9 | 16.4 |
| Horde, depth 0 | 254 | 74.9 | 13.3 |
At 500 enemies the simulation was about 54% of the frame: 2.14 ticks per frame at 9.1 ms each. That is the start of a catch-up spiral. A slow frame means more fixed ticks next frame, which makes that frame slower. Cutting tick cost cuts the number of ticks too.
The capture also turned up noise worth removing. A temporary debug probe from August was still writing about 6,400 log lines with stack traces per arc 3 run, for enemy deaths, wand shots and gate entries. It was deleted.
Step two: Burst kernels that stay deterministic
Enemy separation (soft steering away from neighbours) and overlap pushout (hard de-penetration) were the biggest per-tick costs. The change moved both into Burst-compiled kernels:
EnemyNeighbourGridis a snapshot of every enemy's position, radius and id, taken at the start of the tick and bucketed with a counting sort.SeparationForceJobcomputes the whole crowd's steering force in one parallel job.- Overlap pushout runs as a Burst direct call.
Two details kept this deterministic:
Jacobi, not Gauss-Seidel. Every enemy reads its neighbours from the
snapshot, never from positions that other enemies have already updated this
tick. The result no longer depends on processing order, so it is the same on 1
worker thread or 23. A first cut measured the moved enemy against snapshot
neighbours, and two enemies stacked exactly on top of each other then never
separated. The existing EnemySeparationIntegrationTests caught it, and a new
kernel test pins the fix.
Burst cannot rely on double math in a static constructor. Fixed64 built
its angle constants with FromDouble(Math.PI ...). UTools #179 replaced them
with the raw 64-bit values that expression already produced, and a test asserts
each literal equals its old source. No digest moved from that change.
Measured per tick at 500 enemies:
| Cost | Before (ms) | After (ms) |
|---|---|---|
| Separation | 1.26 | 0.15 |
| Overlap | 1.42 | 0.48 |
| Grid rebuild | 0.34 | 0.23 |
One finding was not code at all: with Burst compilation off in the editor, as it was on the dev machine, the same kernels measured 30.9 FPS instead of 42.4. A capture has to record that setting.
Step three: a wrong guess, measured and reverted
The next item on the profile was HazardPropBehavior: 1.80 ms per tick for
area props like hazard zones. The code looked for each enemy in a member list
with a linear search, so the obvious guess was the lookup. The agent replaced
it with a HashSet.
It made no difference. So instead of guessing again, it added temporary timers over 135,045 area-prop ticks:
| Per prop, per tick | Time |
|---|---|
| Whole membership scan | 19.0 µs |
| Loop over all ~500 enemies | 17.2 µs |
| Shape rebuild | 1.2 µs |
| Apply effect | 0.45 µs |
Zones held on average 0.2 enemies. The cost was not finding members; it was
visiting all 500 enemies, every prop, every tick, to find almost none. The
HashSet change was reverted along with the timers.
The real fix reused the snapshot grid: area props now ask the grid for the
enemies inside their bounds. HazardPropBehavior went from 1.80 to 0.57 ms per
tick, and the full tick at peak from 6.65 to 5.73 ms. Peak FPS on that run was
41.4, within noise of the 42.4 before it.
Did the crowd still behave?
Moving to a tick-start snapshot changes simulation results. So the crowd was checked by measurement, not by eye:
| Horde depth 5, at peak | Overlapping pairs per enemy | Worst overlap depth |
|---|---|---|
| New snapshot path | 0.465 | 0.36 |
| Old live-read path | 0.568 | 0.81 |
The crowd is not packed tighter; it has fewer overlapping pairs and a shallower worst case. The comparison row uses the float-math option, because the old fixed-point live-read path no longer exists to run.
Determinism: re-blessed, then proven
Because results changed, the replay digests changed too. This was not a "nothing moved" optimization, and the pull request says so. The process:
- Confirm the machine reproduces its own digests first (the same-OS run-to-run test was green).
- Regenerate the two PlayMode replay baselines that moved, and re-stamp both canary replays. EditMode, props and boomerang baselines did not move.
- Re-run everything: PlayMode digests 18 of 18, EditMode digests 9 of 9, both canaries, and the multi-segment and six-boss matrix tests.
- Prove the new code is order-independent: the digests were identical with 1 job worker and with 23, and with Burst on and off.
Windows is the only OS with replay fixtures, so the baselines were regenerated there. The simulation version stayed at 1, per the project's rule for pre-release simulation changes.
Presentation: no physics engine for enemies
Two presentation-side changes came from the A/B variants:
- Enemies left Box2D. Their
Rigidbody2Dis no longer simulated. That needed an audit of every trigger and collision callback first, to confirm each one either returned early because a deterministic scan already handled it, or only cared about the player. A/B: Box2D cost about 2.6 ms per frame at the cap, with identical kills on and off. - Off-screen trash enemies stop animating. Boss rigs keep their authored mode.
An A/B with enemy physics and animators both fully off measured 61.1 FPS. What shipped is a little different (animators are culled off-screen, not removed), and I have not re-measured that exact combination, so 42.4 is the number I stand behind.
Review
Two independent adversarial reviews, one for determinism and one for native/engine code, found seven issues before merge. Among them: enemies teleported after the snapshot used a stale slot; area props past the edge of the grid could miss enemies (they now fall back to the old scan); and an exception during the snapshot rebuild would have frozen the simulation. All seven were fixed in the same pull request.
What is left
- Flow-field sampling and obstacle pushout, about 0.7 ms at 500 enemies, are still per-enemy managed code.
- Run-to-run noise is about plus or minus 1.5 ms. The capture test makes re-measuring on a quiet machine a one-line command.
What generalizes
Reproduce before you optimize. A 90-second test that recreates the complaint is worth more than any single fix, because every later claim is checked against it.
When a guess does not move the number, measure instead of guessing again.
The HashSet was a reasonable idea about the wrong cost. Timers showed the
real one in a single run.
Determinism is a constraint to design within, not a reason to stop. Jacobi reads from a snapshot are both faster to parallelize and independent of order. The proof is the digest match at 1 and 23 workers.