For about three weeks, Kompete’s live match-scoring feature dropped score updates for roughly 1 in 400 events, but only on mid-tier Android devices, and only during tournaments with more than 30 concurrent matches. It never reproduced on an emulator. It never reproduced with a debugger attached. The fix ended up being four lines of code, but finding those four lines meant giving up on a wrong mental model of how Dart isolates communicate.
The setup: why we used a background isolate at all
Kompete computes live leaderboard deltas as match events stream in over a WebSocket. Each event triggers a recalculation across a match’s roster, which for larger tournaments means sorting and re-ranking a few hundred entries with tie-break logic that isn’t cheap. Doing that on the UI isolate was visibly janky on low-end devices, so it lived in a long-running background isolate, spawned once at app start via `Isolate.spawn`, communicating over a pair of `SendPort`/`ReceivePort` channels.
The architecture was standard: the UI isolate sends a `RecalcRequest` object (immutable, only primitives and lists of primitives), the worker isolate does the sort, and sends back a `RecalcResult`. Nothing about it looked exotic. That’s what made the bug so unsettling — the code doing the actual work was boringly correct.
Chasing a bug that hides from every tool you’d normally use
The first false lead was the WebSocket layer — dropped frames, reconnect races, that kind of thing. We instrumented every inbound event with a monotonic sequence number and confirmed the UI isolate’s `ReceivePort` was receiving every request we sent. The request always went out. Sometimes the result never came back. That narrowed it to the worker isolate itself, or the channel between the two.
Attaching DevTools changed the failure rate from ~1-in-400 to effectively zero. That’s the signature of a heisenbug tied to timing or memory layout, not logic — a debugger attached pins objects and slows GC pauses, which was itself a clue we didn’t weight heavily enough at first. We spent time looking at whether `RecalcRequest` objects were somehow being mutated after being sent (they weren’t; they were frozen with `const`-style immutability) and whether `SendPort.send` was silently failing under backpressure (it isn’t documented to, and it doesn’t).
The breakthrough came from correlating dropped messages against `flutter_driver` timeline output for GC events on the isolate heap. Every single drop lined up with a full (not scavenge) GC pause on the worker isolate, specifically ones that ran long enough to overlap with an in-flight `send`.
What was actually happening
Dart’s isolate messaging model deep-copies most objects across the isolate boundary — but not all of it happens atomically from the sender’s perspective the way you’d assume from “message passing.” We were holding a reference to a mutable staging buffer (a `List` we reused across recalculations to avoid reallocating on every event, a deliberate performance choice under the assumption that isolates don’t share memory so reuse is “safe”). Under normal conditions the send-then-clear-then-refill cycle completed well within a frame. Under GC pressure, the worker isolate’s own compaction pass could stall between when we called `send` and when the message was fully serialized off that buffer, and a subsequent recalculation request — queued while the GC pause was happening — would start refilling the same buffer before the prior send had finished walking it.
This isn’t a documented race in the sense of undefined behavior across isolates sharing memory (they don’t, and Dart’s send/receive contract still held). It’s a race in our code: we assumed the diagnostic behavior we saw in normal testing (send completes “instantly” relative to our reuse cycle) was a guarantee, when it was actually just the common case. GC pressure was the variable that made the uncommon case observable. The bug was ours; Dart’s isolate model behaved exactly as documented. The lesson was about not confusing “haven’t seen it fail” with “can’t fail.”
The fix, and the bigger takeaway
The fix was unglamorous: stop reusing the staging buffer, allocate a fresh `List` per recalculation, and let the isolate’s own GC handle the churn. On the surface that’s a performance regression — more allocation, more short-lived garbage — but measured against real tournament load it was noise, a few hundred microseconds per event, nowhere near what users could perceive. We’d optimized for CPU cycles that didn’t matter and introduced a correctness bug that did.
The broader takeaway, and the reason we treat this as a case study rather than a footnote, is that isolate boundaries in Dart get treated like hard memory-safety walls, and mostly they are. But “mostly” is doing work in that sentence. Any pattern that reuses mutable state across an async boundary — a buffer, a cache, a pooled object — needs to be checked against what happens when the two sides of that boundary don’t execute in lockstep, which GC pauses, scheduler hiccups, and platform channel jitter will all eventually force. Testing on a fast device with a debugger attached is close to the worst environment for finding these bugs, because it suppresses exactly the timing conditions that trigger them.
- Don’t reuse mutable buffers across isolate `send` calls unless you can prove the send has fully drained them first.
- Treat “immutable in practice” as different from “immutable by type” — the compiler won’t catch the former.
- If a bug disappears with a debugger or profiler attached, suspect timing before suspecting logic.
This kind of bug is exactly why we treat shipped, load-bearing code like Kompete’s as the real proving ground rather than a portfolio piece — the failure modes that show up under production concurrency and device diversity don’t show up in a demo. If you’re debugging something similarly resistant to reproduction in your own Flutter codebase, or want a second set of eyes on an architecture before it ships, that’s the kind of work orithLabs does directly, engineer to engineer.