Blog / Engineering deep dives

64 agents, one lock: a futex storm in our prefetcher

16 October 2026 · Semurg · 10 min read · Markdown

engineeringperformanceconcurrencyrustelixirmcp

A partner testing Semurg sent us a short report: with many agents writing into one memory, writes did not get faster as agents were added, and with 64 agents some calls failed. A report from outside is a symptom, not a diagnosis, so we ran the same test ourselves.

The report turned out to have two causes. The first was fixed a release earlier. This post is about the second: one lock, held around a lookup that did not need it. It covers how we found it (after first blaming it all on the disk), why it hid so well, and the test that now keeps it from coming back.

What we saw

The test is simple: 64 concurrent clients standing in for agents, each sending remember in a loop over the REST API (/v1/memory/remember, the same door the MCP tool memory_remember calls) with a short passage, two entities and one relation, all into the same scope, on a fresh node limited to 8 cores, with the local embedding model on. Each call is more than an append: the node stores the passage, resolves the two entities against what the scope already holds, links the relation and queues the text for embedding, all on the same CPUs. The rates in this post are tens of writes per second, not thousands; why they are not higher is part of what is still open, at the end.

We ran it on r11.101, the release that already carried the fix for the first cause. There, in a separate sweep with one run per agent count, writes did get faster from one agent up to 16. At 64, some runs collapsed: the same published build swung between runs on fresh containers, and the slow runs had most of the CPU time in the kernel rather than in our code (the figures are in the table under Results). A process that spends most of its time in the kernel while doing little work is usually waiting on something, very busily.

Finding it: first the disk, then the profile

Our first theory had been the disk. The same week, another test had shown the write path reading far more from disk than it needed, and our notes said it explained the partner's report better than any lock: "the callers queue on the disk, not on a mutex". That read amplification was real (a graph image rebuilt from its log by writes that arrived while it was behind), and we fixed it in r11.101. The 64-agent runs above are on that release, and some of them still collapsed. Time to profile.

The release binary is stripped, so a profile of it is a list of addresses. Rebuilding with symbols is the obvious move, with one trap: our container image puts its own CPU-matched build of the native library in place when it starts, so a build copied into the library folder is silently replaced. Ours was: the first profile ran the stripped release library again, named nothing, and happened to catch a fast run with no contention to see. The rule we now follow: build the same variant with the same release flags plus symbols and frame pointers, put it where the image picks variants from, and check the hash of the file the process actually loaded before trusting a profile.

With the right binary loaded, perf gave a clear answer (on that symbol build, a test tree, not a published release: 87% of cycles sat under the reads that plan a write, and 60% in one function, the hand-off from a read to the prefetcher, inside Rust's Mutex::lock_contended and the kernel's futex wait and wake).

text
owned_fetch_batch_status
  routed_caller_wave
    cascade_submit
      belt_motor::submit_predicted_on
        Mutex::lock_contended
        unlock

belt_motor::submit_predicted_on is the hand-off to the prefetcher, and the two frames under it are what a contended lock looks like on Linux: Rust's standard Mutex spins for a moment and then sleeps in the kernel on a futex (Mutex::lock_contended), and an unlock that finds sleepers makes a futex call to wake one. Most of the system time was that traffic: threads putting each other to sleep and waking each other up.

The bug: a lookup under the wrong lock

Semurg reads ahead. When a read touches a page, the graph it stores tells us which pages are likely next, and a prefetch motor fetches them in the background so the next read finds them in memory. To avoid reading a page twice, the motor keeps a set of pages it already owns (queued or in flight), and every hand-off checks each predicted page against that set under one process-wide lock.

The hand-off also asked, for every page, whether it was already in memory, and it asked while holding that lock. On a busy store most predicted pages are already in memory, so most of the work under the lock was a memory lookup that did not need it (the lookup has its own sharded read lock, which many threads can hold at once). And there were many callers. The engine is Rust, called from Elixir, and calls that may wait on a disk run on the BEAM's dirty-IO schedulers, which the release sizes at 4 per hardware thread of the host: our 8-core container sat on a host with 16 hardware threads, so 64 dirty-IO threads could be inside the engine at once, all arriving at one lock. Many dirty-IO threads per core is normal, since they mostly wait on the disk; here they waited on each other.

Before the fix: every write hands its predicted pages to the prefetcher, which takes one process-wide lock and only then asks whether each page is already in memory, so all 64 writers wait in line for the lock although most pages need no read.one at a timeAn agentOther agentsOne process-wide lockAsk of every predictedpage: already inmemory?Queue a readSet it asidewaitswaitno: the few missesyes: most

It hid well because every answer was correct. The slow path returned the right bytes, every functional test passed, and a single agent barely contends for the lock.

The fix

Do the residency check first, outside the lock, and take the lock only for the pages that were not in memory.

After the fix: each write first asks, with no lock, whether its predicted pages are already in memory and sets those aside; only the few misses take the lock, are asked about again under it, and are queued for the prefetch motor, so the writers wait for each other only over the few misses.one at a timeAn agentOther agentsAsk of every predictedpage, with no lock:already in memory?The same lockSet it asideAsk again: landedsince, or alreadyqueued?Queue a readno: the few missesyes: mostneither

In simplified code (the budget and admission checks are left out), the hand-off before the fix:

rust
let mut found = Vec::new();
let owned = owned_set.lock().unwrap();
for page in predicted {
    // the common case, checked while holding the lock
    if in_memory(page) { found.push(page); continue; }
    if owned.contains(page) { continue; } // queued or in flight
    queue_read(page);
}
drop(owned);
follow_through(&found); // read the next predictions from them

And after it, where the common case takes no process-wide lock:

rust
let (mut found, mut misses) = (Vec::new(), Vec::new());
for page in predicted {
    if in_memory(page) { found.push(page); } else { misses.push(page); }
}
if !misses.is_empty() {
    let owned = owned_set.lock().unwrap();
    for page in misses {
        // landed since the first check: use it, never read it twice
        if in_memory(page) { found.push(page); continue; }
        if owned.contains(page) { continue; }
        queue_read(page);
    }
}
follow_through(&found);

The second in_memory check under the lock keeps the old guarantee exact: a page that lands between the two checks is found in memory and not read a second time, because the motor puts a page in memory before it releases it from the owned set. The plain read-ahead path, which a read takes for the pages it asks for directly, had the same shape and got the same fix. The whole change is a few dozen lines in one file, plus the test below.

That second check had a price we did not see at first. A walk deep into a graph mostly asks for pages that are not in memory yet, and each of those was now looked up twice. Our release gate measures a traversal from a few hops to more than sixteen thousand on every release, and after this fix it read a few percent slower: inside its tolerance at first, then outside it, which stopped a release. The next release keeps the check exact without paying for it twice: the memory is split into shards, each shard counts the pages it has taken in, and a missing page is looked up again only if its shard took in a page since the first look. A page can only arrive by being taken in, so when the count has not moved the answer still stands. The traversal was back where it was before the lock fix.

A test that can fail

A fix for lock contention without a test that fails on the old code is a hope, and contention is hard to reproduce on demand. So the test does not try to recreate 64 threads; it holds the lock itself. Another thread takes the owned set and keeps it for 300 ms, and the test asks for 8 pages that are already in memory, once as a direct read-ahead and once as a prediction. The fixed code answers both within 100 ms, with all 8 counted as in memory and nothing queued for a read. The old code waits for the other thread to let go, and the test fails with a message saying how long a read of resident pages waited for a lock it did not need. The test and its negative control (the same test against the code before the fix, which must fail) ran in the gate that landed the fix, and the test now lives in the engine's test suite beside the code it guards.

Results, and what is still open

The fix shipped in r11.102. To compare, we alternated the shipped and patched builds on fresh containers, checking the hash of the loaded library before each run, then ran the same test against the published r11.102 image downloaded the way a user gets it. These are our own runs, not release-gate records: 64 writers each time, on one 8-core node, three runs per build.

Our runs (test tree, not a published release)Writes per secondSystem time
Published r11.101 image8.2, 27.4, 12.7up to 59%
Patched build, same A/B35.5, 34.7, 36.011 to 26%
Published r11.102 image31.9, 42.0, 37.610 to 31%

Neither the patched build nor the published release collapsed in any run, and both spent much less time in the kernel. On the partner's failed calls: we did not reproduce them, since the published r11.101 and r11.102 refused none in our runs, while the patched build, a test tree, not a published release, refused about one call in 1,400 at admission in two of its runs, and we have not yet found out why. We have not shown that this lock caused the partner's failures; a re-run of their test on the new release will tell us.

Two things remain, and we are working on both. Writes into one scope still stop scaling well before 64 writers on 8 cores: on r11.102, 64 writers wrote fewer per second than 16 writers did in a run of a symbol build of that release (test tree, not a published release). With 64 calls in flight at tens of writes per second, each call lives for over a second end to end, while one write alone takes milliseconds inside the node. In these runs the 8 cores stayed nearly fully busy, mostly in user time, so the calls are queuing for CPU, and we have not yet named the work that fills it; the local embedding model is our first suspect (test tree, not a published release). And a single write spent a large share of its time hashing the whole signed model catalogue again, only to look up a verdict it had already cached; that fix shipped in r11.104 (test tree, not a published release: from about 14.7 to 9.5 ms per write inside the node).

What we took from it

  • Profile the binary your users run: the same variant and flags plus symbols, loaded where the image loads it, and checked by hash.
  • Re-run the original test after every fix. Our first fix was for a real bug, and the 64-agent test still collapsed in some runs.
  • When throughput falls as you add threads and the lost time is system time, look for a contended lock first; on Linux it shows up as futex wait and wake.
  • Inside a lock, do only the work the lock protects. A check that is usually true and needs no lock belongs in front of it.
  • When a result swings between runs, alternate the versions you compare and repeat, rather than trusting one run of each.
  • A correct answer from a slow path never looks like a bug. Look at where the time goes, not only at what comes back.

This one was found because a partner ran many agents into one memory and told us what they saw. If you do the same and the time goes somewhere it should not, we want to hear about it, at semurg@semurg.org.

Related