I stress-tested Meta Muse until its agent control plane started timing out

Meta Muse is Meta’s individual AI agent, launched connected September 8, 2026. Rather than only answering questions, it is designed to transportation retired tasks connected a user’s behalf: it has its ain browser, tin support moving aft the app is closed, and runs successful a dedicated Muse Secure VM. In position it resembles Grok Bot — a personified supplier controlled done speech — but that is an interface-level analogy, not an presumption of shared architecture. This article goes 1 furniture little and examines a constrictive portion of the runtime: subagent fan-out, the durable authorities it leaves behind, and the spawn way nether load.

I ran these tests successful my ain Muse session, utilizing only interfaces exposed to that session: subagent.spawn, a ammunition successful the assigned environment, and a bounded read-only interface to durable diagnostic and trace data. I did not effort to entree different users, tenants, aliases information extracurricular the situation assigned to me, and I did not bypass entree controls. I americium not presenting this arsenic a Meta-authorized information assessment, and entree unsocial is not grounds that each load research was separately authorized. This is simply a black-box/reverse-engineering write-up from the position of the entree granted to my session.

Post-publication update — September 13, 2026. I clarified the entree scope, added independent architectural discourse from Rohan Adwankar’s analysis, separated STAGGERED-80 from the burst-style runs, and described 2 controls that would amended isolate cadence, topology, and concurrency. The experimental data, published CSVs, and figures were not changed.

At 06:45:32 UTC I asked a chat convention to spawn 120 subagents astatine once. Each 1 had a deliberately trivial job: tally slumber 30 successful a ammunition and study a azygous statement back.

Thirty-three of those calls created an agent. Eighty-seven grounded pinch the aforesaid database error. The aggregated reply ne'er arrived, and the interface yet showed an Error state. In the durable trace, each 1 of the 33 created agents reached a terminal completed grounds - 32 of them pinch a confirmed workload completion, and 1 still unresolved - while the grounds of the genitor still said running.

What follows is simply a black-box investigation of that gap, built wholly from records the runtime wrote to PostgreSQL arsenic it worked: an supplier registry, a spawn ledger, per-worker advancement tables and a context-item store. The load tests were not re-run to constitute this article. Everything present was reconstructed from stored state, pinch 1 documented objection I will travel backmost to.

What Muse looked for illustration from the outside

From a user’s spot this was an mean chat session. The persona matter was plain; the instrumentality database was not.

Among the devices the convention could telephone were subagent.spawn, subagent.close and subagent.resume, positive a read-only interface to a PostgreSQL database. That database was not incidental - it held the runtime’s ain bookkeeping. Tables specified arsenic agent.agents, agent.subagent_spawns and agent.subagent_progress_tool_events recorded which agents existed, who spawned them, which devices they ran and erstwhile they finished.

That is really the multi-agent building became visible astatine all. Not done the interface, which presented 1 conversation, but done the records the runtime kept for itself.

Two labels request attraction earlier thing other successful this article.

The first is the exemplary string. Every supplier statement I publication - the root, the coordinator, each 33 workers of the largest burst - carried the aforesaid value:

That drawstring is observed. What it intends is not. It could beryllium an soul exemplary build, a routing alias, aliases thing other entirely; the trace does not say, and I americium not going to guess. I americium publishing it verbatim because it is 1 of the fewer difficult identifiers the durable trace provides.

The 2nd is the runtime’s explanation of itself: Muse Spark 1.3, from Meta’s Muse family. That comes from the runtime’s ain discourse alternatively than the trace, truthful it is self-reported, not verified. Everything other successful this article is anchored successful records.

Independent architectural context

After publication I recovered an independent teardown of a modern Muse lawsuit by Rohan Adwankar. In his environment, PostgreSQL ran wrong the per-user VM complete a section Unix socket, and the harness binary contained some avocado-5.16-v4 and ipnext/... paths. That fits respective of my observations and gives them useful architectural context, but it does not alteration the bound of the evidence: my information still does not place the circumstantial lock, table, row, index, query, aliases transaction responsible for the timeouts.

I still dainty ipnext/avocado-5.16-v4 arsenic an observed exemplary identifier. Adwankar interprets ipnext arsenic Meta’s soul transport/gateway and avocado arsenic an soul exemplary family; straight mapping avocado to Muse Spark 1.3 remains an inference, not thing established by my trace. Likewise, my ain probe established KVM visibility, while Adwankar identified Cloud Hypervisor moving connected KVM successful his instance. I did not independently found the circumstantial VMM utilized by mine.

External source: Rohan Adwankar, “What’s successful a Muse?”

Method: counting overlapping agents

The experiments utilized 1 workload, 4 configurations and nary retries. Three of them - PROBE-40, BURST-80 and BURST-120 - were burst-style runs issued from the guidelines agent: each effort successful a configuration went retired successful a azygous turn. The fourth, STAGGERED-80, was deliberately dispersed complete clip and utilized a coordinator supplier to spawn its workers. No grounded telephone was repeated.

Concurrency present has a narrow, deliberate definition. An supplier counts arsenic progressive from its first instrumentality telephone until its terminal record:

Concurrency complete clip is simply a expanse statement complete those intervals, pinch ends processed earlier starts erstwhile timestamps tie. The timestamps person one-second resolution, truthful the expanse is deterministic for the recorded information but cannot retrieve arena ordering wrong the aforesaid second.

Three consequences use to each number below:

1. This measures supplier activity, not inference. Thirty-three overlapping activity windows are not thirty-three simultaneous exemplary calls. Nothing present measures the conclusion backend.
2. A highest is an observation, not a limit. The tests ne'er exceeded 120 simultaneous attempts, truthful they cannot found a concurrency headdress - aliases norm 1 out.
3. A missing grounds is evidence. Failed spawn calls time off thing down successful the supplier registry. That asymmetry shaped really failures had to beryllium verified, and it is why the nonaccomplishment counts were the hardest numbers to pin down.

The first probe: 40 calls

The first research was a calibration run: 40 spawn calls issued astatine 06:10:36 UTC.

It created 39 agents, admitted wrong 2 seconds of each other. One telephone failed, and that nonaccomplishment is worthy a person look, because it was recovered from the durable instrumentality trace alternatively than taken from the chat: the telephone went retired astatine 06:10:44 UTC, the correction came backmost astatine 06:10:56 UTC, and nary kid supplier was created.

The highest observed concurrency was 39. That fig is an archival recomputation from the session’s trace table, not a unrecorded measurement, and the published summary says so.

One much item from that reconstruction carries weaker provenance than thing other successful this section. In the session-built table, eleven of the thirty-nine created agents vanished pinch a background-processing position connection alternatively of the requested DONE line. Because it survives only successful a hand-built table, I study it pinch that caveat and tie nary conclusions from it - isolated from that “terminal status” and “the workload finished” are not the aforesaid statement. That favoritism matters overmuch much later.

Eighty astatine once, and eighty dispersed out

The adjacent 2 experiments look for illustration a cleanable A/B test. They are not.

Burst-style runs

STAGGERED-80 — abstracted configuration

I show STAGGERED-80 separately because this tally changed some cadence and topology. Its 0% nonaccomplishment complaint is truthful not a 4th constituent successful the aforesaid bid arsenic 2.5%, 6.25%, and 72.5%.

BURST-80 issued 80 spawn calls astatine once, astatine 06:30:43 UTC. Seventy-five agents were created - each of them nonstop children of the guidelines supplier - and 5 calls failed. All 5 returned the identical database fastener timeout, each astir 58 seconds aft the telephone was placed. The highest observed concurrency was 72, recomputed aft the tally from the study input for that phase.

STAGGERED-80 started astatine 06:32:03 UTC and issued the aforesaid number of calls, dispersed out. It created each 80, pinch zero failures. It is besides the research that refuses to beryllium a cleanable control, because 2 things changed astatine once. First the cadence: the intended spacing was 100-200 milliseconds, but the measured mean admittance spread was 1.1266 seconds, astir 10 times wider than planned. Then the topology: the guidelines spawned 1 coordinator astatine extent 1, and the coordinator spawned the 80 workers astatine extent 2. The burst experiments spawned workers straight from the root.

The staggered tally besides peaked little - 38 progressive workers against 72 successful the comparable burst - truthful the 2 runs disagree connected much than their nonaccomplishment counts.

Observed active-agent concurrency complete time. Activity is defined arsenic first_tool_at <= t < finished_at, pinch finishes processed earlier starts connected timestamp ties; timestamps person one-second resolution. Each configuration was tally once. STAGGERED-80 besides changed topology, truthful this is not a cadence-only comparison; the curves show supplier activity, not conclusion concurrency. Source: BURST-80 aggregate reconstructed from the complete archived trace; STAGGERED-80 independently recomputable from the published worker rows.

Three admittance shapes, 3 observed outcomes. Source: sanitized admittance windows and worker activity rows published pinch this article.

The cliff betwixt 80 and 120

The 4th research is wherever the nonaccomplishment complaint stops being a curiosity.

At 06:45:32 UTC the guidelines supplier issued 120 spawn calls successful 1 turn. The research wrote its ain ledger for that turn, and it is complete:

• 33 created (27.5%), each 1 a nonstop kid of the root, each admittance recorded successful the aforesaid second: 06:46:01 UTC, twenty-nine seconds aft the burst was issued.
• 87 failed (72.5%), each pinch the database fastener timeout, nary retried.

Across the 3 burst runs the observed nonaccomplishment rates were 2.5%, 6.25% and 72.5% - 1 tally per configuration. The staggered run’s 0% does not beryllium successful that comparison, because it besides changed topology. The largest measurement came betwixt 80 and 120 attempts, and that is astir each the information says: a suggestive difference, not a complaint law.

Failure complaint by configuration. The staggered version is plotted separately because it changed 2 variables astatine once. Source: sanitized research summary published pinch this article.

The highest observed concurrency successful this tally was 33, astatine 06:46:24 UTC. The runtime was not asked for 33 workers; it was asked for 120. What the trace shows is 33 workers pinch activity intervals and 87 attempts recorded arsenic spawn failures. Their call-to-error latencies were not captured, truthful it does not show erstwhile those failures came backmost comparative to the admissions.

Observed: some peaks are recomputable from the published rows. A expanse statement complete the 33 BURST-120 intervals gives 33 astatine 06:46:24 UTC, and the aforesaid process complete the 80 staggered rows gives 38 astatine 06:32:48 UTC. Neither is simply a cap; some are azygous observations.

The 33 workers that existed, from first instrumentality telephone to terminal record. One barroom is dashed: C-85. Source: sanitized per-worker timings published pinch this article.

Even the calls that succeeded were slow to go workers. Measured from the infinitesimal the burst was issued, the median created worker reached its first instrumentality telephone aft astir 36.5 seconds; measured from its admission, the median was astir 7.5 seconds. The trace establishes erstwhile the admissions were recorded and erstwhile each worker first called a instrumentality - it does not decompose the interval betwixt those 2 points, truthful thing present should beryllium publication arsenic a connection astir what the runtime was doing during it.

Counting failures successful this strategy has a structural problem: a grounded spawn creates nary child-agent statement and nary spawn-ledger row. The nonaccomplishment itself is not mislaid - it survives successful the durable instrumentality trace, arsenic the stored telephone and its output - but you person to cognize wherever to look. The archive of this convention was stiff earlier that walk was done. Six grounded calls were recovered later from discourse items, which is besides why the research needed its ain ledger to count the rest:

All six returned the aforesaid payload, characteristic for character:

And for each six, the antagonistic checks agree: nary child-agent row, nary spawn-ledger row, nary kid agent. Each telephone grounded connected a database cognition pinch a fastener timeout earlier immoderate of those rows appeared. Which SQL cognition was progressive is not known from this data.

Twelve seconds, fifty-eight seconds, and what they do not mean

Both latencies successful that array - 12 seconds for the probe failure, astir 58 seconds for the 5 burst failures - are elapsed clip betwixt 2 events connected the telephone path: the telephone going out, and the correction coming back. They are not a measurement of really agelong those calls waited connected a lock, and they do not show america the configured timeout value.

A timeout worth is simply a configuration fact. The trace records elapsed clip betwixt 2 events. Those are different things, and the quality matters, because this is precisely the benignant of spread wherever a confident-sounding number turns into a incorrect architectural claim.

Unknown: whether that long is fixed, whether the telephone is retried anyplace else, and whether the calls spent the full interval waiting connected 1 lock. The latency of the 87 failures successful the 120-attempt burst was ne'er captured astatine each - that ledger records outcomes and correction classes only.

Where the grounds points

The nonaccomplishment grounds points toward 1 place: the way that writes a spawn. The failures hap connected subagent.spawn, they transportation a PostgreSQL fastener timeout from the database layer, they create nothing, and they look successful 2 abstracted burst runs pinch an identical fingerprint. The workers that do get created tally their trivial workload without incident: 32 of the 33 were confirmed to person tally the afloat 30-second sleep, pinch activity windows of 32 to 43 seconds.

That is simply a coherent picture, and it supports a circumstantial hypothesis.

Two unknowns beryllium astir that hypothesis, and some are easy to gloss over:

• Admission policy. The trace shows 33 admissions successful the aforesaid 2nd and 87 timeouts. It records thing astir ordering, queueing aliases admittance logic. The information cannot separate “rejected aft a wait” from “queued internally and past dropped”, and it cannot show you which telephone was served earlier which.
• Failure latency for the large burst. The 87 failures were logged arsenic outcomes, not arsenic timed events. Only the six recovered failures person timestamps connected some ends.

The reply that ne'er arrived

The nonaccomplishment that matters astir to a personification is not a timeout. It is what the convention looked for illustration afterwards.

After the 120-attempt burst, the durable authorities was unambiguous astir the workers and unusual astir the parent:

The last aggregation ne'er reached the user. The past worker vanished astatine 06:46:56 UTC. When the guidelines statement was publication during the betterment pass, its updated_at was 07:00:59 UTC and its position was still running; the interface showed Error astir that time. That interface authorities is simply a presentation-layer study pinch nary corresponding grounds successful the trace.

What the trace supports is narrower - and much useful - than the communicative it invites: durable worker results tin beryllium independently of the genitor delivering an aggregated answer. That is simply a connection astir records, not astir survival. No genitor supplier was ever grounded connected intent during these experiments, and thing successful the trace records a genitor nonaccomplishment while workers were running. Whether children outlive a genitor nonaccomplishment was not tested and is not known.

The study that sewage up of the trace

One smaller incident belongs connected the record, because it is the aforesaid correction successful miniature.

The first generated type of the 120-attempt study asserted that the genitor had grounded while its workers kept running. The trace did not support that. The draught was produced astatine 07:05:54 UTC; a corrected HTML study followed astatine 07:13 UTC, built connected the durable grounds alternatively - guidelines running, nary nonaccomplishment entry, past worker vanished astir thirteen minutes earlier the Error authorities appeared. The rendered PDF was past synchronised pinch the corrected conclusion successful 2 edit rounds, and the consistency audits passed.

What produced the bad first draught is unknown. A plausible reference is that an asynchronously generated artifact worked from an earlier snapshot of the reasoning than the chat did. That is simply a presumption astir tooling, not a uncovering astir the runtime - but the operational instruction is concrete: for asynchronously generated artifacts, the canonical root has to beryllium declared, and cross-artifact audits are not optional.

Silence is not a hang

One much observation, because it costs thing and taught something.

An artifact update was requested astatine 07:28:29 UTC. For the adjacent thirty-two minutes the interface showed nary caller advancement events astatine all. At 08:00:48 UTC the task completed successfully, without being prompted aliases restarted.

Progress events are a anemic liveness signal. Their absence is not grounds that a task is stuck, and acting connected that absence - sidesplitting the activity and re-issuing it - would person thrown distant a task that was going to finish. Why the builder went quiet for half an hr is chartless from durable data.

The situation snapshot is the slightest glamorous portion of this evidence, and 1 of the much useful parts if you want to logic astir wherever the activity really ran.

The container-in-a-VM reference - KVM above, systemd-nspawn around, overlay connected Btrfs - is an mentation of those signals, and it describes the sandbox wherever instrumentality calls ran. It says thing astir wherever conclusion happens. Whether the exemplary was served connected the aforesaid host, successful the aforesaid cluster, aliases location other wholly is unknown; thing successful this information connects the sandbox to the serving path.

What this does and does not establish

Supported by the evidence

• A consumer-facing chat convention ran a multi-agent runtime pinch a durable PostgreSQL trace, an definitive parent/child registry, and depth-1 and depth-2 delegation successful use.
• In the burst runs, the observed spawn-failure rates were 2.5% astatine 40 attempts, 6.25% astatine 80 attempts and 72.5% astatine 120 attempts - 1 tally per configuration, reported descriptively alternatively than arsenic a complaint law.
• Six failures were recovered from the durable instrumentality trace; each six are the aforesaid nonaccomplishment type, and each six created nothing.
• Contention connected the PostgreSQL spawn constitute way is simply a powerfully supported hypothesis.
• Durable worker results tin beryllium moreover erstwhile the last aggregated reply ne'er reaches the user.
• Absence of UI advancement events is not impervious of a hang.

Not established by this evidence

• Which lock, table, row, index, query aliases transaction was contended.
• The configured fastener timeout, aliases why the 2 observed nonaccomplishment latencies differ.
• Any scheduler aliases admittance limit; the peaks (39, 72, 38, 33) are azygous observations, not measurements of a cap.
• The existent parallelism of the conclusion backend. Overlapping supplier activity is not overlapping exemplary calls.
• The admittance policy, the ordering, aliases whether immoderate soul queueing was involved.
• Whether a kid supplier survives a genitor nonaccomplishment - that research was ne'er run.
• Whether extent 3 is allowed, whether subagents tin beryllium fixed a different model, aliases what ipnext/avocado-5.16-v4 denotes.

What the adjacent controlled tally would request to resolve

These information do not abstracted cadence, topology, and highest concurrency cleanly, because STAGGERED-80 changed the first 2 astatine the aforesaid clip and besides reached a little observed peak. Two elemental controls would discriminate betwixt those explanations overmuch better.

The first would sphere the guidelines → worker topology and burst-style admittance while imposing a difficult headdress of astir 38 concurrently progressive workers. The 2nd would sphere staggered admittance but spawn workers straight from the root, without the intermediate coordinator. The first would thief abstracted highest concurrency from burst admittance itself; the 2nd would abstracted cadence from topology.

I did not tally either control. I americium listing them arsenic a replication creation and a measurement to isolate the variables, not arsenic missing results from this study.

Every configuration was tally once. These are azygous observations of 1 convention connected 1 afternoon, not a capacity characterisation of a platform.

What I would return distant if I built supplier power planes

1. Look astatine the power level first, but do not extremity there. The load-related failures happened connected the admittance path: 87 attempts ne'er became agents, while the workers that did beryllium ran their trivial workload. The extremity of the tally grounded otherwise - the last aggregation ne'er reached the user, and C-85’s result is still unresolved.
2. Durable records hit position flags. The interface said Error, the genitor said running, the workers said completed. Only the trace could beryllium interrogated afterwards.
3. Keep antagonistic results. A grounded spawn leaves nary child-agent and nary spawn-ledger row; the only durable trace of it is the instrumentality telephone and its output. Without a ledger of its own, the research could not person counted the 87.
4. Terminal position is not semantic success. C-85 is the counter-example.
5. Log latencies for failures, not conscionable outcomes. Six failures person call-to-error timings; eighty-seven person none. That asymmetry limits what tin beryllium said astir the largest nonaccomplishment arena of the session.
6. Progress events are a hint, not a heartbeat. Thirty-two minutes of soundlessness ended successful a successful artifact build.
7. Declare the canonical artifact. An asynchronously generated study tin transportation a conclusion that later reasoning has already abandoned.
8. Test the nonaccomplishment modes you intend to claim. Parent-failure survivability is the evident communicative here, and it is precisely the 1 this information cannot support.

None of that requires entree to a model’s internals. It requires a runtime that writes down what it did.

Limitations, and wherever the numbers travel from

Everything beneath is published astatine /evidence/meta-muse-black-box-testing/ - the CSVs, the figures, and the book that redraws them from the data:

Agent identifiers were removed from each of these files. What is published is timing, status, correction people and correction text.

Three limitations beryllium to each declare above:

• No replication. One tally per configuration, nary repeats, nary intermediate effort counts.
• Archival provenance wherever it applies. The peaks for the 40-attempt and 80-attempt bursts were recomputed aft the truth from the session’s trace table; the peaks for the 120-attempt burst and the staggered tally are independently recomputable from the published rows.
• One deliberate omission. I do not people the earthy per-worker rows for BURST-80 because the archived trace contains shortened supplier identifiers covered by the publication policy. The archived trace itself is complete: it contains each 75 created workers, and the aforesaid expanse recomputes the canonical highest of 72 astatine 06:31:38 UTC. BURST-80 truthful remains aggregate successful the nationalist release.

The absorbing consequence present is not that a strategy fell complete astatine immoderate number. It is that it fell complete successful a measurement that near a legible way - circumstantial capable to norm retired astir of the stories 1 would want to show astir it. That way is published pinch this article, truthful the parts I sewage incorrect tin beryllium checked too.