Cross-request KV cache exposure in the GLM-5.3 Workers AI model (155 responses)

On September 8, 2026, we were alerted by a safety researcher, Gadi Evron from Knostic, to a tenant isolation matter affecting the GLM-5.3 Workers AI example (@cf/zai-org/glm-5.3) starting on August 27, 2026. We have remediated the issue, and we restored the example on September 9. We have confirmed that no another example hosted on Workers AI was affected.

Between August 27 and September 9, a small figure of customers using the GLM 5.3 Workers AI example received responses that contained data from another customers' requests made to the identical example shortly before theirs. In total, 155 responses were affected, all to prompts longer than 681,280 tokens.

After excluding our own evaluation and the researcher's, the requests we identified are accordant in timing and quantity alongside average long-context use, not alongside person often exploiting the bug. We established no evidence of deliberate exploitation.

Today we are publishing this study alongside a article from SGLang, the open-source conclusion motor that we use, as a coordinated disclosure. This article describes the bug that caused this issue, our reply timeline, and the consequent hardening we have conducted on our manufacturing SGLang deployments. We additionally item our continued efforts alongside the upstream maintainers of SGLang to location this and akin bugs.

What happened

GLM is a content generation example from Z.ai that is built for agentic coding workflows; it is among the many models available through Workers AI. Gadi Evron submitted a study on September 8 through our HackerOne program stating that extremely lengthy prompts sent to @cf/zai-org/glm-5.3 on Workers AI sometimes returned content that belonged to another users' sessions. Cloudflare reproduced the matter the next dawn and took the example offline inside 40 minutes of declaring an incident. We afterward deployed a corrected configuration and a patched conclusion engine, and restored assistance the identical day.  

This particular study of cross-request KV cache visibility was the outcome of two overlapping issues. The archetypal was a misconfiguration of the open-source conclusion engine, SGLang, used by Cloudflare. We deployed GLM 5.3 alongside a 1,048,576-token environment window, but the KV cache pond on our prefill GPUs lone had area for 681,280 tokens. We were conscious of this mismatch before launch. We expected the motor to come back an error for any immediate it could not process, and our long-context tests returned prosperous responses, so we did not accept this to be an issue.

The second was a bug in how SGLang handled the resulting over-length requests in disaggregated prefill/decode mode. A immediate that did not fit the pond was internally marked as rejected and replaced alongside a one-token placeholder, but alternatively of aborting the petition and returning a client error, prefill generated the KV cache for the placeholder immediate and transferred it to the decode node. The decode node received a "transfer complete" communication and assumed that prefill overwrote the KV cache from former requests alongside the complete immediate for this request, whenever in fact it remaining the bulk of the former province intact. The decode node subsequently generated a reply during attending complete old cache pages that motionless held data from before requests. As a result, responses to prompts longer than 681,280 tokens could merge data from another users' GLM 5.3 sessions.

Immediate remediation

After disabling the model, we took the following actions to accurate this exposure before restoring community admission to it:

To fix our configuration, we made the pond clasp the complete context. We adjusted our GLM 5.3 configuration to equivalent our GLM 5.2 configuration, which freed adequate GPU recollection that the prefill KV pond now holds concerning 2.04 myriad tokens, approximately twice the 1,048,576-token environment length. A immediate that fits the environment opening now continually fits the pool, so the rejection way is never reached for a valid request. This solitary prevents the bug from being exercised.

To fix the engine, we additionally made the rejection way safe. Relying on pond size solitary is fragile: any alter to quantization, tensor parallelism, CUDA chart sizes, or a forthcoming example type changes the pond size in a way that's uncomplicated to miss. Furthermore, there may be another situations in which the prefill scheduler can refuse a petition and trigger the identical bug. To forestall this, we additionally patched the motor so that a petition carrying a pending rejection is former at the disaggregation admission queue, alongside its first error, before any bootstrap, allocation, or KV transfer happens. We projected this upstream as #38704, which additionally had the decode node verify that the figure of tokens transferred matches the figure it reserved. The fix was already on SGLang chief in #36988, merged September 6 for a connected nonaccomplishment way but not yet released. The maintainers additionally merged #38935, which rejects these requests earlier, following our approach.

We validated the two changes against the researcher's reproduction manuscript and our own canary-marker tests before restoring the model.

Background: how we assist GLM 5.3

Workers AI runs GLM 5.3 alongside SGLang in a disaggregated configuration: a pond of prefill GPUs processes the immediate and produces the model's KV cache for it, afterward hands that cache complete distant straightforward recollection admission (RDMA), via Mooncake, to a pond of decode GPUs that create the response. We have written concerning this architecture before in our posts on high-performance LLMs and smaller, faster, safer models.

The KV cache in GPU recollection is a shared pond of fixed-size pages. Each petition is assigned pages whenever it is admitted and releases them whenever it finishes. Pages are not currently zeroed between requests; the next petition that gets a leaf merely overwrites it. This method is average among conclusion engines and is necessary to accomplish the finest performance, but it method that the conclusion motor is liable for tracking ownership of these recollection pages. KV cache leaf ownership and life tracking has to agreement alongside not fair shared ownership inside a node (KV cache offload and token generation the two admission the recollection alongside distinct lifetimes), but additionally ownership that is technically shared alongside the distant prefill node (which holds a simultaneous mutable citation to the page).

Furthermore, the size of this pond is not a configuration value. SGLang sizes it at startup from any GPU recollection remains following loading the example weights. The model's advertised environment dimension is a distinct configuration value. Nothing forces the two to agree, and it is our job to inspect that they do.

Issue one: misconfiguration

On our GLM 5.3 prefill nodes alongside our configuration, the pond came to 681,280 tokens during the model's environment dimension is 1,048,576 tokens. The selected KV pond size is printed by SGLang in logs during startup and exposed as a metric. We were conscious of this size mismatch; however, during pre-launch evaluation we observed prosperous HTTP position codes and a streaming build alongside no error frames during benchmarking. We assumed that the motor was capable of gracefully handling the difference between the KV cache pond and requested environment opening via streaming or several another mechanism.

This was one crucial gap in our testing: we lone validated HTTP reply codes and motor achievement at lengthy contexts, without validating example output and coherence. We now inspect KV pond size against environment dimension directly, from the operating engine, before any disaggregated deployment alter ships, fairly than inferring it from reply codes.

Issue two: SGLang bug

In v0.5.21, the SGLang squad has now addressed this bug in how it was handling prompts longer than the KV pond size. Here’s what was happening before.

When a immediate arrived that was longer than the KV pond could hold, SGLang's scheduler rejected it. The rejection way (Req.set_finish_with_abort) recorded the error to be returned to the client and exchanged the immediate for a one-token placeholder so that the petition could drain through the normal lot machinery without allocating genuine memory. On a single-node deployment this worked: the placeholder ran, the lot finished, and the client received an error.

In disaggregated mode such as we operate, the admission queues on the two prefill and decode checked whether a petition was finished, not whether it was rejected. A rejected petition reported itself as live. So:

1. Bootstrap. The router told the decode node the immediate dimension (8 pages in the diagram). Decode reserved 8 KV pages for the request.
2. Prefill admission. The prefill admission queue admitted any petition anywhere req.finished() was False. A rejected petition stored its rejection in to_finish, not finished_reason, so finished() still returned False. The rejected petition was admitted.
3. Prefill onward pass. Rejection had already replaced the immediate alongside a one-token placeholder. Prefill ran a trivial onward continue on that one token.
4. KV transfer. Prefill sent one leaf of KV to decode and signaled that the transfer was complete.
5. Decode admission. Decode's admission queue had the identical finished() check, so it accepted the handoff.
6. Decode attention. Decode started generating and notice peruse all 8 reserved pages. Only leaf P0 had been written by this request. Pages P1 to P7 motionless held what before requests remaining in them: another users' prompts and partial generations.
7. Response. The client got HTTP 200 alternatively of HTTP 400. prompt_tokens matched the lengthy immediate and the answer was coherent. It arrived faster than an 8-page prefill should take, and it contained satisfied from another users' KV pages.

Every petition complete the threshold behaved this way. This bug had been reported to SGLang by another person as matter #30233 on July 6, 2026, which Cloudflare noticed during examining this incident. We had been flagging these types of issues alongside SGLang since v0.5.10 and this safety event sped up activity that Cloudflare had already started in collaboration alongside the SGLang team. The place for this matter was released in SGLang v0.5.20, alongside follow-ups in v0.5.21 and v0.5.22, and is accessible for all another operators reliant on SGLang to use. We extremely propose all operators update SGLang requirements to v0.5.22 or later. For additional particulars on particular PRs, peruse on below.

GLM 5.3 was the lone example affected by this bug. Every another disaggregated deployment on Workers AI has KV pools larger than its environment dimension on the two prefill and decode, so no petition could attain the rejection path. We have applied the motor fix and the hardening described below to all example we assist alongside SGLang, and our deployment checks necessitate them for all new deployment.

Timeline

All times UTC.

Ongoing hardening effort

This bug belongs to a category of KV cache memory-safety bugs that Cloudflare has been operating to location for multiple months. In all case, a leaf is freed, reused, or peruse during item alternatively motionless has a assertion on it. We see two average causes.

The archetypal is leaf life inside a sole node. Several tasks on a decode node can clasp the identical leaf at once: token generation, offload to presenter memory, the KV transfer receiver, and the abort and error paths. If one of them frees the leaf during another is motionless using it, a new petition can be stated that leaf during it is motionless being peruse or written. Our before fixes for decode offload (#24226, #30899) and twice publish on grammar errors (#30937) were of this kind.

The second is a competition throughout nodes. When a petition is aborted during prefill is motionless penning its KV cache to decode complete RDMA, the writes already sent cannot be recalled. If decode frees those pages correct away, those pages can be stated to another request, and the delayed writes district in that request's KV cache. Cloudflare has contributed to the attempt to fix this competition through projected fixes (#31357, #38961) and a reproduction manuscript (#32564). In the two cases the outcome is corrupted output, or one user's data in another user's response, alongside no error raised. The bug in this event was different: a rejected petition went through the handoff all time, fairly than by a timing accident. But it had the identical result: decode study pages this petition never wrote. None of these covered the nonaccomplishment in this event in which prefill rejected the request, and decode never learned of it.

Since the incident, we have:

• Extended deferred publish to clasp prefill's origin pages as well, and to shield all nonaccomplishment type, not fair aborts. We projected it upstream as #38961, which was superseded by #40645, #41402, and #41404.
• Deployed bounded Mooncake transfer batches (#32758, from the SGLang community), so no sole transfer call can outlast the publish timeout.
• Changed our burden balancer so a client disconnect no longer aborts a petition mid-transfer. It holds the prefill association until prefill finishes and the decode association until the archetypal token, afterward closes both.
• Added a inspect to all disaggregated deployment change: KV pond size measured against environment length, deferred publish on the two roles, transfer lot limits inner the publish timeout, and all required patches applied.

All of these changes were released to our disaggregated manufacturing deployments.

Working alongside the SGLang team

We met alongside the SGLang maintainers at RadixArk on September 10, the day following the incident. We accepted that RadixArk would examine the competition conditions we had reported, upstream the integrity-checking idea, and build reproduction tests for this category of bug.

The following have landed on SGLang chief to location this event and connected KV cache competition conditions, and all but one are included in v0.5.22. Operators operating disaggregated SGLang should update to v0.5.22 or backport these patches as needed. 

These follow-ups are motionless plant in progress:

• #39729 Additional cleanup for --disaggregation-enable-kv-checksum. Merged, awaiting release.
• Zero KV pages on release, outline proposal: #46435
• Extend KV checksum validation from the prefill-to-decode handoff to all decode stage (see Integrity by construction, below)
• Broader KV Canary and KV Checksum example and characteristic support

What we propose to another operators

If you run SGLang in disaggregated prefill/decode mode, we propose the following:

1. Measure your KV pond and difference it to your environment length, on all prefill and decode node, from the engine's startup log, following all alter to weights, quantization, parallelism, or recollection fraction. If the pond is smaller, lesser the environment dimension to fit. Do not depend on a proxy to enforce the limit, and do not assume the motor volition refuse what does not fit. This is the inspect we did not enforce.
2. Upgrade to a build that includes #36988 and #38935 (v0.5.22 or later), so an over-length petition is rejected before handoff despite of pond size. This is the engine-side fix, and it is the backstop for the day your inspect in stage one is wrong.
3. Run alongside deferred KV publish on the two roles (default following #41023), and border your Mooncake transfer batches so a sole timed-out call cannot depart an unbounded rear of writes.
4. Turn on misconduct finding anywhere your example supports it. --disaggregation-enable-kv-checksum and --kv-canary do not yet shield all example architecture. Check your logs to verify requests are being verified. We are operating alongside the SGLang squad to broaden the two checks to additional architectures, including GLM 5.3, GLM 5.3 Flash, and DeepSeek V4.
5. Ensure your proxy is not generating unnecessary aborts. Configure your burden balancer to authorize requests to continue up to the archetypal token, so that you do not trigger an abort that requires cross-node coordination.

Longer term

Integrity by construction

Configuration checks and lifecycle fixes decrease the figure of ways a leaf can be misused. They do not create misuse impossible, and this event is a reminder that the next bug volition have a form we have not seen. The durable answer is for the motor to detect, at peruse time, that a petition is concerning to act complete a leaf it does not own, and neglect that one petition fairly than come back person else's data.

We built a prototype of this complete the summer and described it in August. Every bodily KV leaf carries a tag that changes whenever the leaf is reallocated, all petition records the tags it expects, and the mapping is checked before decode says the cache. In our tests it disbursal under 1% in throughput and rear latency. We tested it on GLM 5.2 in mid-August, but impaired it shortly afterwards because of reliability issues.

The prototype was additionally tied to particular notice backends and example architectures, so it could not be a broad fix. We are operating alongside the SGLang squad to build a manufacturing type into the motor that plant throughout all model, fairly than maintaining Cloudflare-specific patches. The per-transfer checksum in #39500 is a archetypal step: it validates the handoff from prefill to decode. Page-level tags go further, since they inspect all read, including says lengthy following the transfer.

Integrity by mathematical impossibility

RDMA copies data immediately from one node’s recollection to another’s. It’s critical for the two machines to concur on anywhere the data goes, how ample it is, and whenever it’s finished; any mismatch can logic the receiver to peruse the incorrect data. This bug is an example of a consistency nonaccomplishment in a distributed system, in which the two machines had inconsistent views of the province of that system.

Elsewhere in Cloudflare, we have put attempt into proving the safety of distributed systems using ceremonial verification methods. Topaz enables conflict-free IP choice for our authoritative DNS servers. Models of Meerkat’s QuePaxa implementation have demonstrated consistency bugs that were subsequently fixed.

We are now directing our cognition in ceremonial methods to KV cache transfer engines. This volition commencement alongside modeling and proving the safety of an RDMA-based KV cache transfer protocol. The goal is to example our manufacturing SGLang’s distributed KV cache scheme using Mooncake, afterward track and fix any divergence from the safety properties we desire to guarantee. The aim is to demonstrate — as far as imaginable — that a bug akin this cannot happen again. Realistically, we anticipate gaps in what the provers can inform us, but proving the remainder of the scheme allows us to additional closely assessment and detect those gaps.

Coordinated disclosure

We are publishing this study as part of a coordinated disclosure. We desire to appreciate the researcher, Gadi Evron, who reported this through our HackerOne program alongside a clear, reproducible manuscript and gave us period to fix it and to coordinate alongside the SGLang maintainers before publishing. We additionally desire to appreciate the SGLang squad at RadixArk, who treated this as their issue as fine as ours and landed the upstream fixes alongside urgency.

Read additional at the article from the SGLang team.

If you discover a safety matter in Workers AI, delight study it at hackerone.com/cloudflare.