Cloudflare discloses a cross-request KV cache leak in GLM-5.3 on Workers AI: a config mismatch plus an SGLang bug let 155 responses carry other users' data over 13 days

Core Background & Industry Pain Points

For performance, inference engines reuse pages of a shared GPU KV cache pool across requests without zeroing them, and the engine alone tracks ownership. Cloudflare's report shows what happens when that tracking fails: users get a normal-looking HTTP 200 reply that contains someone else's prompt and generations, invisible to conventional monitoring. It is a tenant-isolation risk for every cloud hosting open models.

Architecture Highlights & Internals

Workers AI runs GLM-5.3 on SGLang in disaggregated prefill/decode mode: prefill GPUs compute the KV cache and hand it over RDMA via Mooncake to decode GPUs. Two issues overlapped. The context was configured at 1,048,576 tokens while the prefill KV pool held only 681,280; the team knew but assumed the engine would return an error. And SGLang's admission queues checked only whether a request was finished, so rejected requests were admitted; prefill ran a one-token placeholder, reported the transfer complete, and decode generated over stale, unwritten pages.

Authoritative Benchmarks & Measured Scores

There are no benchmarks here. Key figures per Cloudflare: 155 affected responses over about 13 days; the fixed prefill pool holds about 2.04 million tokens; the model went offline within 40 minutes of the incident being declared. The engine fix shipped in SGLang v0.5.20, with follow-ups in v0.5.21 and v0.5.22.

Developer Hands-on Guide

Teams self-hosting SGLang should upgrade to v0.5.22 or later and, before any deployment change, verify from the running engine that the KV pool is at least as large as the context length, rather than trusting HTTP status codes. Long-context tests should check output content and coherence, and canary markers can detect cross-request leakage.