Cloudflare Bug Put Other Users' Data in 155 AI Replies

For almost two weeks this summer, a few customers of Cloudflare’s Workers AI who sent very long prompts to the GLM-5.3 model got back answers containing pieces of other people’s requests. Cloudflare disclosed the problem on 26 October, saying 155 responses were affected, “all to prompts longer than 681,280 tokens”, between 27 August and 9 September.

The incident is seven weeks old, but this is the first time Cloudflare has said publicly that data crossed between customers. At the time, its status page described only “increased Workers AI errors” with GLM 5.3. The company says it found “no evidence of deliberate exploitation” and that no other model hosted on Workers AI was affected.

In brief

Cloudflare says 155 responses from its GLM-5.3 model on Workers AI contained data from other customers’ requests between 27 August and 9 September.
Only prompts longer than 681,280 tokens were affected, and Cloudflare says it found no evidence of deliberate exploitation.
The cause was a known memory-pool shortfall combined with a bug in the open-source SGLang engine; both are fixed.
The leak was first disclosed on 26 October; at the time, the status page reported only errors.

A researcher’s report took the model offline the next day

The bug was reported on the evening of 8 September by security researcher Gadi Evron of Knostic, through Cloudflare’s HackerOne bug bounty programme. His report said that very long prompts to the model “sometimes returned text that belonged to other users’ sessions”. Cloudflare says it reproduced the leak the next morning using two independent accounts and took the model offline “within 40 minutes of declaring an incident”. A corrected deployment went live that evening, at 19:40 UTC on 9 September.

After setting aside its own tests and the researcher’s, Cloudflare says the affected requests “are consistent in timing and volume with ordinary long-context use, not with someone repeatedly exploiting the bug”. It has not said which customers were involved or what the leaked fragments contained.

The disclosure was coordinated with the maintainers of SGLang, the open-source inference engine Cloudflare uses, who published their own account of the fix the same day. The SGLang team says the researcher’s report will follow.

A known size mismatch met an engine bug

Two problems lined up. Cloudflare launched GLM-5.3 with a context window of 1,048,576 tokens, but the memory pool on its prefill GPUs, which hold the model’s working cache for each prompt (the KV cache), only had room for 681,280 tokens. Cloudflare says it knew about the gap before launch. It expected the engine to return an error for any prompt that did not fit, and its long-context tests came back with successful responses.

The second problem was in SGLang. Cloudflare splits the work between “prefill” machines that read the prompt and “decode” machines that write the reply. In that set-up, a prompt too long for the pool was marked as rejected but not stopped. Prefill processed a one-token placeholder instead and told decode the transfer was complete. Decode then wrote its reply while reading cache pages that still held earlier users’ prompts and partial answers. The customer received a normal, coherent response, not an error.

The engine flaw was not new. Another user had reported it to SGLang on 6 July as issue #30233, which Cloudflare says it noticed only while investigating. A related fix had been merged into SGLang’s main code on 6 September but not yet released. Cloudflare’s own patch was proposed upstream as #38704. SGLang says that on a single machine, the same rejection path correctly returned an error.

Cloudflare’s fix was to enlarge the pool to about 2.04 million tokens, roughly twice the context window, so a valid prompt can no longer reach the faulty path. It also patched the engine so a rejected request is retired before any memory is handed over, and now checks pool size against context length before every change to this kind of deployment. It recommends that other operators running SGLang in the same split mode upgrade to version 0.5.22 or later.

Shared GPU memory is now part of the attack surface

The striking admission in Cloudflare’s post is how ordinary the cause was. A number in a start-up log disagreed with a number in a configuration file, and the test suite checked for successful status codes rather than for whether the answers made sense. Cloudflare says so itself: “This is the check we did not enforce.” For anyone buying hosted AI, the lesson is that tenant isolation in a model service is not only about accounts and permissions. It also depends on how the inference engine manages GPU memory that many customers share.

Cloudflare notes that cache pages are not wiped between requests, because doing so would cost speed, and that this is common across inference engines. That puts the burden on the engine’s bookkeeping, and the company describes this bug as one of a wider class of cache memory-safety problems it has been working on for months. It had built a prototype that tags each memory page to catch exactly this kind of misread, at a cost of under 1% in throughput, tested it in mid-August and disabled it shortly afterwards because of reliability issues.

The disclosure timetable deserves scrutiny too

The fix took a day. The public explanation took seven weeks, while upstream patches were merged and released. That is a normal pattern for coordinated disclosure, and the detail Cloudflare has now published is unusually thorough. But customers who sent long prompts to GLM-5.3 in that window have learned only now that fragments of their data could have appeared in someone else’s reply. Neither post says whether affected customers were contacted directly, and that is the question a data protection officer would ask first.

The exposure also shows how model hosting has become a software supply chain of its own. A bug in an open-source engine, inherited by a large provider, is a close cousin of the risks behind the poisoned LiteLLM release on PyPI and the Gheist campaign that used stolen GitHub tokens, even though nobody attacked anything here. GLM-5.3 is also one of the models Cloudflare uses in its own demonstrations, including the claims pipeline it built around Clairic last week.

Cloudflare now says it wants to use formal verification to prove that a bug like this “cannot occur again”, while admitting it expects gaps in what the proofs can show. The next document to read is Knostic’s own report, which should show how the leak looked from the outside.