Smaller, faster, safer: running Kimi and GLM at scale

Workers AI runs conclusion for immoderate of the champion unfastened models successful the world connected GPUs successful Cloudflare information centers adjacent to your users. Two of the astir capable, and astir demanding, are Moonshot's Kimi K-series and Z.ai's GLM. They are large, long-context, mixture-of-experts models, and they are awesome to use. They are besides very difficult to service efficiently because of representation constraints.

We've written earlier astir really we serve ample models connected Workers AI and astir separating the prefill and decode phases of conclusion to get much retired of each GPU. This station looks astatine 3 techniques we furniture connected apical of that to fresh these models into representation and support them fast: quantizing the KV cache, compressing the exemplary weights, and, because some of those battalion much requests onto shared hardware, protecting the cache those requests share. These optimizations alteration america to support much customers astatine little costs, pinch nary alteration successful exemplary accuracy.

All our experiments and accumulation postulation are moving and benchmarked pinch SGLang, an open-source conclusion serving framework. We recovered that SGLang offers the champion capacity successful the market, and we activity intimately pinch the SGLang squad to upstream patches and caller features to make our activity disposable to the open-source community.

Quantizing the KV cache

As a exemplary generates text, it stores the attraction keys (K) and values (V) for each token it has already processed successful a building called the KV cache. The cache is what lets the exemplary widen a agelong speech without re-reading the full discourse connected each caller token. For a long-context model, it grows quickly, and it is usually the KV cache, not the model's weights, that fills up GPU representation first.

By default, the cache is stored successful 16-bit precision (BF16). We shop it successful 8-bit floating constituent alternatively (FP8, e4m3), which halves its size. On Kimi K2.6, that raises the magnitude of discourse we tin clasp successful representation from astir 686,000 tokens to astir 1.37 million, doubly arsenic much.

It's worthy being precise astir wherever the use comes from, because it isn't earthy speed. Quantizing the cache adds a mini magnitude of activity per token, since the FP8 attraction kernel has to person values arsenic it sounds them. What it changes is really galore requests we tin support resident astatine once. The pursuing measurements are for Kimi K2.6 decoding connected a disaggregated H200 deployment, comparing the attraction kernels directly:

At immoderate azygous concurrency level, BF16 is simply a fewer percent faster per token. But BF16 runs retired of cache astatine 32 concurrent requests and can't admit a 33rd, while FP8 keeps going to 64 and reaches 2,192 tokens per second, astir 41% higher than BF16's peak, for astir 30% little costs per token. Because we tally prefill and decode arsenic abstracted pools, we tin use this wherever it helps most: prefill is compute-bound alternatively than memory-bound, truthful location we time off the cache successful BF16 and support its somewhat higher throughput.

None of this would matter if it changed the model's answers, truthful we checked. Across our information suite, FP8 and BF16 caches are indistinguishable:

Compressing the exemplary weights

The KV cache is 1 request connected GPU memory; the model's weights are the other. For GLM 5.2, we compress the weights from 8-bit floating constituent down to 4-bit integers (INT4) pinch nary nonaccomplishment successful accuracy. The checkpoint shrinks from 705 GB to 421 GB, astir 40%, and per-GPU representation crossed an 8-way tensor-parallel deployment drops from astir 88 GB to 52 GB, which leaves room for astir 1.18 cardinal tokens of KV cache connected the aforesaid hardware.

Across our information suite, INT4 and FP8 weights are indistinguishable:

Smaller weights make the decode shape faster, and for a clear reason: generating each token intends streaming the model's weights retired of GPU memory, truthful decode velocity is constricted by representation bandwidth. Move little information and each token arrives sooner. The effect is largest astatine debased concurrency, wherever per-request latency matters most:

Prefill behaves differently. It is compute-bound, and INT4 weights person to beryllium expanded backmost retired earlier the exemplary tin multiply pinch them, truthful that other measurement makes prefill slower alternatively than faster, GLM sustains astir 10,160 tokens per 2nd of prefill successful FP8 versus 8,660 successful INT4. As pinch the KV cache, the disaggregated creation turns this into a prime alternatively than a compromise: we tally INT4 for decode, wherever it wins, and FP8 for prefill, wherever it wins. Model accuracy stays wrong 0.8 points of the FP8 exemplary crossed each benchmark we run, making its value indistinguishable.

Protecting a shared KV cache

Both techniques supra person the aforesaid effect: they fto galore much requests stock 1 GPU's representation astatine the aforesaid time. That ratio is the full point, but it besides intends hundreds of requests are reference and penning pages of the aforesaid beingness KV cache. The mechanisms that make this fast, paged attention, continuous batching, cache reuse, each trust connected getting the bookkeeping precisely right, and astatine our petition volumes, moreover a one-in-a-billion correction would show up regularly.

So we built KV cache integrity checking arsenic a furniture of defense. The thought is straightforward: each beingness cache page gets a tag that changes whenever the page is reallocated, and the server records which pages and tags each petition expects to use. Before supported decode operations publication from the cache, those mappings are checked. If thing doesn't match, the affected petition is aborted alternatively than allowed to return information from the incorrect page.

The mobility that decides whether a information cheque ships is what it costs. We measured it connected a mid-sized accumulation exemplary successful a two-prefill, two-decode configuration, pinch 8,192-token inputs and 1,000-token outputs:

The costs is nether 1% connected some throughput and tail latency, and moreover the precocious bound of the 95% assurance interval stays adjacent 1%. We kept it computationally inexpensive by moving the validation arsenic a abstracted batch cheque alternatively than fusing it into the attraction kernel, which would person introduced a title betwixt GPU thread groups. It's enabled per deployment, and the default way uses a no-op locator pinch nary measurable overhead, truthful deployments that don't request it salary nothing.

What's next

Serving frontier models efficiently is simply a moving target, and this is the ongoing activity down it. We're expanding FP8 KV caches crossed much of the fleet, validating NVFP4 weights connected Blackwell (NVIDIA’s GPU architecture), and moving toward making integrity checks thing we tin time off connected everyplace astatine negligible cost. These optimizations will let america to proceed to support much customers astatine a little costs and astatine the aforesaid accuracy.

If squeezing the champion unfastened models onto GPUs and serving them to millions of developers sounds for illustration your benignant of problem, come activity pinch us.