Prefill/Decode Disaggregation: Why Production LLM Inference Is Splitting Onto Separate Hardware

Beyond the prefill tax: what it intends to tally prefill and decode connected abstracted GPU pools, who’s doing it successful production, and erstwhile it’s worthy the complexity.

Imagine a edifice room astatine 8 PM connected a Saturday. One cook is 20 minutes into chopping vegetables for a twelve-course tasting menu, while 30 tables that ordered an hr agone watch their half-finished plates get acold each clip the cook turns backmost to the chopping board. No existent room runs this way: prep cooks grip the chopping successful the back, statement cooks grip the plating up front. Two different jobs, 2 different stations.

And yet this is really astir teams service ample connection models today. One GPU does some the dense prep activity (reading and processing your prompt) and the delicate plating (generating the response, 1 connection astatine a time). Every large prep occupation makes everyone else’s consequence late.

That’s starting to change. Some of the astir precocious conclusion systems successful production, astatine Moonshot AI, DeepSeek, and wrong NVIDIA’s newest serving stack, person softly made a very strategical determination to divided the 2 jobs onto physically abstracted hardware. This portion explains why, really it works, and erstwhile you shouldn’t bother.

Prefill/decode disaggregation runs each conclusion shape connected its ain GPU pool

Prefill/decode disaggregation is simply a serving architecture that runs the 2 phases of LLM conclusion connected abstracted GPU pools. Prefill (processing the input prompt) runs connected 1 group of GPUs, and decode (generating the consequence token by token) runs connected another. When prefill finishes, the model’s representation of the punctual (the KV cache) is sent complete the web to a decode GPU, which past generates the response.

The divided exists because the 2 phases accent different hardware limits: prefill is bound by compute, decode by representation bandwidth. Separating them stops agelong prompts from interfering pinch everyone else’s token generation. The value is other hardware, web transfers, and operational complexity. Production systems utilizing it coming see Moonshot AI’s Mooncake, DeepSeek’s conclusion clusters, and NVIDIA’s Dynamo framework.

Prefill and decode compete for other GPU resources

We’ve written earlier astir the prefill tax, but here’s the short type successful lawsuit you’re arriving fresh. Every LLM petition has 2 phases: the exemplary first sounds and processes your full punctual (prefill), past starts generating its reply 1 token astatine a clip (decode). That first shape is the tax. It’s latency you salary earlier a azygous connection comes back, and it grows pinch each token successful your prompt.

The deeper problem, the 1 the prefill taxation only hints at, is that the 2 phases want wholly different things from your hardware.

Prefill happens erstwhile the exemplary sounds your prompt. It processes each token astatine once, successful parallel, successful 1 large burst of matrix math. GPUs emotion this; it’s precisely what they were built for. The compute cores tally hot, the representation strategy is hardly stressed, and the occupation finishes accelerated comparative to its size.

Decode happens erstwhile the exemplary writes its answer. It generates 1 token astatine a time, and for each caller token, the GPU has to re-read a immense magnitude of stored data: the model’s weights, positive everything it remembers astir the speech truthful acold (the KV cache). The computation for each token is small. Most of the clip is spent moving information to and from memory, while the GPU’s compute cores beryllium mostly idle.

So the 2 phases request very different things 1 starves for compute, the different for representation bandwidth and we’ve been asking a azygous GPU to present both.

Here is really it useful successful existent life. Modern serving engines batch galore users’ requests together connected the aforesaid GPU excavation to support utilization high. Decode steps are mini and quick, truthful the GPU cycles done everyone’s adjacent token complete and over, and things consciousness smooth. Then a petition shows up pinch a agelong prompt.

Say personification pastes a 60-page statement into your document-analysis app. The GPU drops everything to process that agelong prompt, and each different personification has to wait. Their tokens conscionable extremity streaming. The serving world calls this head-of-line blocking, and it’s why a chatbot that feels accelerated successful testing becomes slow successful accumulation the infinitesimal existent postulation shows up pinch its messy operation of short questions and elephantine pasted documents.

This isn’t an separator case; it’s the default style of accumulation traffic. It’s arguably the unfastened problem successful conclusion serving correct now, and each squad moving LLMs astatine standard hits it. The package fixes group scope for first (chunked prefill, smarter schedulers, privilege queues) soften the blow, but they each tally into the aforesaid ceiling: arsenic agelong arsenic prefill and decode compete for the aforesaid GPU, 1 of them loses.

The hole is amazingly simple. Stop making them share. Disaggregated serving (you’ll besides spot it called P/D disaggregation) splits conclusion crossed 2 abstracted pools of GPUs:

• A prefill pool that does thing but process incoming prompts. The prep kitchen.
• A decode pool that does thing but make tokens. The line.

A petition arrives, a prefill GPU processes the punctual astatine afloat speed, and the petition hands disconnected to a decode GPU that streams the consequence retired token by token. Long prompts nary longer stall anyone’s generation, because the machines doing procreation ne'er spot a prompt. Each excavation tin besides beryllium sized, scheduled, and moreover hardware-selected for its existent job: much earthy compute for prefill, much representation bandwidth for decode.

There’s a catch, though, and it’s the portion that makes this an engineering problem alternatively than a config flag.

When the exemplary finishes reference your prompt, it has built up a moving representation of everything it read. That’s the KV cache. In a accepted setup, this representation conscionable sits connected the aforesaid GPU that will make the response. In a disaggregated setup, it has to physically recreation crossed the web from the prefill instrumentality to the decode instrumentality earlier a azygous token tin beryllium generated. For a agelong punctual connected a ample model, the KV cache tin tally into the tens of gigabytes. Moving it is not a rounding error. It’s the caller bottleneck you’ve signed up to manage, and immoderate writeup that treats the transportation arsenic a solved broadside item is trading you something.

It’s worthy being clear astir what disaggregation gives up. The modular attack today, what you get pinch vanilla vLLM connected a unified GPU pool, is continuous batching: each GPU handles some phases, and the scheduler weaves caller prompts and ongoing generations together arsenic tightly arsenic it can. It’s simple, it’s mature, and for galore workloads it’s excellent. Every GPU stays busy, thing crosses a network, and there’s 1 strategy to run alternatively of 3 (prefill pool, decode pool, and the transportation furniture betwixt them). Disaggregation deliberately trades that simplicity for predictable latency. Whether the waste and acquisition is worthy it depends wholly connected your workload, which we’ll get to.

Mooncake, DeepSeek, and NVIDIA Dynamo person shipped disaggregation successful production

This isn’t a whiteboard idea. Several existent systems person shipped it, and they’re worthy knowing by name, partially because they’re the reference points for the full section and partially because their differences show you wherever the difficult parts are.

DistServe (Zhong et al., OSDI 2024). The world insubstantial that introduced the thought explains it astir clearly. The halfway insight: erstwhile prefill and decode stock a GPU, you can’t optimize time-to-first-token and time-per-generated-token independently. Improving 1 degrades the other. Separate them and you tin tune each excavation for its ain latency goal. DistServe reported serving respective times much requests wrong the aforesaid latency targets compared to colocated serving.

Mooncake is the serving level down Kimi, the adjunct from Moonshot AI, and it’s the astir compelling accumulation grounds to date. Mooncake goes further than conscionable splitting the pools. It treats the KV cache itself arsenic the halfway of the full design, pooling spare CPU representation and SSD retention crossed the cluster into a elephantine shared cache, truthful repeated aliases overlapping prompts don’t request to beryllium prefilled from scratch astatine all. Moonshot published the architecture and reported that it is handling existent postulation astatine a superior scale. This is not a laboratory demo.

DeepSeek disclosed successful its published method report that accumulation serving for DeepSeek-V3 and R1 runs prefill and decode connected abstracted GPU clusters, pinch different parallelism strategies for each phase. When 1 of the astir cost-efficient conclusion operations successful the world tells you really it serves its ain models, that’s worthy reference twice.

NVIDIA Dynamo is the clearest awesome that this is going mainstream. Dynamo is NVIDIA’s open-source distributed serving framework, the successor to Triton for LLM workloads, and disaggregated serving is its header characteristic alternatively than a footnote. It sits supra engines for illustration vLLM, TensorRT-LLM, and SGLang, routes prefill and decode to dedicated worker pools, and ships a dedicated transportation room (NIXL) whose full occupation is moving KV caches betwixt machines arsenic accelerated arsenic the hardware allows. NVIDIA has reported throughput gains of up to respective times connected ample models nether latency constraints. vLLM itself has been building autochthonal prefill/decode disaggregation support too, truthful the capacity is arriving successful the default open-source stack.

These frameworks are still comparatively new, and location isn’t overmuch applicable guidance disposable yet. Most of the teams utilizing disaggregated conclusion successful accumulation are ample organizations building their ain infrastructure and scheduling systems. That’s besides why you won’t find galore elaborate articles from unreality providers, galore are still exploring aliases rolling retired these capabilities.

The hardest portion is this: erstwhile you disaggregate, interconnect bandwidth decides whether the full point works. The KV cache for a azygous long-context petition connected a 70B-class exemplary tin scope tens of gigabytes. Push that complete mean datacenter Ethernet and the transportation takes agelong capable to swipe retired everything you gained by splitting the phases. Push it complete NVLink, InfiniBand, aliases high-bandwidth RDMA-capable Ethernet and the handoff tin hide wrong clip the decode GPU was going to walk anyway. This is why each superior strategy connected the database supra invests heavy successful the transportation furniture (Mooncake’s cache-centric design, Dynamo’s NIXL), and why “just divided the pools” without reasoning astir the web successful betwixt usually produces a slower strategy than the 1 you started with.

Disaggregation isn’t ever the correct choice

Many articles make disaggregated conclusion sound for illustration the evident adjacent step. The reality is much nuanced.

First, it tin require much hardware. In a accepted setup, each GPU tin grip some prefill and decode work, truthful resources are shared much efficiently. With disaggregation, immoderate GPUs are dedicated to prefill while others only grip decoding. If your workload changes passim the day, for example, agelong documents successful the greeting and short chat requests successful the evening, 1 group of GPUs whitethorn beryllium idle while the different becomes overloaded. That intends you whitethorn extremity up paying for other capacity that isn’t ever being used.

Second, it makes the strategy much complex. Instead of a petition staying connected 1 instrumentality from commencement to finish, it now moves betwixt aggregate machines. That creates much opportunities for things to spell wrong, specified arsenic web delays, overloaded decode workers, aliases requests getting interrupted if a instrumentality fails. As a result, monitoring, debugging, and operating the strategy becomes much challenging.

Finally, the benefits only look astatine very ample scale. If you’re serving a mini aliases medium-sized application, disaggregation often won’t make a noticeable difference. Modern conclusion engines for illustration vLLM already usage techniques specified arsenic continuous batching and chunked prefill to amended GPU utilization without splitting the workload crossed abstracted GPU pools. For galore teams, these optimizations supply astir of the capacity gains while keeping the strategy overmuch simpler.

Our position is straightforward: disaggregated conclusion is an important guidance for the early of large-scale AI serving, particularly for organizations moving thousands of concurrent requests. But that doesn’t mean each squad should adopt it today. For galore deployments, a well-optimized unified conclusion strategy is easier to operate, costs less, and delivers fantabulous performance. The champion attack is to understand erstwhile disaggregation solves a existent problem and only present it erstwhile you’ve reached that point.

On DigitalOcean hardware, the divided maps to H100 prefill pools and H200 decode pools

A speedy statement earlier we start: this conception describes really a disaggregated setup could activity connected DigitalOcean infrastructure. These are not benchmark results, and this is not a DigitalOcean product.

Conceptually, the mapping is clean. DigitalOcean’s GPU Droplets connection H100 and H200 configurations. The H100 delivers the earthy compute that prefill uses. The H200 pairs akin compute pinch substantially much and faster representation — 141 GB of HBM3e astatine astir 4.8 TB/s versus the H100’s 80 GB astatine astir 3.35 TB/s — which is the floor plan decode wants: much room for KV caches, faster sounds per generated token. A prefill excavation connected H100s feeding a decode excavation connected H200s is simply a workable design, and honestly, it’s the textbook illustration of why heterogeneous pools are 1 of disaggregation’s astir absorbing promises.

The determining factor, arsenic always, is the ligament betwixt the pools. DigitalOcean’s multi-node GPU configurations link 8-GPU nodes complete a dedicated high-speed cloth pinch RDMA support, and Bare Metal GPU offerings database east-west bandwidth up to 400 Gbps pinch GPU interconnect speeds up to 3.2 Tbps — the people of networking that KV cache transportation needs. (DigitalOcean’s ain serving level has begun utilizing this pattern: the Inference Tax post describes disaggregated serving complete a high-speed RoCE web arsenic portion of the Serverless Inference stack.) The back-of-envelope mathematics immoderate designer should run: return your emblematic KV cache size per petition (it grows pinch discourse length, and pinch the model’s size and architecture — a 70B-class exemplary stores astir a 3rd of a megabyte per token), disagreement by realistic sustained bandwidth betwixt nodes, and comparison the consequence against your time-to-first-token budget. If the transportation fits comfortably wrong the latency you were already paying for prefill, disaggregation has room to work. If it doesn’t, nary orchestration model will prevention you. Frameworks for illustration vLLM’s disaggregated mode aliases NVIDIA Dynamo are the earthy package furniture connected top; some are unfastened root and some tally connected modular NVIDIA hardware of precisely this class.

Three workload signals find whether disaggregation pays off

Disaggregated conclusion isn’t thing each AI exertion needs. Three properties of your workload do astir of the deciding.

Traffic volume: the triumph appears erstwhile GPUs are saturated

The biggest advantage of disaggregation appears erstwhile your GPUs are perpetually busy.

If you’re moving a mini deployment pinch only a fewer GPUs and requests get occasionally, a accepted setup, wherever each GPU handles some prefill and decode, is usually the amended choice. It’s simpler, cheaper, and easier to manage.

However, if you’re serving thousands of users astatine the aforesaid clip and your GPUs are ever processing requests, prefill and decode commencement competing for GPU resources. In that case, separating them into dedicated GPU pools tin amended wide throughput and trim latency.

Latency consistency: interactive applications consciousness the interference most

Not each exertion needs responses successful a fraction of a second.

For workloads specified arsenic batch processing, archive analysis, aliases soul tools, users usually don’t announcement if 1 petition takes a small longer than another. Small delays aren’t a awesome concern.

But for interactive applications, specified arsenic chatbots, AI coding assistants, aliases sound assistants, users expect responses to statesman almost instantly and proceed streaming smoothly. A azygous agelong punctual tin hold each different petition sharing the aforesaid GPU, creating an inconsistent personification experience.

If predictable latency is simply a priority, disaggregation becomes overmuch much valuable.

Prompt length: long-context workloads summation the most

Prompt magnitude is often the deciding factor.

If your exertion mostly handles short prompts, the prefill shape is quick, truthful there’s small use successful separating it from decoding.

On the different hand, applications that process agelong documents, execute Retrieval-Augmented Generation (RAG), aliases support lengthy speech histories walk overmuch much clip successful prefill. Since punctual lengths tin alteration dramatically from 1 petition to another, these workloads use the astir from disaggregation.

If your workload sits location successful the mediate — existent traffic, mean latency sensitivity, mixed punctual lengths — effort the simpler optimizations first. Chunked prefill, amended scheduling, and continuous batching often present astir of the betterment without requiring you to redesign your serving architecture. Disaggregation is the measurement you return erstwhile those tally retired of headroom connected each 3 signals astatine once.

One optimization among many

Disaggregated conclusion isn’t a metallic bullet. It’s conscionable 1 method for improving LLM serving.

Other optimizations lick different bottlenecks. For example, speculative decoding speeds up the decode shape by generating tokens much efficiently, while quantization reduces representation usage and continuous batching improves GPU utilization.

In practice, accumulation conclusion systems harvester respective of these techniques alternatively than relying connected conscionable one. Think of disaggregation arsenic different instrumentality successful your optimization toolbox — 1 that’s particularly useful for large, latency-sensitive workloads, but unnecessary for galore smaller deployments.

References

1. Zhong, Y. et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024.
2. Qin, R. et al. (2024). Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. Moonshot AI.
3. DeepSeek-AI (2024). DeepSeek-V3 Technical Report — Section 3.4 covers the disaggregated prefill/decode deployment.
4. NVIDIA (2025). Introducing NVIDIA Dynamo: A Low-Latency Distributed Inference Framework. NVIDIA Technical Blog.
5. NVIDIA. Dynamo disaggregated serving documentation.
6. DigitalOcean (2026). The Inference Tax: How Prefix-Aware Routing Eliminates the Hidden Cost of LLMs astatine Scale.
7. DigitalOcean. GPU Droplets · What is the NVIDIA H200 · Multi-node GPU configuration docs · Bare Metal GPU features.