Continuous Batching Mechanics: The Scheduler Behind vLLM, TGI, and SGLang

Introduction

Serving a ample connection exemplary is fundamentally a scheduling problem disguised arsenic a hardware problem. A modern GPU is simply a throughput machine. It wants thousands of arithmetic operations astatine once. But the requests that a work receives mostly get 1 astatine a time, astatine unpredictable moments, pinch prompts and responses of very different lengths. The occupation of an conclusion scheduler is to negociate a instrumentality that excels astatine bulk activity pinch a workload that trickles successful irregularly.

This article starts by explaining really a azygous petition connected a azygous GPU works, past scales it the evident way, batching galore requests together, showing really it tin break nether existent traffic. Then it explains continuous batching, the scheduler creation that powers vLLM, Text Generation Inference (TGI), SGLang, and fundamentally each superior conclusion strategy today. Finally, it looks astatine wherever those systems really disagree and why.

Key Takeaways

• Continuous batching updates the batch each decoding step alternatively than erstwhile per batch. This eliminates the head-of-line blocking and idle slots that make naive fixed batching inefficient, and it is the scheduler creation shared by vLLM, TGI, and SGLang.

• The state to admit activity each measurement requires 2 supporting mechanisms. Preemption (recomputation aliases swapping) handles the lawsuit wherever the KV cache runs retired mid-generation, and paged KV-cache management (PagedAttention) makes that representation agelong overmuch further by eliminating fragmentation.

• Every awesome strategy shares this foundation, and progressively a shared toolkit. They each look the aforesaid difficult problem, integrating a caller request’s compute-heavy prefill into the watercourse of ongoing decodes, and the main techniques for it (chunked prefill, prefix caching, paged memory) are now communal to vLLM, TGI, and SGLang alike. The differences are mostly successful accent and defaults.

A Single Request connected a GPU

Generating matter pinch a transformer happens successful 2 phases. The first shape is prefill. The exemplary ingests the full punctual successful a azygous guardant pass, computing attraction crossed each its tokens astatine erstwhile and building up the key-value (KV) cache, the stored keys and values for each token that later tokens will be to. Prefill is compute-bound. There is simply a batch of parallel mathematics to do, and the GPU’s arithmetic units are the bottleneck. The 2nd shape is decode. The exemplary generates the consequence 1 token astatine a time, and each caller token requires its ain guardant pass. Decode is memory-bound. Each measurement does only a sliver of computation but must watercourse the model’s full weight group and the increasing KV cache done representation to do it.

The KV cache is the assets that will predominate each determination later successful this article. Every token the exemplary processes, either the punctual aliases generated, adds an introduction to the cache, and that introduction lives successful GPU representation for arsenic agelong arsenic the petition is active. A agelong speech intends a ample cache. Many concurrent requests mean galore caches competing for the aforesaid finite excavation of memory. The cache is what lets the exemplary debar recomputing attraction complete the full series astatine each step, but it is besides what makes memory, not compute, the point you usually tally retired of first.

When moving 1 petition astatine a time, the GPU is almost idle during decode. A azygous decoding measurement moves gigabytes of weights done the spot to nutrient 1 token’s worthy of arithmetic, leaving the compute units mostly unused.

The Obvious Way to Scale: Static Batching

If 1 petition wastes the GPU during decode, the intuitive hole is to tally galore requests together. Because decode is memory-bound, this useful well. Once you person paid to watercourse the model’s weights done the spot for 1 sequence, adding much sequences to the aforesaid measurement is astir free. This is static batching (sometimes called request-level batching).

It collects a group of N requests and pads each series to the magnitude of the longest 1 because tensors request a azygous shape. It past runs guardant passes complete the full batch, measurement by step, until each series successful it has vanished generating. Then it returns each N responses together and starts complete pinch the adjacent group.

But announcement that erstwhile the batch starts, it commits to a fixed group of requests, processed together, from commencement to finish. That azygous decision, freezing the batch’s “membership” for its full lifetime, is wherever each problem pinch fixed batching comes from.

Where Static Batching Breaks

The first nonaccomplishment is head-of-line blocking. A fixed batch runs until its longest personnel finishes. If a petition that needs 10 tokens is unlucky capable to stock a batch pinch 1 that needs 500, the short petition is held. It vanished agelong agone but cannot beryllium returned until the full batch drains. Latency for short requests becomes hostage to the worst-case magnitude successful their batch.

The 2nd and 3rd failures compound this. As sequences decorativeness early, their slots successful the batch spell empty, but they enactment allocated, truthful you walk the remainder of the batch’s life computing connected a half-empty tensor. Then location is no mid-flight admission. A petition that arrives 1 measurement aft the batch starts must hold for the full batch to decorativeness earlier it tin begin. Under dependable traffic, this produces latency spikes.

The guidelines origin underneath each 3 problems is that fixed batching makes its scheduling determination once per batch, but the workload changes every token.

The Fix: Continuous Batching

The fix, introduced by the Orca strategy successful 2022, is to move the scheduling determination from the batch to the individual token. Instead of picking a batch and moving it to completion, the scheduler runs before each azygous guardant pass and re-decides which requests are successful the batch. The batch is nary longer a stiff group. It is simply a surviving roster that changes pinch each token. This is continuous batching, besides called iteration-level aliases in-flight batching.

The motor runs a loop. The scheduler picks the existent batch, the exemplary runs 1 guardant walk complete it, the sampled tokens are appended to each sequence, and the scheduler updates its bookkeeping, retiring immoderate petition that conscionable emitted an end-of-sequence token and instantly freeing its KV cache. Because that freed capacity is disposable astatine the very adjacent step, waiting requests tin beryllium admitted correct away. Head-of-line blocking disappears, because a short petition leaves the instant it is done alternatively than waiting for its batch-mates. Idle slots disappear, because freed capacity is backfilled immediately. And the mid-flight admittance problem disappears, because a recently arrived petition joins wrong a measurement aliases 2 alternatively of waiting for a full batch to drain.

Continuous batching introduces 2 caller problems. First, a recently admitted petition cannot simply commencement decoding. Its punctual must beryllium prefilled first, a compute-heavy guardant walk that tin stall everyone else’s decoding if you are not careful. Second, because you are free to admit requests each step, you tin overcommit the KV cache and tally retired of representation mid-generation.

Interleaving Prefill and Decode

A caller request’s prefill is simply a big, compute-bound burst, while the decode steps keeping existing requests live are a watercourse of small, memory-bound updates. Each motor has to fresh that burst into the ongoing decode watercourse without stalling it.

The 3 answers to this problem are prefill-first, chunked prefill, and disaggregation. Prefill-first pauses the decodes to tally the prefill. It is simple, but it produces a visible hitch successful each different request’s output whenever a caller 1 arrives. Chunked prefill slices the agelong prefill into smaller pieces and weaves them into the decode steps, truthful the caller petition makes advancement without immoderate azygous measurement being dominated by its prefill. Disaggregation goes further, moving prefill and decode connected wholly abstracted pools of GPUs and transferring the KV cache betwixt them, truthful the 2 phases ne'er compete astatine all.

Chunked prefill has go the default successful astir modern engines because it smooths latency without other hardware. Disaggregation is simply a newer, heavier attack that pays disconnected chiefly astatine very ample scale.

Running Out of KV-Cache Memory

The 2nd problem is memory. Because continuous batching keeps admitting requests arsenic agelong arsenic location is room, and because each progressive request’s KV cache grows pinch each token it generates, a batch that fresh a infinitesimal agone tin overflow the GPU’s representation pool. To lick this, you request to usage the representation you person acold much efficiently, and, erstwhile that is not enough, return immoderate back.

Early systems gave each petition a azygous contiguous slab of representation sized for its maximum imaginable length, which fragmented the excavation and near immoderate of it wasted connected over-reservation and gaps. PagedAttention, introduced by vLLM, borrows the operating-system thought of virtual representation and paging. The KV cache is surgery into mini fixed-size blocks, and a request’s logically contiguous series of blocks is mapped, done a per-request block table, onto beingness blocks that tin beryllium anyplace successful GPU memory. The petition sees a clean, contiguous view, but the beingness reality is scattered blocks mixed pinch those of different requests and pinch free space. This eliminates fragmentation, lets representation beryllium allocated connected request arsenic sequences grow, and makes some preemption and prefix sharing cheap.

Paged representation stretches the excavation a agelong way, but nether capable load it tin still fill. When it does, the scheduler needs the authority to return representation back, evicting aliases freezing a petition it antecedently admitted truthful the remainder tin continue. This is called preemption, and location are 2 ways to do it. Recomputation drops the request’s KV cache to free representation instantly, past rebuilds it later by re-running prefill: inexpensive connected memory, but paying again successful compute. Swapping copies the KV blocks retired to CPU representation and backmost erstwhile the petition resumes, preserving the computation but paying successful Peripheral Component Interconnect Express (PCIe) bandwidth and big RAM. vLLM defaults to recomputation, but some are wide used. Which petition to preempt is simply a argumentation choice. Engines typically evict the newest aliases lowest-priority requests first and reconstruct them erstwhile representation frees up.

How the Engines Differ

The techniques above, chunked prefill, prefix caching, and paged memory, person been mostly mixed into a shared toolkit that the mature engines each implement. What still differs is little astir which techniques an motor has and much astir wherever each thought originated, what each motor emphasizes by default, and really it is built.

vLLM is the memory-first baseline and the reference implementation astir others are measured against. PagedAttention is its signature contribution, and its scheduler shows the aforesaid lineage. The older V0 motor utilized the prefill-first approach, pinch the latency hitches that travel pinch it, while the rearchitected V1 motor makes chunked prefill the default and simply decides really galore tokens each petition advances per step.

TGI (Hugging Face’s Text Generation Inference) has the aforesaid halfway techniques, chunked prefill and prefix caching included, but its unique prime is structural. It splits serving into 2 parts. First, a accelerated router, written successful Rust, handles petition queuing and each the batching decisions. Second, abstracted workers do thing but tally the model’s guardant passes. Its scheduling knobs, specified arsenic max_batch_total_tokens and waiting_served_ratio, unrecorded successful that router and power really aggressively it fills each batch. Hugging Face has leaned particularly difficult connected prefix caching, reporting that TGI’s handling of it makes the server beardown connected very long, repeated prompts. The root present is accumulation serving.

SGLang pushed prefix reuse the furthest. Its RadixAttention organizes cached prefixes successful a character truthful that requests sharing a communal prefix, specified arsenic a strategy prompt, a artifact of few-shot examples, aliases a group of instrumentality definitions, reuse each other’s KV cache alternatively of recomputing it. Prefix caching is communal crossed engines now, but SGLang’s radix-tree type was an early and influential shape of it, and the motor remains particularly beardown connected workloads pinch dense punctual overlap, specified arsenic agents and structured-generation pipelines. Vendor-optimized engines push successful different guidance entirely. NVIDIA’s TensorRT-LLM compiles the exemplary up of clip and does “in-flight batching,” winning connected earthy velocity astatine the costs of flexibility.

Common Questions

Is continuous batching the aforesaid point arsenic PagedAttention?

No. Continuous batching is simply a scheduling method (which requests tally each step). PagedAttention is simply a memory-management method (how the KV cache is stored). They are complementary. PagedAttention is what makes the representation spell acold capable for continuous batching to admit galore requests astatine once, but you tin instrumentality 1 without the other.

Does continuous batching wounded per-request latency?

It can, indirectly. Continuous batching improves throughput and keeps the GPU full, but larger, fuller batches mean each measurement does much activity and representation unit rises. Tuning the batch-size and token budgets, and utilizing chunked prefill to debar prefill stalls, is really you equilibrium throughput against time-to-first-token and inter-token latency.

Conclusion

The single-GPU type of this problem is mostly solved. Continuous batching keeps the hardware busy, paged representation keeps the cache from overflowing, and the awesome engines person settled connected that design. Most of the existent activity has moved outward, crossed galore GPUs, and downward, into the decode measurement itself.

Outward intends moving prefill and decode connected abstracted pools of hardware (as successful NVIDIA’s Dynamo) and treating the KV cache arsenic a gradual assets shared crossed requests and machines. The downward push attacks the decode bottleneck directly. Techniques for illustration speculative decoding and KV-cache quantization compression much useful activity retired of each memory-bound step.

• The LLM Inference Optimization: Quantization to Speculative Decoding Part 1
• The MoE-ification of the Open Model Ecosystem, and What It Means for Your Inference Bill
• Continuous Batching vs. Static Batching successful LLM Inference