Why your vLLM p99 latency blows up in production and how chunked prefill and scheduling fix it

Here is simply a point that has happened to maine much than once. I benchmark a model, the numbers look great, p50 latency is low, tokens per 2nd is high. Then existent postulation shows up and the p99 latency is abruptly 5 to 10 times worse than thing I measured. Same exemplary and aforesaid GPU. So what happened? The benchmark was excessively clean.

Most load tests nonstop requests astatine a dependable pace, each astir the aforesaid size. Real postulation does not do that. It comes successful bursts, it mixes mini prompts pinch immense ones, and it shares the GPU crossed a batch of users astatine once. Under that benignant of load, 1 point hurts your tail much than earthy compute ever will: a agelong prefill blocking the decodes that are already running.

When a large punctual starts its prefill, the token procreation for everyone other tin get stuck waiting down it. The GPU is stuck successful a scheduling problem. So let’s locomotion done why this happens, why continuous batching does not afloat hole it, and really chunked prefill and amended scheduling get your tail backmost nether control.

If you want the groundwork first, publication the Continuous Batching Mechanics article that explains the scheduler that each of this sits connected apical of.

Continuous batching helped, but it did not decorativeness the job!

If you service LLMs astatine all, you person already moved beyond static request-level batching. With fixed (request-level) batching, the server groups a batch, runs it, and does not fto anyone time off until the slowest petition successful the group is done. Fast requests conscionable beryllium location waiting for slow ones. It wastes GPU clip and it wrecks latency.

Continuous batching fixed a large chunk of that. Instead of locking successful 1 group of requests for the full run, the scheduler re-decides the batch earlier each step. Two things travel from that. A petition that finishes leaves correct distant and frees its memory, alternatively of holding up everyone other until the batch drains. And a waiting petition tin beryllium pulled successful arsenic soon arsenic location is room, alternatively of sitting successful a queue until the existent batch is wholly done. This step-by-step scheduling is the thought from the Orca paper (Yu et al., 2022) that vLLM and astir modern servers usage today.

It leaves 1 spread open, and that spread is wherever your tail lives. Prefill and decode are 2 very different kinds of work:

• Prefill sounds the full punctual successful 1 guardant pass. It is compute-bound. Big matrix multiplies, precocious GPU use. A agelong punctual intends a long, dense prefill. Prefill is usually the biggest portion of your clip to first token (TTFT), but it is not the full story. TTFT is really the sum of a fewer delays: clip spent waiting successful the queue, scheduler overhead successful admitting the request, the prefill itself, and web information trips to and from the client.
• Decode generates 1 token astatine a time. It is memory-bandwidth-bound. It mostly shuffles weights and KV cache successful and retired of memory, and it needs predominant short turns to support inter-token latency (ITL) debased truthful the watercourse feels smooth.

Continuous batching still tends to tally a prefill arsenic 1 large unit. So erstwhile a agelong punctual lands successful the batch, that loop takes a while, and each decode already successful formation has to hold for it. Users who were happily streaming tokens abruptly deed a pause. That region is your p99. Mixing prefill and decode severely is wherever tail latency comes from, and it is simply a scheduling problem.

This is not a mini effect. The Sarathi-Serve squad (Agrawal et al., OSDI 2024) measured that a azygous prefill sharing the batch tin push inter-token latency up by arsenic overmuch arsenic 28x versus a decode-only batch. That is why a benchmark tin look good connected mean and still autumn isolated astatine the tail.

PagedAttention, scoped correctly

PagedAttention is astir KV cache memory. It is not a scheduler. The idea, from the vLLM insubstantial (Kwon et al., 2023), is to dainty the KV cache for illustration virtual representation successful an operating system. Instead of reserving 1 large artifact per sequence, it splits the cache into mini fixed-size blocks that tin beryllium anyplace successful GPU memory. That cuts the abstraction wasted connected fragmentation, lets you support much sequences successful representation astatine once, and makes it inexpensive to stock cache betwixt sequences that stock a prefix.

So wherever does latency travel in? Only indirectly. Before PagedAttention, representation was usually the first limit you ran into connected really galore sequences you could clasp successful a batch, because each 1 had to over-reserve cache abstraction for its worst-case length. Once representation stops being the point that boxes you in, the scheduler has a batch much room to determine really to operation prefills and decodes. PagedAttention does not schedule anything. It conscionable clears the runway truthful a bully scheduler tin do its job.

But clearing the representation constraint does not show the scheduler really to fresh a large prefill into the batch without stalling the decodes astir it. That is the nonstop problem chunked prefill exists to solve.

Chunked prefill: really it useful and wherever it costs you

Chunked prefill is the nonstop hole for the blocking problem. The thought is elemental to say. Instead of moving a agelong prefill each the measurement to the extremity successful 1 iteration, you divided it into smaller chunks of tokens. Then successful each loop you tally 1 prefill chunk alongside the decodes that are already going. The decodes support getting their turns, truthful the watercourse stays smooth, while the agelong punctual still makes dependable advancement successful the background.

This is the “stall-free” scheduling thought from the Sarathi-Serve activity (Agrawal et al.). In their tests, admitting prefills this measurement fto a server grip up to 2.6x much postulation wrong the aforesaid latency target for Mistral-7B connected a azygous A100, and up to 6.9x much for the overmuch larger Falcon-180B dispersed crossed 8 A100s, compared pinch Orca and vLLM.

The drawback is chunk size, and this is wherever a batch of contented stays excessively shallow. There is simply a existent tradeoff:

• Chunks excessively small: decodes enactment snappy, but the agelong punctual now needs galore iterations to finish, truthful its ain clip to first token goes up. You besides salary much per-iteration overhead and get little businesslike prefill.
• Chunks excessively large: prefill finishes fast, but you are correct backmost to the blocking you were trying to remove.

In vLLM you mostly power this done the per-iteration token fund (max_num_batched_tokens) pinch chunked prefill turned on. That fund is fundamentally your chunk size dial. It is besides the nonstop point the Sarathi-Serve authors floor plan per deployment alternatively than hardcode; their experiments usage budgets for illustration 512 and 2048 and tune to the latency target, which is simply a sensible scope to commencement testing from.

Chunked prefill is not the only reply to the prefill-versus-decode clash. There are 3 communal approaches, and it is worthy knowing wherever chunked prefill sits among them. Prefill-first conscionable pauses the decodes to tally the full prefill, which is elemental but causes the nonstop hitch shown above. Chunked prefill slices and interleaves, and it has go the default successful astir modern engines because it smooths latency without other hardware. Disaggregation goes further and runs prefill and decode connected abstracted pools of GPUs truthful the 2 phases ne'er compete astatine all, which pays disconnected chiefly astatine very ample scale. If you are connected a azygous node, chunked prefill is almost ever the correct tool.

One much point that usually gets missed: chunked prefill and PagedAttention interact. Chunking changes the representation entree shape of prefill, and it changes really your cache blocks capable up complete time. These 2 are almost ever explained connected their own, but successful a existent deployment they compound. If you tune your token fund without reasoning astir cache behavior, you are only seeing half the picture.

Scheduling policy: the choices that style your tail

Chunked prefill decides really the activity gets sliced. Scheduling argumentation decides whose activity runs first. Both style the tail, and the 2nd 1 is usually near connected its default.

Here are the communal policies and what they trade:

• FCFS (first come, first served): vLLM’s default. Simple and fair, but a burst of agelong prompts arriving together will still push everyone’s tail out, because thing says “this decode is much urgent than that prefill.”
• Priority-based: You tag requests truthful immoderate jump the queue. Handy erstwhile you person paid tiers aliases interactive postulation that must enactment fast, but it tin starve low-priority activity if you are not careful.
• SLA-aware: The scheduler tries to deed a latency target per petition alternatively of conscionable ordering them. More activity to run, but it is the only attack that goes consecutive astatine the number you really attraction about.

The different half of scheduling is preemption, and this is wherever the tail really lives. When representation gets tight, the server has to footwear retired a series to support going. vLLM tin bring it backmost 2 ways: recompute its prefill from scratch later, aliases switch its KV cache retired to CPU representation and back. Both are expensive. The cardinal constituent is that they are rare. And rare, costly events are precisely what group your p99. Your median petition ne'er gets preempted. The 1 that eats a immense punishment is the petition sitting astatine your tail. So if you want to power p99, you power who gets preempted and really often. That is simply a scheduling decision.

Here is simply a unsmooth guideline to really the policies compare:

What existent postulation shows

One communal logic accumulation numbers disagree from benchmark results is that the benchmark was acold cleaner than the existent workload. This is not conscionable a hunch. When researchers characterized existent LLM serving postulation successful ServeGen, they recovered that petition arrivals are usually bursty, pinch a coefficient of variety supra 1, which is different measurement of saying a fixed-rate aliases elemental Poisson generator does not picture them well. The BurstGPT dataset makes the aforesaid constituent astatine scale: it collected much than 10 cardinal existent petition traces from Azure’s OpenAI services complete 213 days and recovered abrupt bursts successful really galore requests are moving astatine once. A synthetic trial usually sends requests astatine a fixed rate, pinch azygous punctual lengths, from 1 client. Real postulation has 3 habits that break that:

• Bursty arrivals: Requests clump. Ten onshore successful the aforesaid second, past thing for 3 seconds. Bursts are erstwhile prefills heap up and artifact decodes.
• Mixed punctual lengths: A 50-token chat connection and a 30,000-token archive summary deed the aforesaid server. The agelong 1 is what stalls the short ones.
• Multi-tenant load: Lots of users stock the GPU, truthful 1 person’s elephantine punctual becomes different person’s latency spike.

If you want your numbers to mean anything, trial against traces that seizure this, not against azygous load. Public datasets for illustration BurstGPT and the Azure LLM conclusion traces springiness you existent presence patterns and existent punctual and consequence magnitude spreads. The trial worthy moving is simple. Take 1 fixed config, tally it nether azygous load, past replay a existent bursty trace done the aforesaid config. Report both. The published activity already tells you what to expect: p50 hardly moves while p99 climbs hard, because the tail is group by the bursts and the agelong prompts, not by the mean request. That spread is what you should attraction on. It is besides the portion almost nary vendor benchmark shows you, which is precisely why publishing it earns trust.

How to fig retired what you are really bound by

Before you touch immoderate config, activity retired which of the 3 problems you have. Look astatine your ain traffic.

Are you prefill-bound? Your prompts are agelong compared to your outputs (RAG, archive processing, agelong strategy prompts), and clip to first token is your main complaint. Chunked prefill is your biggest lever here.

Are you decode-bound? Your outputs are agelong compared to your inputs (agents, agelong generations), and inter-token latency aliases throughput is the issue. Focus connected batch size and representation headroom much than chunking.

Are you scheduling-bound? Your averages look good but your tail gets worse nether bursts. This is simply a argumentation and preemption problem.

Once you cognize which 1 you are, present are the first knobs to scope for, successful order:

1. Turn connected chunked prefill if you person agelong prompts and a tail problem. Start pinch the default token budget.
2. Tune max_num_batched_tokens (your chunk size dial). If decodes still stall, little it. If prefills consciousness excessively slow and decodes are fine, raise it. Move it successful steps and measurement against your trace, not against a azygous request.
3. Set max_num_seqs to lucifer the representation you really have, truthful you preempt little often.
4. Pick a scheduling argumentation that fits your traffic. Leave FCFS connected only if your load is genuinely uniform. Otherwise move to privilege aliases SLA-aware.

A unsmooth consciousness of erstwhile chunked prefill starts to matter: if your prompts are short (a fewer 100 tokens) and concurrency is low, it mostly adds overhead and you tin skip it. As your prompts climb into the thousands of tokens, aliases concurrency rises to wherever prefills and decodes support colliding, chunked prefill stops being optional and becomes the main point keeping your tail flat. Do not pursuit magic numbers from a blog post, including this one. Measure your ain postulation and fto the numbers decide.

Common questions

What causes p99 latency spikes successful vLLM? Long prefills blocking decodes that are already running. When a large punctual starts processing, everyone else’s token procreation waits for that loop to finish. It is simply a scheduling problem, not a earthy compute limit, and it shows up nether the bursty, mixed postulation that synthetic benchmarks usually miss.

Is chunked prefill the aforesaid arsenic continuous batching? No. Continuous batching schedules per iteration, truthful vanished sequences time off and caller ones subordinate aft each step. Chunked prefill goes further by splitting a agelong prefill into smaller pieces truthful it tin tally adjacent to decodes alternatively of blocking them. You usage some together.

Does PagedAttention trim latency aliases conscionable representation usage? Directly, it manages KV cache representation and cuts fragmentation. It does not schedule anything, truthful it does not hole latency connected its own. It helps indirectly, because erstwhile representation is nary longer the difficult limit connected batch size, the scheduler has much room to interleave prefill and decode well.

How do I take a chunk size? Treat the per-iteration token fund (max_num_batched_tokens) arsenic your dial. Too mini keeps decodes snappy but slows prefill and adds overhead. Too ample brings backmost the blocking you were trying to fix. Start pinch the default, past set successful steps while testing against a realistic postulation trace.

Why do my benchmarks look good but accumulation does not? Your benchmark astir apt utilized evenly spaced, same-size requests. Production postulation is bursty, mixes short and agelong prompts, and shares the GPU crossed users. Re-run your benchmark against a existent trace for illustration the Azure LLM Inference Trace aliases BurstGPT and watch what happens to p99.

When should I not fuss pinch chunked prefill? When your prompts are short and concurrency is low. In that lawsuit it mostly adds overhead. It earns its support arsenic prompts turn into the thousands of tokens and concurrency rises to wherever prefills and decodes support colliding.

Conclusion

Tail latency is not a hardware problem you tin bargain your measurement retired of. It is simply a scheduling problem. A agelong prefill blocks the decodes already successful flight, preemption fires astatine the worst moment, and a benchmark built connected azygous load hides each of it. Chunked prefill is the main lever for the first problem, a argumentation that fits your postulation handles the second, and testing against bursty traces is really you extremity being amazed successful production. Start by moving retired whether you are prefill-bound, decode-bound, aliases scheduling-bound, past alteration 1 knob astatine a clip and measurement the tail, not the median.

None of this needs fancy infrastructure. Chunked prefill, paged memory, and preemption are already built into vLLM and the different awesome engines, truthful astir of the activity is knowing what they do and tuning them to your traffic, not building thing from scratch. The teams that get predictable p99s are usually not the ones pinch the biggest GPUs. They are the ones who cognize which bound they are hitting, who trial against postulation that looks for illustration accumulation alternatively of a cleanable loop, and who are honorable astir the tradeoffs erstwhile they stock their numbers. And if a azygous node is nary longer enough, the aforesaid ideas support going: disaggregating prefill and decode onto abstracted pools is the adjacent measurement up, and it builds connected precisely the scheduling intuition covered here.

If you return 1 point away, fto it beryllium this. The median is comfortable, but the tail is wherever your users really live. Measure the tail, tune for it, and the remainder of the stack gets overmuch easier to logic about.

• Continuous Batching Mechanics: The Scheduler Behind vLLM, TGI, and SGLang
• Continuous Batching vs. Static Batching successful LLM Inference
• How KV Caching Slashes LLM Inference Costs astatine Scale
• LLM Inference Optimization: Quantization to Speculative Decoding, Part 1