Continuous Batching Improves Your P50 and Can Wreck Your P99: The Measured Tradeoff

Test Setup

Everything successful this article is simply a unrecorded measurement connected a DigitalOcean H200 GPU Droplet moving vLLM v0.24.0 pinch Llama 3.1 8B, driven by a mixed-length postulation trace (mostly short requests, immoderate medium, immoderate long):

• Continuous vs static-style (gated, B=16) admission connected the aforesaid server, aforesaid trace, aforesaid seed
• Arrival rates ramped from 1 to 20 requests/second for the continuous arm
• Chunked prefill connected vs off, 1 honorable config toggle

Github repository pinch the earthy JSON files, the suite log, the plotting script, the thesis notes, and the 3 measured charts from the H200 run: github.com/anishsingh20/continuous-vs-static-batching.

TL;DR

• The celebrated 23x number is simply a throughput and p50 result, not a tail result. Anyscale’s ain benchmark title says “while reducing p50 latency.” The tail behaviour nether mixed-length concurrent load is the under-documented half of the story, and it is wherever accumulation symptom lives, arsenic my related article connected p50 vs p99 Latency: Why Median Benchmarks Mislead AI Agent Workloads argues successful wide form. Sources: Anyscale continuous batching benchmark, Yu et al., Orca, OSDI 2022.
• Continuous batching’s tail costs appears wrong the token stream, done 2 separable mechanisms: agelong prefills stalling in-flight decodes, and preemption nether KV cache pressure. The GPU Droplet harness beneath measures the size of some effects connected existent hardware.
• Modern vLLM defaults already mitigate the first mechanism. vLLM V1 enables chunked prefill by default pinch a decode-first scheduling policy, per vLLM’s ain optimization documentation. The melodramatic prefill-stall communicative describes engines and configurations without this mitigation.
• The 2nd mechanism, preemption, survives the first fix. vLLM V1’s default preemption mode is RECOMPUTE: nether KV cache unit a moving petition is evicted and its full prefill runs again later. The motor counts these events successful its Prometheus metrics, and the harness successful this portion records that antagonistic astir each run.
• Static-style admittance keeps 1 genuine advantage the measurement makes visible: a calmer token watercourse erstwhile the reply is moving (gated spread p50 52.8 sclerosis vs 129.6 sclerosis for continuous astatine 10 req/s), because a locked batch admits thing that could interrupt it. Its niche is existent and narrow.
• The GPU research ran connected an H200 Droplet. Gated admittance mislaid severely connected TTFT (p50 196 sclerosis vs astir 19 to 48 sclerosis continuous). Chunked-prefill-off astatine 10 req/s worsened worst-gap p99 from 189.9 sclerosis to 267.8 sclerosis (1.41x). Preemptions stayed astatine zero for 8B connected 141 GB. Full JSON and charts: github.com/anishsingh20/continuous-vs-static-batching.

Table of Terms and Concepts

In lawsuit you are caller to the world of LLM serving, present is simply a array of position and concepts that will beryllium utilized successful this article.

How the 2 batching models really work

Every LLM petition has 2 steps. First the exemplary reads your punctual (prefill). Then it writes the reply 1 token astatine a clip (decode). Batching is only the norm for who shares the GPU while that happens.

Static makes you hold at the door (slow clip to first token). Continuous lets you successful fast, but a caller agelong punctual joining mid-flight tin make answers already streaming stutter. The subsections beneath unpack some sides.

Static batching: the batch is simply a locked room

Static batching is request-level scheduling. The server collects requests until a batch fills aliases a timer expires, runs each their prefills, past decodes the full batch together until each series successful it finishes. Three costs travel straight from the definition.

The batch-fill wait: early arrivals beryllium idle until the batch fills aliases times out, paying latency earlier immoderate compute happens. The longest-sequence floor: the batch releases only erstwhile its longest series finishes, truthful a petition that needed 100 tokens waits for the neighbour that needed 600, and the full batch’s occupancy clip equals the maximum, not the mean. The utilization decay: arsenic sequences finish, their compute slots idle wrong the still-locked batch, truthful a batch of 16 mightiness walk its last iterations doing the activity of 3 while 13 slots do nothing. Those idle grey slots are the nonstop inefficiency the Orca insubstantial documented earlier fixing. Source: Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022, pages 521 to 538.

The 3 costs successful 1 picture: the capable hold astatine the door, the grey dormant capacity inside, and the merchandise statement group by the slowest occupant.

The capacity consequence is the portion that matters for the tail. Static batching’s sustainable throughput is astatine astir B divided by the batch occupancy time, and the occupancy clip is group by the longest sequence. Push arrivals past that ceiling and the queue grows without bound, which intends admittance delay, and pinch it clip to first token, grows without bound too. This is mean queueing theory, and it is the system down the gated arm’s achy admittance numbers successful the measured results below.

Continuous batching: the batch is simply a revolving door

Orca’s penetration was to schedule astatine the granularity of a azygous loop alternatively than a full request. After each guardant pass, the scheduler re-decides the batch: sequences that conscionable vanished exit instantly and return to their clients, and waiting requests subordinate successful the freed capacity, mid-flight. The paper’s ain absurd states the problem it fixes successful 1 sentence: requests that decorativeness earlier than others successful a batch cannot return to the client, while recently arrived requests must hold for the batch to afloat finish. Iteration-level scheduling removes some waits. vLLM builds connected the aforesaid creation and adds PagedAttention, which manages KV cache representation successful non-contiguous blocks truthful that the aggressive, move batch creation continuous batching wants is not defeated by representation fragmentation. Sources: Yu et al., OSDI 2022; Kwon et al., Efficient Memory Management for Large Language Model Serving pinch PagedAttention, SOSP 2023, pages 611 to 626.

No capable wait, nary dormant capacity, nary merchandise line. The unfastened mobility is what joining and leaving mid-flight does to the requests already inside.

Where the tail costs enters

Everything continuous batching wins, it wins by making the batch creation dynamic. The tail costs enters done the aforesaid door, by 2 chopped mechanisms worthy keeping separate, because they respond to different fixes.

Mechanism one: prefill insertion. When a caller petition joins mid-flight, its punctual needs prefill, and prefill is compute-bound. In the scheduling argumentation vLLM documents for its pre-chunking behavior, the scheduler prioritizes prefills and does not batch prefill and decode into the aforesaid guardant pass. A 6,000-token punctual arriving becomes a prefill-only loop during which each in-flight decode watercourse produces nothing. Each of those streams shows 1 agelong inter-token gap, astatine the aforesaid instant, done nary responsibility of their own. The stall repeats each clip a agelong punctual arrives, which nether mixed-length postulation is constantly. Source for the scheduling argumentation description: vLLM Optimization and Tuning documentation.

Mechanism two: preemption nether KV pressure. Continuous batching admits aggressively, and each admitted sequence’s KV cache grows pinch each token it generates. When the cache fills, the scheduler must evict someone. vLLM V1’s documented default preemption mode is RECOMPUTE alternatively than SWAP: the victim’s cache is dropped and its full prefill runs again erstwhile capacity frees. For the unfortunate this is simply a mid-stream stall followed by a afloat 2nd prefill delay, a axenic tail arena invisible successful immoderate median. The motor exposes a cumulative preemption antagonistic done its Prometheus metrics and logs it erstwhile disable_log_stats=False is set, which is what makes this system straight observable alternatively than inferred. Source: vLLM Optimization and Tuning documentation, preemption section.

The 2nd system survives the hole for the first one. The harness successful this portion records the engine’s ain preemption antagonistic astir each run.

The mitigation that changed the story: chunked prefill

The prefill-insertion system has a documented, deployed fix, and being honorable astir it is what separates this portion from the folklore. Chunked prefill splits a agelong punctual into pieces, 2,048 tokens is simply a typical budget, and batches each portion alongside the ongoing decodes alternatively of moving it alone. The scheduling argumentation inverts: decodes are scheduled first, each iteration, and prefill chunks capable immoderate token fund remains. In-flight streams spot somewhat wider token spacing while chunks tally alternatively of 1 agelong dormant gap. The thought was developed successful Sarathi-Serve, which framed it arsenic taming the throughput-latency tradeoff by removing prefill-decode interference. Source: Agrawal et al., Taming Throughput-Latency Tradeoff successful LLM Inference pinch Sarathi-Serve, OSDI 2024.

Here is the portion immoderate benchmark connected this taxable must disclose, because it moves the results by an bid of magnitude: in vLLM V1, chunked prefill is enabled by default whenever possible, pinch the decode-first argumentation active. vLLM’s archiving states the tradeoff some ways pinch different directness: smaller per-iteration token budgets astir 2,048 springiness amended inter-token latency because less prefill tokens slow the decodes, larger budgets springiness amended clip to first token, and budgets supra 8,192 are recommended for earthy throughput. The melodramatic stall communicative successful system 1 describes engines and configurations without this mitigation, which includes older vLLM versions wherever the characteristic was disconnected by default and immoderate existent deployment that abnormal it aliases raised the fund acold capable to recreate the problem. Source: vLLM Optimization and Tuning documentation.

The azygous astir important config disclosure successful immoderate benchmark connected this topic. The measured results beneath quantify precisely really overmuch on-versus-off moves the tail connected existent hardware.

The experiment

One model, 1 GPU, 1 motor build, 1 petition trace, 1 seed. The only adaptable is the admittance policy, positive 1 config toggle wrong the continuous arm.

• Hardware: DigitalOcean GPU Droplet pinch size gpu-h200x1-141gb, NVIDIA H200, region NYC2, created from the 1-Click Inference Ready image. You tin cheque existent hourly rates connected the GPU Droplet pricing page.
• Model: Llama 3.1 8B Instruct astatine BF16, served arsenic RedHatAI/Llama-3.1-8B-Instruct (ungated BF16 redistributable of the aforesaid lineage). An 8B exemplary keeps decode iterations accelerated and scheduler dynamics visible without multi-hour wall time.
• Engine: vllm/vllm-openai:v0.24.0. Prefix caching abnormal pinch --no-enable-prefix-caching truthful identical filler prompts do not illness prefill cost.
• Trace: 70 percent short, 20 percent medium, 10 percent long, seed 7, 500 measured requests positive 25 warmup per run. Published pinch the results successful the public GitHub repo.

Note: vLLM does not person a accepted fixed batching mode. It is built wholly astir iteration-level continuous batching (also called in-flight aliases move batching) paired pinch PagedAttention to maximize GPU utilization and destruct the idle hold times inherent to fixed batches

The ramp and the toggle

The continuous limb runs open-loop Poisson arrivals ramped done 1, 5, 10, and 20 requests per second, pinch 500 measured requests per level aft a discarded warmup. The gated limb runs the aforesaid trace done its batch gate. The continuous limb past repeats its ramp pinch chunked prefill explicitly disabled, because the chunked-versus-unchunked delta is the azygous astir decision-relevant number this research produces. Both configs are recorded verbatim successful the output file.

What gets measured

Per request, successful some arms: TTFT from a client-side timestamp connected the first streamed contented chunk, the worst inter-token spread from timestamps connected each chunk, and full completion time. Per run: the engine’s vllm:num_preemptions Prometheus antagonistic scraped earlier and after, truthful system 2 is observed alternatively than inferred, positive the moving batch occupancy if the deployed vLLM type exposes it. Client-side concurrency uses existent OS threads alternatively than a azygous asyncio loop, for the aforesaid logic arsenic the companion latency piece: a 2026 measurement-bias insubstantial models single-process async clients arsenic an M/G/1 queue whose ain bottleneck inflates the tail metrics nether measurement. Source: Chandrasekar and Kramberger, Identifying and Mitigating Systemic Measurement Bias successful Production LLM Inference Benchmarks, arXiv.

Everything held changeless isolated from the admittance policy. The gated fixed limb is disclosed arsenic a stand-in, and the guidance of its bias is stated.

Measured results connected a DigitalOcean H200 GPU Droplet

This conception is the spine of the article. Everything supra it explained the mechanisms. Everything beneath it is what really happened erstwhile the mixed-length trace deed a existent motor connected a existent GPU Droplet.

Github repository: github.com/anishsingh20/continuous-vs-static-batching, containing the harness, per-request JSON, suite log, metadata, and the 3 charts explained here.

What was held constant

The only intentional variables crossed cells: admittance argumentation (open continuous vs gated B=16), presence complaint for the continuous arm, and 1 config toggle (chunked prefill connected vs off).

How to publication each column

• TTFT p50 / p99: client-side clip from petition nonstop to the first streamed contented chunk. This is wherever gated/static admittance usually loses.
• Worst spread p50 / p99: the largest region betwixt successive streamed chunks wrong 1 response. This is wherever continuous batching’s prefill-insertion stalls show up. A dashboard that only tracks TTFT will miss it.
• Total p99: end-to-end completion clip astatine the 99th percentile. Under open-loop Poisson load this besides reflects queueing while the GPU is busy; it is not a axenic “scheduler quality” number by itself.
• Preemptions: delta of vllm:num_preemptions_total earlier vs aft the run. Nonzero intends system 2 (KV reclaim / RECOMPUTE) really fired.

Full measured table

All values are milliseconds isolated from the preemption count. Zero errors connected each cell.

Row-by-row: continuous limb pinch motor defaults (chunked on)

Rate 1 req/s. This is the light-load baseline. Median TTFT is 17.0 ms: the petition is admitted instantly and the first token arrives quickly. Median worst spread is 5.9 ms, which is simply a soft stream. The p99 spread of 129.7 sclerosis and p99 full of 3434.1 sclerosis are already telling you the long-prompt / long-generation tail of the mixed trace is successful the data: 10% of requests inquire for up to 600 output tokens aft a 6,000-token-ish prefill, truthful the completion-time tail is agelong moreover erstwhile the scheduler is idle astir of the time. Preemptions: 0.

Rate 5 req/s. TTFT hardly moves (p50 19.3, p99 161.2). The first spot load shows up is the stream: spread p50 rises from 5.9 to 18.3 ms. That is the commencement of system 1 nether defaults: different requests’ prefills are already sharing iterations pinch your decode. Total p99 rises to 4409.3 ms. Preemptions still 0.

Rate 10 req/s. This is the knee for this hardware/model/trace. TTFT p50 is still good astatine 24.1 ms, but TTFT p99 climbs to 252.2 ms. The smoking weapon is spread p50: 129.6 ms. The median petition now sees a worst inter-token region complete a 100 milliseconds. Gap p99 is 189.9 ms. If you only watched TTFT medians, you would still judge the strategy was healthy. Users watching tokens look connected surface would already consciousness hitching. Total p99 is 5723.2 ms. Preemptions still 0.

Rate 20 req/s. Open-loop unit is now intelligibly past comfort. TTFT p50 doubles again to 47.7 ms; TTFT p99 hits 394.8 ms. Gap p50/p99 are 159.9 / 212.7 ms. Total p99 blows retired to 12510.4 ms: astatine this presence complaint the GPU cannot drain the offered load arsenic accelerated arsenic it arrives, truthful completion times see existent queueing, not conscionable per-request compute. Still zero preemptions: the H200’s KV headroom for an 8B exemplary is tremendous for this trace magnitude mix.

Outcome for the continuous-defaults ramp: continuous admittance keeps median TTFT mini crossed the ladder, but the token watercourse roughens arsenic soon arsenic concurrency rises, and by 10 to 20 req/s the tails are load-dominated. That is precisely the “pain moved from the beforehand doorway into the stream” communicative from the first half of the piece.

Row-by-row: continuous limb pinch chunked prefill off

Rate 5, chunked off. Compared to defaults astatine the aforesaid rate, the numbers are almost a copy: TTFT p99 161.1 vs 161.2, spread p99 134.3 vs 134.9, full p99 4482.3 vs 4409.3. At this mean load connected an H200 moving 8B, turning chunking disconnected does not recreate the folklore disaster. Publish that. It intends V1’s default mitigation and the headroom of this GPU are doing the quiet work.

Rate 10, chunked off. Now the toggle matters, but modestly. TTFT is fundamentally unchanged (p99 249.4 vs 252.2). Worst-gap p99 rises from 189.9 to 267.8 ms, a 1.41x increase. Gap p50 stays astir 129 sclerosis either way. So disabling chunked prefill widens the tail of watercourse stalls without changing time-to-first-token. Direction of the mechanism: confirmed. Magnitude of the folklore cliff: rejected for this setup.

Row-by-row: gated admittance (static stand-in), B=16

Gated is not a 2nd engine. It is the aforesaid vLLM server pinch a customer that only ever has 16 in-flight requests, waits for each 16 to finish, past admits the adjacent 16. That reproduces fixed batching’s outer shape: capable boundary, all-finish-together release, longest-sequence occupancy.

Measured: TTFT p50 195.8 ms, TTFT p99 637.2 ms. Against continuous astatine 10 req/s (24.1 / 252.2), gated’s median TTFT is astir 8x worse, and its p99 TTFT is 2.5x worse. That is the locked room: early requests successful a batch hold for the batch to shape and for slow neighbors to decorativeness earlier the adjacent batch tin start, truthful time-to-first-token absorbs admittance delay.

Worst-gap for gated is calmer than continuous nether load: spread p50 52.8 ms, spread p99 203.6 ms. The watercourse is not the disaster mode. Admission is. Total p99 (4073.6 ms) is really lower than continuous astatine 10 to 20 req/s successful this array because gated people rate-limits itself. It ne'er open-loops 20 arrivals per 2nd into the engine, truthful you cannot publication full p99 arsenic “gated is faster overall.” You publication it arsenic “gated refuses to judge the aforesaid offered load.”

Outcome for gated: static/gated loses astatine the beforehand door. Its niche remains workloads that request watercourse smoothness and tin technologist astir admittance delay, not wide API serving.

Preemption column: why each compartment is zero

Mechanism 2 is existent successful vLLM’s docs and successful accumulation connected tighter GPUs. It did not look here. An 8B BF16 exemplary connected 141 GB of H200 memory, pinch this prompt/output mix, ne'er forced RECOMPUTE. The harness scraped vllm:num_preemptions_total earlier and aft each run; the delta was 0.0 each time. That is simply a finding, not a missing measurement: if your p99 communicative connected akin hardware is “we are preempting,” you are astir apt connected a larger model, longer context, higher max_num_seqs. So please support an oculus connected the preemption antagonistic successful production.

Chart 1: TTFT p99 and worst-gap p99 versus presence rate

This floor plan has 2 side-by-side graphs. Both only screen the “continuous” trial (new requests tin subordinate anytime).

Takeaway: the near chart is “time to commencement talking”; the correct chart is “stuttering while talking.” Chunked prefill helps the stuttering chart much than the commencement graph.

Chart 2: p99 comparison crossed arms astatine the determination point

Three grouped bars per arm: TTFT p99, worst-gap p99, full p99.

This is simply a barroom floor plan pinch 3 setups broadside by side:

1. Continuous, normal settings, astir 10 caller requests each second
2. Continuous, aforesaid traffic, but chunked prefill turned off
3. Gated groups of 16 (our stand-in for old-style fixed batching)

Each setup has 3 colored bars:

Takeaway: With continuous, you get a reply faster; pinch gated, you person to hold successful statement earlier thing happens; and if you move chunked prefill off, the reply comes retired successful much awkward, choppy bursts, for illustration personification pausing a batch while talking.

What the measurement says (and does not)

Gated/static admittance loses precisely wherever the system study said it would: TTFT. Median TTFT for gated B=16 was 195.8 sclerosis against 19.3 sclerosis for continuous astatine 5 req/s and 24.1 sclerosis astatine 10 req/s. That is the locked-room fill-and-release cost, measured connected existent hardware. Worst-gap for gated stayed mean (p99 203.6 ms), accordant pinch “smooth stream, achy admission.”

Bottom statement from the experiment

1. Gated/static admittance loses connected TTFT, hard. Median 196 sclerosis vs continuous teens-to-low-forties. Confirms the locked-room thesis.
2. Continuous’s interactive consequence is watercourse hitching nether load, visible successful spread p50/p99 from 10 req/s upward, not successful median TTFT.
3. Chunked prefill still helps, pinch a measured gap-p99 triumph of 1.41x astatine 10 req/s. Folklore overstated the default-config cliff connected existent vLLM V1.
4. Reproduce aliases mention from the repo: github.com/anishsingh20/continuous-vs-static-batching.

Harness stdout

This is the earthy output of the harness. It is simply a log of the measurements taken during the experiment.

==== CONTINUOUS DEFAULTS RATE RAMP ==== --- continuous rate=1 2026-08-04T07:00:15Z --- { "arm": "continuous", "n": 500, "started": "2026-08-04T07:00:15Z", "rate_or_batch": 1.0, "ttft_p50": 17.0, "ttft_p99": 154.0, "gap_p50": 5.9, "gap_p99": 129.7, "total_p50": 507.2, "total_p99": 3434.1, "preemptions_delta": 0.0, "errors": 0 } --- continuous rate=5 2026-08-04T07:09:36Z --- { "arm": "continuous", "n": 500, "ttft_p50": 19.3, "ttft_p99": 161.2, "gap_p50": 18.3, "gap_p99": 134.9, "total_p99": 4409.3, "preemptions_delta": 0.0, "errors": 0 } --- continuous rate=10 2026-08-04T07:11:31Z --- { "arm": "continuous", "n": 500, "ttft_p50": 24.1, "ttft_p99": 252.2, "gap_p50": 129.6, "gap_p99": 189.9, "total_p99": 5723.2, "preemptions_delta": 0.0, "errors": 0 } --- continuous rate=20 2026-08-04T07:12:30Z --- { "arm": "continuous", "n": 500, "ttft_p50": 47.7, "ttft_p99": 394.8, "gap_p50": 159.9, "gap_p99": 212.7, "total_p99": 12510.4, "preemptions_delta": 0.0, "errors": 0 } ==== GATED B=16 ==== { "arm": "gated", "n": 500, "rate_or_batch": 16, "ttft_p50": 195.8, "ttft_p99": 637.2, "gap_p50": 52.8, "gap_p99": 203.6, "total_p99": 4073.6, "preemptions_delta": 0.0, "errors": 0 } ==== RESTART NO-CHUNK ==== Using chunked-disable flag: --no-enable-chunked-prefill --- nochunk continuous rate=5 --- { "ttft_p99": 161.1, "gap_p99": 134.3, "total_p99": 4482.3, "preemptions_delta": 0.0 } --- nochunk continuous rate=10 --- { "ttft_p99": 249.4, "gap_p99": 267.8, "total_p99": 5844.2, "preemptions_delta": 0.0 } ==== SUITE COMPLETE ====

Runbook: reproducing this trial connected your DigitalOcean GPU Droplet

What you request earlier starting. A deployed GPU Droplet pinch the NVIDIA driver and Docker moving (the 1-Click Inference Ready image ships some preinstalled). SSH entree to the Droplet. Either Hugging Face entree to meta-llama/Llama-3.1-8B-Instruct, aliases the ungated redistributable RedHatAI/Llama-3.1-8B-Instruct utilized successful the published run. Roughly 30 GB of free disk for BF16 8B weights. Python 3 connected the Droplet. Nothing else: the harness uses only the Python modular library. Clone anishsingh20/continuous-vs-static-batching for the harness and plotting scripts.

Step 1. Confirm the GPU is visible.

nvidia-smi

Record the driver type and GPU sanction from the output into your results file. If this bid fails, extremity and hole the driver earlier thing else.

Step 2. Pull the pinned motor image and grounds its digest.

docker propulsion vllm/vllm-openai:v0.24.0 docker images --digests | grep vllm

Copy the sha256 digest verbatim into your results file. The pin positive digest is what makes the tally reproducible aft vLLM’s scheduler changes again.

Step 3. Start the server, limb one, each defaults.

# Meta repo is license-gated; published tally utilized RedHatAI/Llama-3.1-8B-Instruct (ungated BF16). export HF_TOKEN=your_hugging_face_token_here # only needed for meta-llama/* docker tally -d --name vllm-default --gpus each --ipc=host -p 8000:8000 \ -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:v0.24.0 \ --model RedHatAI/Llama-3.1-8B-Instruct \ --no-enable-prefix-caching

V1 defaults time off chunked prefill connected (decode-first) per vLLM’s optimization documentation. Prefix caching is abnormal truthful identical filler prompts do not illness prefill costs crossed requests. The first commencement downloads the weights, which takes respective minutes. Follow advancement pinch docker logs -f vllm-default and hold for the server-ready line.

Step 4. Health-check the endpoint and the metrics scrape.

curl -s http://localhost:8000/v1/models | head -c 400 curl -s http://localhost:8000/metrics | grep -i preemption

The first bid must return a exemplary database containing the nonstop exemplary ID. The 2nd must return the preemption antagonistic line; statement its nonstop metric name, since the harness scrapes the metrics endpoint and the antagonistic sanction pronunciation connected your pinned type is worthy confirming erstwhile by eye. If the grep returns nothing, tally curl -s http://localhost:8000/metrics | grep vllm: and grounds what the antagonistic is called connected this version.

Step 5. Save the harness and tally the warm-up positive baseline.

Copy the harness from the conception beneath into batching_bench.py connected the Droplet, past tally the continuous limb crossed a complaint ramp. The correct rates dangle connected your hardware, truthful ramp until the tail visibly inflates alternatively than trusting immoderate fixed list. A reasonable starting ladder for an 8B exemplary connected 1 H200:

python3 batching_bench.py --arm continuous --rate 1 --n 500 --out cont_r1.json python3 batching_bench.py --arm continuous --rate 5 --n 500 --out cont_r5.json python3 batching_bench.py --arm continuous --rate 10 --n 500 --out cont_r10.json python3 batching_bench.py --arm continuous --rate 20 --n 500 --out cont_r20.json

Each tally discards 25 warmup requests by default and prints per-level summaries arsenic they finish. If p99 has not moved by complaint 20, support doubling the complaint until you find the knee, and grounds each level you ran.

Step 6. Run the gated limb connected the aforesaid server.

python3 batching_bench.py --arm gated --batch-size 16 --n 500 --out gated_b16.json

The gated limb submits batches of 16, waits for each to finish, past submits the adjacent batch. This is the disclosed stand-in for existent fixed batching, since vLLM has nary fixed mode, and the portion flags it arsenic mildly flattering to fixed batching.

Step 7. Restart the server pinch chunked prefill abnormal and rerun the continuous ramp.

docker rm -f vllm-default docker tally -d --name vllm-nochunk --gpus each --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:v0.24.0 \ --model RedHatAI/Llama-3.1-8B-Instruct \ --no-enable-prefix-caching \ --no-enable-chunked-prefill

Verified connected the published pin: v0.24.0 accepts --no-enable-chunked-prefill. Flag spellings still displacement betwixt releases, truthful corroborate against your image earlier quoting a different tag. Then repetition rates 5 and 10 pinch output names specified arsenic nochunk_r5.json.

Step 8. Archive everything.

Copy each JSON output files, the digest, the nvidia-smi line, some verbatim docker commands, and the dates and times of each tally disconnected the Droplet. Regenerate figures pinch python3 benchmarks/plot_results.py. The published archive lives astatine anishsingh20/continuous-vs-static-batching.

The harness

#!/usr/bin/env python3 """ Batching-policy latency harness for a vLLM server connected a DigitalOcean GPU Droplet. Arms: continuous : open-loop Poisson arrivals astatine --rate req/s (real OS threads) gated : batches of --batch-size, taxable all, hold for all, repeat Per request: TTFT, worst inter-token gap, full clip (client-side streaming timestamps). Per run: vllm preemption antagonistic scraped earlier and after. Start the server first (pin the image tag AND grounds the digest): docker tally --gpus each -p 8000:8000 vllm/vllm-openai:v0.24.0 \ --model meta-llama/Llama-3.1-8B-Instruct # chunked-off arm: adhd --no-enable-chunked-prefill # (verified connected v0.24.0; re-check if you alteration the pin) Run: python3 batching_bench.py --arm continuous --rate 5 --n 500 --out cont_r5.json python3 batching_bench.py --arm gated --batch-size 16 --n 500 --out gated.json """ import argparse, concurrent.futures, json, os, random, threading, time import urllib.request def make_trace(n, seed): rng = random.Random(seed) trace = [] for one in range(n): r = rng.random() if r < 0.70: p, o = 200, 100 elif r < 0.90: p, o = 1000, 300 else: p, o = 6000, 600 trace.append({"id": i, "prompt_tokens": p, "max_tokens": o}) return trace def build_prompt(n_tokens): # ~1 token per connection for a filler word; exactness is not required, # only that some arms usage the identical trace. return "ocean " * n_tokens def percentile(xs, p): xs = sorted(xs) k = (len(xs) - 1) * p / 100 f = int(k); c = min(f + 1, len(xs) - 1) return xs[f] if f == c else xs[f] * (c - k) + xs[c] * (k - f) def scrape_preemptions(base): """Prefer vllm:num_preemptions_total (v0.24+); autumn backmost to unlabelled name.""" try: with urllib.request.urlopen(base.replace("/v1", "") + "/metrics", timeout=10) as r: lines = r.read().decode().splitlines() for prefix in ("vllm:num_preemptions_total", "vllm:num_preemptions"): for statement in lines: if line.startswith(prefix) and not line.startswith(prefix + "_"): return float(line.split()[-1]) except Exception: pass return None def one_request(base, model, item): payload = { "model": model, "prompt": build_prompt(item["prompt_tokens"]), "max_tokens": item["max_tokens"], "temperature": 0, "stream": True, } req = urllib.request.Request( guidelines + "/completions", data=json.dumps(payload).encode(), headers={"Content-Type": "application/json", "Authorization": "Bearer " + os.environ.get("VLLM_API_KEY", "none")}, ) commencement = time.perf_counter() ttft = None; past = None; worst_gap = 0.0; n_chunks = 0 try: with urllib.request.urlopen(req, timeout=600) as r: for earthy in r: statement = raw.decode(errors="ignore").strip() if not line.startswith("data:") or line[5:].strip() == "[DONE]": continue now = time.perf_counter() if ttft is None: ttft = (now - start) * 1000 else: worst_gap = max(worst_gap, (now - last) * 1000) past = now; n_chunks += 1 except Exception as e: return {"id": item["id"], "error": str(e)} return {"id": item["id"], "ttft_ms": round(ttft, 1), "worst_gap_ms": round(worst_gap, 1), "total_ms": round((last - start) * 1000, 1), "chunks": n_chunks} def run_continuous(base, model, trace, rate, seed): rng = random.Random(seed + 1) results = []; fastener = threading.Lock() with concurrent.futures.ThreadPoolExecutor(max_workers=256) as pool: futures = [] for point in trace: futures.append(pool.submit(one_request, base, model, item)) time.sleep(rng.expovariate(rate)) for f in concurrent.futures.as_completed(futures): with lock: results.append(f.result()) return results def run_gated(base, model, trace, batch_size): results = [] for one in range(0, len(trace), batch_size): batch = trace[i:i + batch_size] with concurrent.futures.ThreadPoolExecutor(max_workers=batch_size) as pool: futures = [pool.submit(one_request, base, model, it) for it in batch] for f in concurrent.futures.as_completed(futures): results.append(f.result()) # the gate: thing caller is admitted until each petition supra returned return results def main(): ap = argparse.ArgumentParser() ap.add_argument("--base-url", default="http://localhost:8000/v1") ap.add_argument("--model", default="meta-llama/Llama-3.1-8B-Instruct") ap.add_argument("--arm", choices=["continuous", "gated"], required=True) ap.add_argument("--rate", type=float, default=5.0) ap.add_argument("--batch-size", type=int, default=16) ap.add_argument("--n", type=int, default=500) ap.add_argument("--warmup", type=int, default=25) ap.add_argument("--seed", type=int, default=7) ap.add_argument("--out", required=True) args = ap.parse_args() trace = make_trace(args.n + args.warmup, args.seed) pre = scrape_preemptions(args.base_url) t0 = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) if args.arm == "continuous": recs = run_continuous(args.base_url, args.model, trace, args.rate, args.seed) else: recs = run_gated(args.base_url, args.model, trace, args.batch_size) station = scrape_preemptions(args.base_url) bully = [r for r in recs if "error" not in r][args.warmup:] tt = [r["ttft_ms"] for r in good] gp = [r["worst_gap_ms"] for r in good] to = [r["total_ms"] for r in good] summary = { "arm": args.arm, "n": len(good), "started": t0, "rate_or_batch": args.rate if args.arm == "continuous" else args.batch_size, "ttft_p50": round(percentile(tt, 50), 1), "ttft_p99": round(percentile(tt, 99), 1), "gap_p50": round(percentile(gp, 50), 1), "gap_p99": round(percentile(gp, 99), 1), "total_p50": round(percentile(to, 50), 1), "total_p99": round(percentile(to, 99), 1), "preemptions_delta": (post - pre) if (pre is not None and station is not None) else None, "errors": len(recs) - len(good) - args.warmup, } with open(args.out, "w") as f: json.dump({"summary": summary, "records": good}, f, indent=2) print(json.dumps(summary, indent=2)) if __name__ == "__main__": main()

Reading the measured results arsenic a checklist

Use this checklist erstwhile you unfastened personification else’s batching benchmark, aliases erstwhile you re-run the harness from the Github repository.

1. Did TTFT detonate while gaps stayed flat? That is gated/static admittance (or a server that is simply overloaded earlier the first token). Our gated arm: TTFT p50 195.8 ms, spread p99 only 203.6 ms.
2. Did gaps detonate while TTFT medians looked fine? That is continuous prefill insertion (mechanism one). Our continuous defaults: astatine 10 req/s, TTFT p50 still 24.1 sclerosis while spread p50 jumped to 129.6 ms.
3. Did turning chunked prefill disconnected make gaps overmuch worse? If yes, you attributed system 1 correctly. Ours: spread p99 189.9 to 267.8 sclerosis astatine 10 req/s (1.41x). If the delta is tiny, opportunity so, since defaults whitethorn already beryllium doing the job.
4. Did vllm:num_preemptions_total move? If yes, system 2 is successful play and the hole is KV headroom / max_num_seqs, not much folklore astir batching. Ours: ne'er moved.
5. Was the trace mixed-length? Homogeneous short prompts hide some mechanisms. Our seed-7 operation is published pinch the JSON.

When to usage which

Continuous batching, motor defaults. Throughput-dominated work: offline scoring, information suites, synthetic data, summarization queues. Nobody watches a spinner, truthful intra-stream jitter is free, and the 23x-class throughput advantage is the full story. Raise the token fund per vLLM’s guidance and extremity tuning.

Continuous batching, tail levers engaged. Interactive serving pinch a p99 SLA and mixed-length traffic: chat, agents, thing streaming to a human. Keep chunked prefill on, support the fund modest, watch the preemption counter, and measurement the worst inter-token spread alternatively than only TTFT, because that is wherever this piece’s mechanisms hide from modular dashboards. If long-context requests stock the deployment, way them elsewhere first and tune second.

Gated aliases fixed admission. Two honorable niches. Hard real-time soul loops pinch fixed-size, fixed-length batches, wherever an uninterrupted token watercourse is the request and admittance hold is engineered distant by construction. And offline jobs pinch near-identical series lengths, wherever the locked room wastes thing because everyone finishes together anyway. General API serving is neither of these.

Continuous is the default. The determination is which levers you pull, and the 2 fixed niches are existent but narrow.

Conclusion

Continuous batching offers a powerful instrumentality for improving p50 latency successful LLM conclusion workloads, but it besides introduces challenges astatine higher percentiles, sometimes starring to important increases successful p99 latency. This article has demonstrated really the prime of batching argumentation and the usage of features for illustration chunked prefill tin dramatically displacement capacity characteristics, peculiarly for tail latency. Chunked prefill tin thief curb the worst gaps, providing a applicable lever for mixed and real-time workloads wherever predictability genuinely matters.

If you research pinch different GPUs, exemplary sizes, aliases settings and observe different impacts, particularly regarding chunked prefill aliases preemptions, your results are valuable to the community. Please see sharing by opening an rumor aliases propulsion petition connected the Github repository pinch your run_metadata.json and benchmark summaries. The nonstop numbers whitethorn vary, but the larger patterns of really admittance strategies and motor features impact different latency metrics are wide applicable.

Ultimately, location is nary one-size-fits-all solution. Rather, knowing these mechanisms empowers you to make informed choices based connected your workload, leveraging continuous batching for monolithic throughput wherever possible, aliases tuning for interactive, tail-sensitive scenarios wherever personification acquisition comes first. With thoughtful configuration and measurement, you tin get the champion of some worlds: businesslike utilization and predictable performance.

References

Papers

• Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022
• Kwon et al., Efficient Memory Management for Large Language Model Serving pinch PagedAttention, SOSP 2023
• Agrawal et al., Taming Throughput-Latency Tradeoff successful LLM Inference pinch Sarathi-Serve, OSDI 2024
• Chandrasekar and Kramberger, Identifying and Mitigating Systemic Measurement Bias successful Production LLM Inference Benchmarks, arXiv

Engine archiving and benchmarks

• vLLM Optimization and Tuning
• Anyscale, How continuous batching enables 23x throughput successful LLM conclusion while reducing p50 latency
• Anyscale benchmark codification repository

DigitalOcean archiving and related articles

• p50 vs p99 Latency: Why Median Benchmarks Mislead AI Agent Workloads (companion Community tutorial connected why p50 unsocial misleads)
• When Your vLLM p99 is Worse Than Your p50 (companion Community article connected why p99 tin beryllium worse than p50)
• Continuous Batching vs Static Batching: LLM Inference (companion Community article connected continuous batching vs fixed batching)
• GPU Droplets (product utilized for the unrecorded H200 measurement)
• GPU Droplet pricing (check existent hourly rates; do not spot a old number from immoderate article)
• 1-Click Inference Ready image, DigitalOcean Marketplace (image utilized to create the benchmark Droplet)
• anishsingh20/continuous-vs-static-batching (companion GitHub repo: harness, earthy JSON, charts, suite log)