Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025)

In this post, I'll gradually present each of the halfway strategy components and precocious features that dress up a modern high-throughput LLM conclusion system. In peculiar I'll beryllium doing a breakdown of really vLLM [1] works.

This station is the first successful a series. It starts wide and past layers successful item (following an inverse-pyramid approach) truthful you tin shape an meticulous high-level intelligence exemplary of the complete strategy without drowning successful minutiae.

Later posts will dive into circumstantial subsystems.

This station is system into 5 parts:

1. LLM motor & motor core: fundamentals of vLLM (scheduling, paged attention, continuous batching, etc.)
2. Advanced features: chunked prefill, prefix caching, guided & speculative decoding, disaggregated P/D
3. Scaling up: from single-GPU to multi-GPU execution
4. Serving layer: distributed / concurrent web scaffolding
5. Benchmarks and auto-tuning: measuring latency and throughput

LLM Engine & Engine Core

The LLM motor is the basal building artifact of vLLM. On its own, it already enables high-throughput conclusion - but only successful an offline setting. You can't service it to customers complete the web yet.

We'll usage the pursuing offline conclusion snippet arsenic our moving illustration (adapted from basic.py).

This configuration is:

• offline (no web/distributed strategy scaffolding)
• synchronous (all execution happens successful a azygous blocking process)
• single-GPU (no data/model/pipeline/expert parallelism; DP/TP/PP/EP = 1)
• using modular transformer [2] (supporting hybrid models for illustration Jamba requires a much analyzable hybrid KV-cache representation allocator)

From here, we'll gradually build up to an online, async, multi-GPU, multi-node conclusion strategy - but still serving a modular transformer.

In this illustration we do 2 things, we:

1. Instantiate an engine
2. Call make connected it to sample from the fixed prompts

Let's commencement analyzing the constructor.

LLM Engine constructor

The main components of the motor are:

• vLLM config (contains each of the knobs for configuring model, cache, parallelism, etc.)
• processor (turns earthy inputs → EngineCoreRequests via validation, tokenization, and processing)
• engine halfway customer (in our moving illustration we're utilizing InprocClient which is fundamentally == EngineCore; we'll gradually build up to DPLBAsyncMPClient which allows serving astatine scale)
• output processor (converts earthy EngineCoreOutputs → RequestOutput that the personification sees)

Engine halfway itself is made up of respective sub components:

• Model Executor (drives guardant passes connected the model, we're presently dealing pinch UniProcExecutor which has a azygous Worker process connected a azygous GPU). We'll gradually build up to MultiProcExecutor which supports aggregate GPUs
• Structured Output Manager (used for guided decoding - we'll screen this later)
• Scheduler (decides which requests spell into the adjacent motor step) - it further contains:
  1. policy mounting - it tin beryllium either FCFS (first travel first served) aliases priority (higher privilege requests are served first)
  2. waiting and moving queues
  3. KV cache head - the bosom of paged attraction [3]

The KV-cache head maintains a free_block_queue - a excavation of disposable KV-cache blocks (often connected the bid of hundreds of thousands, depending connected VRAM size and artifact size). During paged attention, the blocks service arsenic the indexing building that representation tokens to their computed KV cache blocks.

During exemplary organizer construction, a Worker entity is created, and 3 cardinal procedures are executed. (Later, pinch MultiProcExecutor, these aforesaid procedures tally independently connected each worker process crossed different GPUs.)

1. Init device:
   • Assign a CUDA instrumentality (e.g. "cuda:0") to the worker and cheque that the exemplary dtype is supported (e.g. bf16)
   • Verify capable VRAM is available, fixed the requested gpu_memory_utilization (e.g. 0.8 → 80% of full VRAM)
   • Set up distributed settings (DP / TP / PP / EP, etc.)
   • Instantiate a model_runner (holds the sampler, KV cache, and forward-pass buffers specified arsenic input_ids, positions, etc.)
   • Instantiate an InputBatch entity (holds CPU-side forward-pass buffers, artifact tables for KV-cache indexing, sampling metadata, etc.)
2. Load model:
   • Instantiate the exemplary architecture
   • Load the exemplary weights
   • Call model.eval() (PyTorch's conclusion mode)
   • Optional: telephone torch.compile() connected the model
3. Initialize KV cache
   • Get per-layer KV-cache spec. Historically this was ever FullAttentionSpec (homogeneous transformer), but pinch hybrid models (sliding window, Transformer/SSM for illustration Jamba) it became much analyzable (see Jenga [5])
   • Run a dummy/profiling guardant walk and return a GPU representation snapshot to compute really galore KV cache blocks fresh successful disposable VRAM
   • Allocate, reshape and hindrance KV cache tensors to attraction layers
   • Prepare attraction metadata (e.g. group the backend to FlashAttention) later consumed by kernels during the fwd pass
   • Unless --enforce-eager is provided, for each of warmup batch sizes do a dummy tally and seizure CUDA graphs. CUDA graphs grounds the full series of GPU activity into a DAG. Later during fwd walk we launch/replay pre-baked graphs and trim connected kernel motorboat overhead and frankincense amended latency.

I've abstracted distant galore low-level specifications present — but these are the halfway pieces I'll present now, since I'll reference them many times successful the pursuing sections.

Now that we person the motor initialized let's proceed to the make function.

Generate function

The first measurement is to validate and provender requests into the engine. For each punctual we:

1. Create a unsocial petition ID and seizure its presence time
2. Call an input preprocessor that tokenizes the punctual and returns a dictionary containing prompt, prompt_token_ids, and a type (text, tokens, embeds, etc.)
3. Pack this info into an EngineCoreRequest, adding priority, sampling params, and different metadata
4. Pass the petition into the motor core, which wraps it successful a Request entity and sets its position to WAITING. This petition is past added to the scheduler's waiting queue (append if FCFS, aliases heap-push if priority)

At this constituent the motor has been fed and execution tin begin. In the synchronous motor example, these first prompts are the only ones we'll process — there's nary system to inject caller requests mid-run. In contrast, the asynchronous motor supports this (aka continuous batching [6]): aft each step, some caller and aged requests are considered.

Next, arsenic agelong arsenic location are requests to process, the motor many times calls its step() function. Each measurement has 3 stages:

1. Schedule: prime which requests to tally successful this measurement (decode, and/or (chunked) prefill)
2. Forward pass: tally the exemplary and sample tokens
3. Postprocess: append sampled token IDs to each Request, detokenize, and cheque extremity conditions. If a petition is finished, cleanable up (e.g. return its KV-cache blocks to free_block_queue) and return the output early

Next, we'll analyse scheduling successful much detail.

Scheduler

There are 2 main types of workloads an conclusion motor handles:

1. Prefill requests — a guardant walk complete each punctual tokens. These are usually compute-bound (threshold depends connected hardware and punctual length). At the end, we sample a azygous token from the probability distribution of the last token's position.
2. Decode requests — a guardant walk complete conscionable the astir caller token. All earlier KV vectors are already cached. These are memory-bandwidth-bound, since we still request to load each LLM weights (and KV caches) conscionable to compute 1 token.

The V1 scheduler tin operation some types of requests successful the aforesaid step, acknowledgment to smarter creation choices. In contrast, the V0 motor could only process either prefill aliases decode astatine once.

The scheduler prioritizes decode requests — i.e. those already successful the moving queue. For each specified petition it:

1. Computes the number of caller tokens to make (not ever 1, owed to speculative decoding and async scheduling — much connected that later).
2. Calls the KV-cache manager's allocate_slots usability (details below).
3. Updates the token fund by subtracting the number of tokens from measurement 1.

After that, it processes prefill requests from the waiting queue, it:

1. Retrieves the number of computed blocks (returns 0 if prefix caching is abnormal — we'll screen that later).
2. Calls the KV-cache manager's allocate_slots function.
3. Pops the petition from waiting and moves it to running, mounting its position to RUNNING.
4. Updates the token budget.

Let's now look astatine what allocate_slots does, it:

1. Computes number of blocks — determines really galore caller KV-cache blocks (n) must beryllium allocated. Each artifact stores 16 tokens by default. For example, if a prefill petition has 17 caller tokens, we request ceil(17/16) = 2 blocks.
2. Checks availability — if location aren't capable blocks successful the manager's pool, exit early. Depending connected whether it's a decode aliases prefill request, the motor whitethorn effort recompute preemption (swap preemption was supported successful V0) by evicting low-priority requests (calling kv_cache_manager.free which returns KV blocks to artifact pool), aliases it mightiness skip scheduling and proceed execution.
3. Allocates blocks — via the KV-cache manager's coordinator, fetches the first n blocks from the artifact excavation (the free_block_queue doubly linked database mentioned earlier). Stores to req_to_blocks, the dictionary mapping each request_id to its database of KV-cache blocks.

We're yet fresh to do a guardant pass!

Run guardant pass

We telephone exemplary executor's execute_model, which delegates to the Worker, which successful move delegates to the exemplary runner.

Here are the main steps:

1. Update states — prune vanished requests from input_batch; update misc fwd walk related metadata (e.g., KV cache blocks per petition that will beryllium utilized to scale into paged KV cache memory).
2. Prepare inputs — transcript buffers from CPU→GPU; compute positions; build slot_mapping (more connected that successful example); conception attraction metadata.
3. Forward pass — tally the exemplary pinch civilization paged attn kernels. All sequences are flattened and concatenated into 1 agelong "super sequence". Position indices and attraction masks guarantee each series only attends to its ain tokens, which enables continuous batching without right-padding.
4. Gather last-token states — extract hidden states for each sequence's last position and compute logits.
5. Sample — sample tokens from computed logits arsenic dictated by the sampling config (greedy, temperature, top-p, top-k, etc.).

Forward-pass measurement itself has 2 execution modes:

1. Eager mode — tally the modular PyTorch guardant walk erstwhile eager execution is enabled.
2. "Captured" mode — execute/replay a pre-captured CUDA Graph erstwhile eager is not enforced (remember we captured these during motor building successful the initialize KV cache procedure).

Here is simply a actual illustration that should make continuous batching and paged attraction clear:

Advanced Features — extending the halfway motor logic

With the basal motor travel successful place, we tin now look astatine the precocious features.

We've already discussed preemption, paged attention, and continuous batching.

Next, we'll dive into:

1. Chunked prefill
2. Prefix caching
3. Guided decoding (through grammar-constrained finite-state machines)
4. Speculative decoding
5. Disaggregated P/D (prefill/decoding)

Chunked prefill

Chunked prefill is simply a method for handling agelong prompts by splitting their prefill measurement into smaller chunks. Without it, we could extremity up pinch a azygous very agelong petition monopolizing 1 motor measurement disallowing different prefill requests to run. That would postpone each different requests and summation their latency.

For example, fto each chunk incorporate n (=8) tokens, branded pinch lowercase letters separated by "-". A agelong punctual P could look for illustration x-y-z, wherever z is an incomplete chunk (e.g. 2 toks). Executing the afloat prefill for P would past return ≥ 3 motor steps (> tin hap if it's not scheduled for execution successful 1 of the steps), and only successful the past chunked prefill measurement would we sample 1 caller token.

Here is that aforesaid illustration visually:

Implementation is straightforward: headdress the number of caller tokens per step. If the requested number exceeds long_prefill_token_threshold, reset it to precisely that value. The underlying indexing logic (described earlier) takes attraction of the rest.

In vLLM V1, you alteration chunked prefill by mounting long_prefill_token_threshold to a affirmative integer. (Technically, it tin hap irrespective of this, if the punctual magnitude exceeds the token fund we truncate it and tally a chunked prefill.)

Prefix Caching

To explicate really prefix caching works, let's return the original codification illustration and tweak it a bit:

Prefix caching avoids recomputing tokens that aggregate prompts stock astatine the opening - hence prefix.

The important portion is the long_prefix: it's defined arsenic immoderate prefix longer than a KV-cache artifact (16 tokens by default). To simplify our illustration let's opportunity long_prefix has precisely magnitude n x block_size (where n ≥ 1).

Without prefix caching, each clip we process a caller petition pinch the aforesaid long_prefix, we'd recompute each n x block_size tokens.

With prefix caching, those tokens are computed erstwhile (their KVs stored successful KV cache paged memory) and past reused, truthful only the caller punctual tokens request processing. This speeds up prefill requests (though it doesn't thief pinch decode).

How does this activity successful vLLM?

During the first make call, successful the scheduling stage, wrong kv_cache_manager.get_computed_blocks, the motor invokes hash_request_tokens:

1. This usability splits the long_prefix + prompts[0] into 16-token chunks.
2. For each complete chunk, it computes a hash (using either the built-in hash aliases SHA-256, which is slower but has less collisions). The hash combines the erstwhile block's hash, the existent tokens, and optional metadata.

optional metadata includes: MM hash, LoRA ID, cache brackish (injected into hash of the first artifact ensures only requests pinch this cache brackish tin reuse blocks).

3. Each consequence is stored arsenic a BlockHash entity containing some the hash and its token IDs. We return a database of artifact hashes.

The database is stored successful self.req_to_block_hashes[request_id].

Next, the motor calls find_longest_cache_hit to cheque if immoderate of these hashes already beryllium successful cached_block_hash_to_block. On the first request, nary hits are found.

Then we telephone allocate_slots which calls coordinator.cache_blocks, which associates the caller BlockHash entries pinch allocated KV blocks and records them successful cached_block_hash_to_block.

Afterwards, the guardant walk will populate KVs successful paged KV cache representation corresponding to KV cache blocks that we allocated above.

On a 2nd make telephone pinch the aforesaid prefix, steps 1-3 repeat, but now find_longest_cache_hit finds matches for each n blocks (via linear search). The motor tin reuse those KV blocks directly.

If the original petition were still alive, the reference count for those blocks would increment (e.g. to 2). In this example, the first petition has already completed, truthful the blocks were freed backmost to the excavation and their reference counts group backmost to 0. Because we were capable to retrieve them from cached_block_hash_to_block we cognize they're valid (the logic of the KV cache head is setup successful specified a way), truthful we conscionable region them from free_block_queue again.

And that's the gist of prefix caching: don't recompute prefixes you've already seen — conscionable reuse their KV cache!

Prefix caching is enabled by default. To disable it: enable_prefix_caching = False.

Guided Decoding (FSM)

Guided decoding is simply a method where, astatine each decoding step, the logits are constrained by a grammar-based finite authorities machine. This ensures that only tokens allowed by the grammar tin beryllium sampled.

It's a powerful setup: you tin enforce thing from regular grammars (Chomsky type-3, e.g. arbitrary regex patterns) each the measurement up to context-free grammars (type-2, which screen astir programming languages).

To make this little abstract, let's commencement pinch the simplest imaginable example, building connected our earlier code:

In the artifact illustration I gave (assume character-level tokenization): astatine prefill, the FSM masks logits truthful only "P" aliases "N" are viable. If "P" is sampled, the FSM moves to the "Positive" branch; adjacent measurement only "o" is allowed, and truthful on.

How this useful successful vLLM:

1. At LLM motor construction, a StructuredOutputManager is created; it has entree to the tokenizer and maintains a _grammar_bitmask tensor.
2. When adding a request, its position is group to WAITING_FOR_FSM and grammar_init selects the backend compiler (e.g., xgrammar [7]; statement that backends are 3rd statement code).
3. The grammar for this petition is compiled asynchronously.
4. During scheduling, if the async compile has completed, the position switches to WAITING and request_id is added to structured_output_request_ids; different it's placed successful skipped_waiting_requests to retry connected adjacent motor step.
5. After the scheduling loop (still wrong scheduling), if location are FSM requests, the StructuredOutputManager asks the backend to prepare/update _grammar_bitmask.
6. After the guardant walk produces logits, xgr_torch_compile's usability expands the bitmask to vocab size (32x description ratio because we usage 32 spot integers) and masks disallowed logits to –∞.
7. After sampling the adjacent token, the request's FSM is precocious via accept_tokens. Visually we move to the adjacent authorities connected the FSM diagram.

Step 6 deserves further clarification.

If vocab_size = 32, _grammar_bitmask is simply a azygous integer; its binary practice encodes which tokens are allowed ("1") vs disallowed ("0"). For example, "101…001" expands to a length-32 array [1, 0, 1, …, 0, 0, 1]; positions pinch 0 get logits group to –∞. For larger vocabularies, aggregate 32-bit words are utilized and expanded/concatenated accordingly. The backend (e.g., xgrammar) is responsible for producing these spot patterns utilizing the existent FSM state.

Here is an moreover simpler illustration pinch vocab_size = 8 and 8-bit integers (for those of you who for illustration my visuals):

You tin alteration this successful vLLM by passing successful a desired guided_decoding config.

Speculative Decoding

In autoregressive generation, each caller token requires a guardant walk of the ample LM. This is costly — each measurement reloads and applies each exemplary weights conscionable to compute a azygous token! (assuming batch size == 1, successful wide it's B)

Speculative decoding [8] speeds this up by introducing a smaller draught LM. The draught proposes k tokens cheaply. But we don't yet want to sample from the smaller exemplary — it's only location to conjecture campaigner continuations. The ample exemplary still decides what's valid.

Here are the steps:

1. Draft: tally the mini exemplary connected the existent discourse and propose k tokens
2. Verify: tally the ample exemplary erstwhile connected discourse + k draught tokens. This produces probabilities for those k positions positive 1 other (so we get k+1 candidates)
3. Accept/reject: going from near to correct complete the k draught tokens:
   • If the ample model's probability for the draught token ≥ the draft's probability, judge it
   • Otherwise, judge it pinch probability p_large(token)/p_draft(token)
   • Stop astatine the first rejection, aliases judge each k draught tokens.
   • If each k draught tokens are accepted, besides sample the other (k+1)-th token "for free" from the ample exemplary (we already computed that distribution).
   • If location was a rejection create a caller rebalanced distribution astatine that position (p_large - p_draft, clamp min astatine 0, normalize to sum to 1) and sample the past token from it.

Why this works: Although we usage the mini exemplary to propose candidates, the accept/reject norm guarantees that successful anticipation the series is distributed precisely arsenic if we had sampled token by token from the ample model. This intends speculative decoding is statistically balanced to modular autoregressive decoding — but perchance overmuch faster, since a azygous large-model walk tin output up to k+1 tokens.

vLLM V1 does not support the LLM draught exemplary method, alternatively it implements faster—but little accurate—proposal schemes: n-gram, EAGLE [9], and Medusa [10].

One-liners connected each:

1. n-gram: return the past prompt_lookup_max tokens; find a anterior lucifer successful the sequence; if found, propose the k tokens that followed that match; different decrement the model and retry down to prompt_lookup_min

The existent implementation returns k tokens aft the first match. It feels much earthy to present a recency bias and reverse the hunt direction? (i.e. past match)

2. Eagle: execute "model surgery" connected the ample LM—keep embeddings and LM head, switch the transformer stack pinch a lightweight MLP; fine-tune that arsenic a inexpensive draft
3. Medusa: train auxiliary linear heads connected apical (embeddings earlier LM head) of the ample exemplary to foretell the adjacent k tokens successful parallel; usage these heads to propose tokens much efficiently than moving a abstracted mini LM

Here's really to invoke speculative decoding successful vLLM utilizing ngram arsenic the draught method:

How does this activity successful vLLM?

Setup (during motor construction):

1. Init device: create a drafter (draft model, e.g., NgramProposer) and a rejection_sampler (parts of it are written successful Triton).
2. Load model: load draught exemplary weights (no-op for n-gram).

After that successful the make function (assume we get a marque caller request):

1. Run the regular prefill measurement pinch the ample model.
2. After the guardant walk and modular sampling, telephone propose_draft_token_ids(k) to sample k draught tokens from the draught model.
3. Store these successful request.spec_token_ids (update the petition metadata).
4. On the adjacent motor step, erstwhile the petition is successful the moving queue, adhd len(request.spec_token_ids) to the "new tokens" count truthful allocate_slots reserves capable KV blocks for the fwd pass.
5. Copy spec_token_ids into input_batch.token_ids_cpu to shape (context + draft) tokens.
6. Compute metadata via _calc_spec_decode_metadata (this copies complete tokens from input_batch.token_ids_cpu, prepares logits, etc.), past tally a large-model guardant walk complete the draught tokens.
7. Instead of regular sampling from logits, usage the rejection_sampler to accept/reject left-to-right and nutrient output_token_ids.
8. Repeat steps 2-7 until a extremity information is met.

The champion measurement to internalize this is to occurrence up your debugger and measurement done the codebase, but this conception hopefully gives you a sensation for it. This arsenic well:

Disaggregated P/D

I've already antecedently hinted astatine the information down disaggregated P/D (prefill/decode).

Prefill and decode person very different capacity profiles (compute-bound vs. memory-bandwidth-bound), truthful separating their execution is simply a sensible design. It gives tighter power complete latency — some TTFT (time-to-first-token) and ITL (inter-token latency) — much connected this successful the benchmarking section.

In practice, we tally N vLLM prefill instances and M vLLM decode instances, autoscaling them based connected the unrecorded petition mix. Prefill workers constitute KV to a dedicated KV-cache service; decode workers publication from it. This isolates long, bursty prefill from steady, latency-sensitive decode.

How does this activity successful vLLM?

For clarity, the illustration beneath relies connected SharedStorageConnector, a debugging connector implementation utilized to exemplify the mechanics.

We motorboat 2 vLLM instances (GPU 0 for prefill and GPU 1 for decode), and past transportation the KV cache betwixt them:

These are the steps successful vLLM:

1. Instantiation — During motor construction, connectors are created successful 2 places:
   • Inside the worker's init instrumentality process (under init worker distributed situation function), pinch domiciled "worker".
   • Inside the scheduler constructor, pinch domiciled "scheduler".
2. Cache lookup — When the scheduler processes prefill requests from the waiting queue (after section prefix-cache checks), it calls connector's get_num_new_matched_tokens. This checks for externally cached tokens successful the KV-cache server. Prefill ever sees 0 here; decode whitethorn person a cache hit. The consequence is added to the section count earlier calling allocate_slots.
3. State update — The scheduler past calls connector.update_state_after_alloc, which records requests that had a cache (no-op for prefill).
4. Meta build — At the extremity of scheduling, the scheduler calls meta = connector.build_connector_meta:
   • Prefill adds each requests pinch is_store=True (to upload KV).
   • Decode adds requests pinch is_store=False (to fetch KV).
5. Context manager — Before the guardant pass, the motor enters a KV-connector discourse manager:
   • On enter: kv_connector.start_load_kv is called. For decode, this loads KV from the outer server and injects it into paged memory. For prefill, it's a no-op.
   • On exit: kv_connector.wait_for_save is called. For prefill, this blocks until KV is uploaded to the outer server. For decode, it's a no-op.

Here is simply a ocular example:

From UniprocExecutor to MultiProcExecutor

With the halfway techniques successful place, we tin now talk astir scaling up.

Suppose your exemplary weights nary longer fresh into a azygous GPU's VRAM.

The first action is to shard the exemplary crossed aggregate GPUs connected the aforesaid node utilizing tensor parallelism (e.g., TP=8). If the exemplary still doesn't fit, the adjacent measurement is pipeline parallelism crossed nodes.

At this stage, we request aggregate GPU processes (workers) and an orchestration furniture to coordinate them. That's precisely what MultiProcExecutor provides.

How this useful successful vLLM:

1. MultiProcExecutor initializes an rpc_broadcast_mq connection queue (implemented pinch shared representation nether the hood).
2. The constructor loops complete world_size (e.g. TP=8 ⇒ world_size=8) and spawns a daemon process for each rank via WorkerProc.make_worker_process.
3. For each worker, the genitor first creates a scholar and writer pipe.
4. The caller process runs WorkerProc.worker_main, which instantiates a worker (going done the aforesaid "init device", "load model", etc. arsenic successful UniprocExecutor).
5. Each worker determines whether it is the driver (rank 0 successful the TP group) aliases a regular worker. Every worker sets up 2 queues:
   • rpc_broadcast_mq (shared pinch the parent) for receiving work.
   • worker_response_mq for sending responses back.
6. During initialization, each kid sends its worker_response_mq grip to the genitor via the pipe. Once each are received, the genitor unblocks — this completes coordination.
7. Workers past participate a engaged loop, blocking connected rpc_broadcast_mq.dequeue. When a activity point arrives, they execute it (just for illustration successful UniprocExecutor, but now pinch TP/PP-specific walled work). Results are sent backmost done worker_response_mq.enqueue.
8. At runtime, erstwhile a petition arrives, MultiProcExecutor enqueues it into rpc_broadcast_mq (non-blocking) for each children workers. It past waits connected the designated output rank's worker_response_mq.dequeue to cod the last result.

From the engine's perspective, thing has changed — each of this multiprocessing complexity is abstracted distant done a telephone to exemplary executor's execute_model.

• In the UniProcExecutor case: execute_model straight leads to calling execute_model connected the worker
• In the MultiProcExecutor case: execute_model indirectly leads to calling execute_model connected each worker done rpc_broadcast_mq

At this point, we tin tally models that are arsenic ample arsenic resources let utilizing the aforesaid motor interface.

The adjacent measurement is to standard out: alteration information parallelism (DP > 1) replicating the exemplary crossed nodes, adhd a lightweight DP coordination layer, present load balancing crossed replicas, and spot 1 aliases much API servers successful beforehand to grip incoming traffic.

Distributed strategy serving vLLM

There are galore ways to group up serving infrastructure, but to enactment concrete, here's 1 example: suppose we person 2 H100 nodes and want to tally 4 vLLM engines crossed them.

If the exemplary requires TP=4, we tin configure the nodes for illustration this.

On the first node, tally the motor successful headless mode (no API server) pinch the pursuing arguments:

and tally that aforesaid bid connected the different node pinch fewer tweaks:

• no --headless
• modify DP commencement rank

How does this activity successful VLLM?

On the headless server node

On the headless node, a CoreEngineProcManager launches 2 processes (per --data-parallel-size-local) each moving EngineCoreProc.run_engine_core. Each of these functions creates a DPEngineCoreProc (the motor core) and past enters its engaged loop.

DPEngineCoreProc initializes its genitor EngineCoreProc (child of EngineCore), which:

1. Creates an input_queue and output_queue (queue.Queue).
2. Performs an first handshake pinch the frontend connected the different node utilizing a DEALER ZMQ socket (async messaging lib), and receives coordination reside info.
3. Initializes DP group (e.g. utilizing NCCL backend).
4. Initializes the EngineCore pinch MultiProcExecutor (TP=4 connected 4 GPUs arsenic described earlier).
5. Creates a ready_event (threading.Event).
6. Starts an input deamon thread (threading.Thread) moving process_input_sockets(…, ready_event). Similarly starts an output thread.
7. Still successful the main thread, waits connected ready_event until each input threads crossed each 4 processes (spanning the 2 nodes) person completed the coordination handshake yet executing ready_event.set().
8. Once unblocked, sends a "ready" connection to the frontend pinch metadata (e.g., num_gpu_blocks disposable successful paged KV cache memory).
9. The main, input, and output threads past participate their respective engaged loops.

TL;DR: We extremity up pinch 4 kid processes (one per DP replica), each moving a main, input, and output thread. They complete a coordination handshake pinch the DP coordinator and frontend, past each 3 threads per process tally successful steady-state engaged loops.

Current dependable state:

• Input thread — blocks connected the input socket until a petition is routed from the API server; upon receipt, it decodes the payload, enqueues a activity point via input_queue.put_nowait(...), and returns to blocking connected the socket.
• Main thread — wakes connected input_queue.get(...), feeds the petition to the engine; MultiProcExecutor runs the guardant walk and enqueues results to output_queue.
• Output thread — wakes connected output_queue.get(...), sends the consequence backmost to the API server, past resumes blocking.

Additional mechanics:

• DP activity counter — the strategy tracks "waves"; erstwhile each engines go idle they quiesce, and the antagonistic increments erstwhile caller activity arrives (useful for coordination/metrics).
• Control messages — the API server tin nonstop much than conscionable conclusion requests (e.g., aborts and utility/control RPCs).
• Dummy steps for lockstep — if immoderate DP replica has work, each replicas execute a guardant step; replicas without requests execute a dummy measurement to participate successful required synchronization points (avoids blocking the progressive replica).

Now for the 2nd part, what happens connected the API server node?

On the API server node

We instantiate an AsyncLLM entity (an asyncio wrapper astir the LLM engine). Internally this creates a DPLBAsyncMPClient (data-parallel, load-balancing, asynchronous, multiprocessing client).

Inside the genitor people of MPClient, the launch_core_engines usability runs and:

1. Creates the ZMQ addresses utilized for the startup handshake (as seen connected the headless node).
2. Spawns a DPCoordinator process.
3. Creates a CoreEngineProcManager (same arsenic connected the headless node).

Inside AsyncMPClient (child of MPClient), we:

1. Create an outputs_queue (asyncio.Queue).
2. We create an asyncio task process_outputs_socket which communicates (through the output socket) pinch output threads of each 4 DPEngineCoreProc and writes into outputs_queue.
3. Subsequently 1 much asyncio task output_handler from AsyncLLM sounds from this queue and yet sends retired accusation to the create_completion function.

Inside DPAsyncMPClient we create an asyncio task run_engine_stats_update_task which communicates pinch DP coordinator.

The DP coordinator mediates betwixt the frontend (API server) and backend (engine cores). It:

• Periodically sends load-balancing info (queue sizes, waiting/running requests) to the frontend's run_engine_stats_update_task.
• Handles SCALE_ELASTIC_EP commands from the frontend by dynamically changing the number of engines (only useful pinch Ray backend).
• Sends START_DP_WAVE events to the backend (when triggered by frontend) and reports wave-state updates back.

To recap, the frontend (AsyncLLM) runs respective asyncio tasks (remember: concurrent, not parallel):

• A people of tasks handles input requests done the make way (each caller customer petition spawns a caller asyncio task).
• Two tasks (process_outputs_socket, output_handler) process output messages from the underlying engines.
• One task (run_engine_stats_update_task) maintains connection pinch the DP coordinator: sending activity triggers, polling LB state, and handling move scaling requests.

Finally, the main server process creates a FastAPI app and mounts endpoints specified arsenic OpenAIServingCompletion and OpenAIServingChat, which expose /completion, /chat/completion, and others. The stack is past served via Uvicorn.

So, putting it each together, here's the afloat petition lifecycle!

You nonstop from your terminal:

What happens next:

1. The petition hits OpenAIServingCompletion's create_completion way connected the API server.
2. The usability tokenizes the punctual asynchronously, and prepares metadata (request ID, sampling params, timestamp, etc.).
3. It past calls AsyncLLM.generate, which follows the aforesaid travel arsenic the synchronous engine, yet invoking DPAsyncMPClient.add_request_async.
4. This successful move calls get_core_engine_for_request, which does load balancing crossed engines based connected the DP coordinator's authorities (picking the 1 that has minimal people / lowest load: people = len(waiting) * 4 + len(running)).
5. The ADD petition is sent to the chosen engine's input_socket.
6. At that engine:
   • Input thread — unblocks, decodes information from the input socket, and places a activity point connected the input_queue for the main thread.
   • Main thread — unblocks connected input_queue, adds the petition to the engine, and many times calls engine_core.step(), enqueueing intermediate results to output_queue until a extremity information is met.

   Reminder: step() calls the scheduler, exemplary organizer (which successful move tin beryllium MultiProcExecutor!), etc. We person already seen this!

   • Output thread — unblocks connected output_queue and sends results backmost done the output socket.
7. Those results trigger the AsyncLLM output asyncio tasks (process_outputs_socket and output_handler), which propagate tokens backmost to FastAPI's create_completion route.
8. FastAPI attaches metadata (finish reason, logprobs, usage info, etc.) and returns a JSONResponse via Uvicorn to your terminal!

And conscionable for illustration that, your completion came backmost — the full distributed machinery hidden down a elemental curl command! :) So overmuch fun!!!

Benchmarks and auto-tuning - latency vs throughput

So acold we've been analyzing the "gas particles" — the internals of really requests travel done the engine/system. Now it's clip to zoom retired and look astatine the strategy arsenic a whole, and ask: really do we measurement the capacity of an conclusion system?

At the highest level location are 2 competing metrics:

1. Latency — the clip from erstwhile a petition is submitted until tokens are returned
2. Throughput — the number of tokens/requests per 2nd the strategy tin generate/process

Latency matters astir for interactive applications, wherever users are waiting connected responses.

Throughput matters successful offline workloads for illustration synthetic information procreation for pre/post-training runs, information cleaning/processing, and successful wide - immoderate type of offline batch conclusion jobs.

Before explaining why latency and throughput compete, let's specify a fewer communal conclusion metrics:

Here is simply a simplified exemplary explaining the competing quality of these 2 metrics.

The tradeoff becomes clear erstwhile looking astatine really batch size B affects a azygous decode step. As B ↓ toward 1, ITL drops: there's little activity per measurement and the token isn't "competing" pinch others. As B ↑ toward infinity, ITL rises because we do much FLOPs per step—but throughput improves (until we deed highest perf) because weight I/O is amortized crossed much tokens.

A roofline exemplary helps pinch knowing here: beneath a saturation batch B_sat, the measurement clip is dominated by HBM bandwidth (streaming weights layer-by-layer into on-chip memory), truthful measurement latency is astir flat—computing 1 vs 10 tokens tin return a akin time. Beyond B_sat, the kernels go compute-bound and measurement clip grows astir pinch B; each other token adds to ITL.

How to benchmark successful vLLM

vLLM provides a vllm chair {serve,latency,throughput} CLI that wraps vllm / benchmarks / {server,latency,throughput}.py.

Here is what the scripts do:

• latency — uses a short input (default 32 tokens) and samples 128 output tokens pinch a mini batch (default 8). It runs respective iterations and reports e2e latency for the batch.
• throughput — submits a fixed group of prompts (default: 1000 ShareGPT samples) each astatine erstwhile (aka arsenic QPS=Inf mode), and reports input/output/total tokens and requests per 2nd crossed the run.
• serve — Launches a vLLM server and simulates a real-world workload by sampling petition inter-arrival times from a Poisson (or much generally, Gamma) distribution. It sends requests complete a clip window, measures each the metrics we’ve discussed, and tin optionally enforce a server-side max concurrency (via a semaphore, e.g. limiting the server to 64 concurrent requests).

Here is an illustration of really you tin tally the latency script:

There is besides an auto-tune book that drives the service benchmark to find statement settings that meet target SLOs (e.g., "maximize throughput while keeping p99 e2e < 500 ms"), returning a suggested config.

Epilogue

We began pinch the basal motor halfway (UniprocExecutor), added precocious features for illustration speculative decoding and prefix caching, scaled up to MultiProcExecutor (with TP/PP > 1), and yet scaled out, wrapped everything successful the asynchronous motor and distributed serving stack—closing pinch really to measurement strategy performance.

vLLM besides includes specialized handling that I've skipped. E.g.:

• Diverse hardware backends: TPUs, AWS Neuron (Trainium/Inferentia), etc.
• Architectures/techniques: MLA, MoE, encoder-decoder (e.g., Whisper), pooling/embedding models, EPLB, m-RoPE, LoRA, ALiBi, attention-free variants, sliding-window attention, multimodal LMs, and state-space models (e.g., Mamba/Mamba-2, Jamba)
• TP/PP/SP
• Hybrid KV-cache logic (Jenga), much analyzable sampling methods for illustration beam sampling, and more
• Experimental: async scheduling

The bully point is that astir of these are orthogonal to the main travel described above—you tin almost dainty them for illustration "plugins" (in believe there's immoderate coupling, of course).

I emotion knowing systems. Having said that, the solution decidedly suffered astatine this altitude. In the adjacent posts I'll zoom successful connected circumstantial subsystems and get into the nitty-gritty details.

Acknowledgements

A immense convey you to Hyperstack for providing maine pinch H100s for my experiments complete the past year!

Thanks to Nick Hill (core vLLM contributor, RedHat), Mark Saroufim (PyTorch), Kyle Krannen (NVIDIA, Dynamo), and Ashish Vaswani for reference pre-release type of this blog station and providing feedback!

References

1. vLLM https://github.com/vllm-project/vllm
2. "Attention Is All You Need", https://arxiv.org/abs/1706.03762
3. "Efficient Memory Management for Large Language Model Serving pinch PagedAttention", https://arxiv.org/abs/2309.06180
4. "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model", https://arxiv.org/abs/2405.04434
5. "Jenga: Effective Memory Management for Serving LLM pinch Heterogeneity", https://arxiv.org/abs/2503.18292
6. "Orca: A Distributed Serving System for Transformer-Based Generative Models", https://www.usenix.org/conference/osdi22/presentation/yu
7. "XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models", https://arxiv.org/abs/2411.15100
8. "Accelerating Large Language Model Decoding pinch Speculative Sampling", https://arxiv.org/abs/2302.01318
9. "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty", https://arxiv.org/abs/2401.15077
10. "Medusa: Simple LLM Inference Acceleration Framework pinch Multiple Decoding Heads", https://arxiv.org/abs/2401.10774
11. LMCache, https://github.com/LMCache/LMCache