The efficient frontier of LLM inference

In the AI industry, we borrowed the word “efficient frontier” from economists. We usage it to talk astir managing tradeoffs, astir often the tradeoff betwixt costs and capabilities for models. A exemplary is simply a “frontier model” if it offers the highest grade of intelligence astatine a fixed costs aliases size.

We besides person businesslike frontiers successful conclusion engineering. Most often, this is expressed arsenic a tradeoff betwixt latency and throughput (which determines cost), though we tin besides speech value for throughput (via quantization, distillation, and pruning) aliases intelligence for velocity (in the shape of reasoning level).

There are 2 types of techniques disposable to conclusion engineers:

1. Techniques which make a tradeoff betwixt 2 factors to move a deployment on an businesslike frontier.

2. Techniques which push retired the full frontier for a fixed deployment, creating much wide ratio which tin beryllium allocated to immoderate result is astir beneficial.

Both types of techniques are valuable.

It’s useful to beryllium capable to target immoderate constituent on an businesslike frontier by making tradeoffs. Giving up per-user velocity makes it imaginable to build high-throughput, low-cost pipelines for batch workloads. Sacrificing throughput to amended velocity makes consciousness erstwhile latency-sensitive users person a precocious willingness to pay.

And of course, it’s incredibly useful to push retired the full frontier. Unlocking much ratio creates gains that tin beryllium allocated to little latency, higher throughput, aliases a operation of the two.

This article specifications which conclusion engineering techniques fto you target a constituent connected the frontier, and which techniques push the full frontier out. For this article, we’ll presume we’re moving an LLM for illustration GLM-5.3 aliases Kimi K3 for agentic coding pinch KV cache reuse enabled and optimal KV-aware routing.

Techniques that negociate tradeoffs

Hitting a definite target successful accumulation is often little astir discovering immoderate caller attack and much astir uncovering the correct group of configurations fixed the quality of the traffic.

In practice, the businesslike frontier is very jagged. Rather than a smooth, continuous statement betwixt outcomes, mini changes tin person large impacts. These cutoff points are often unintuitive and must beryllium discovered empirically done sweeps.

Batch sizing

The astir evident tradeoff betwixt latency and throughput comes from batch sizing. A batch is the number of requests that are processed concurrently. While token-level continuous batching intends that location isn’t immoderate latency from waiting for batches to start, the configured batch size determines the per-user latency and the wide throughput.

With mini batch sizes, per-user latency is excellent, but fewer full tokens are generated per GPU. This intends the costs per token is rather high. Increasing batch size has the other effect: worse per-user latencies, amended wide throughput for little cost.

Parallelism strategy

Today’s LLMs measurement successful the hundreds of billions aliases trillions of parameters and must beryllium dispersed crossed aggregate GPUs. The measurement successful which they are shared, aliases parallelized, crossed GPUs tin boost either latency aliases throughput.

For latency-sensitive deployments, attraction connected expanding Tensor Parallelism (TP). While TP has costly all-to-all communication, it is effective for lowering latencies arsenic these operations are accelerated complete high-bandwidth NVLink interconnects.

Expert Parallelism (EP) tin thief pinch some latency and throughput. A little grade of EP is often associated pinch amended latencies, while wide EP, including EP crossed a afloat rack of GPUs, mostly supports higher throughput.

Another parallelism method for improving throughput is Attention Data Parallelism (ADP). This method replicates attraction layers for parallel computation, which boosts strategy throughput astatine the disbursal of per-request speed.

Quantization

Quantization, aliases moving a exemplary pinch a little level of precision successful weights, activations, and/or KV cache values, improves some latency and throughput. A quantized exemplary pushes retired the businesslike frontier connected serving tradeoffs.

However, quantization introduces a caller group of tradeoffs betwixt value and serving efficiency. This is simply a peculiarly jagged frontier, wherever a ample grade of betterment to serving ratio is imaginable pinch little-to-no simplification successful exemplary quality, particularly erstwhile utilizing microscaling floating-point number formats for illustration MXFP4 and NVFP4.

Techniques that move the frontier

These techniques are the ones that make the headlines. Improving wide capacity is the astir nosy portion of conclusion engineering.

The champion portion is that these techniques often compound. For example, doubling capacity from amended hardware while besides doubling capacity from amended package intends a 4 times betterment successful wide serving, which tin beryllium allocated crossed latency and throughput.

Kernel optimization and runtime improvements

A CUDA kernel is simply a low-level usability that executes a azygous portion of the conclusion process, for illustration a matrix multiplication. Improving the capacity of individual kernels, arsenic good arsenic the end-to-end capacity of a guardant walk successful the conclusion engine, intends less resources are needed to make each token. These ratio gains compound passim the stack and push the frontier of performance.

For much connected kernel-level performance, publication this excellent writeup by Baseten intern Brian Li.

Speculative decoding

Speculative decoding is the process of guessing which tokens a exemplary mightiness generate, past validating those guesses. When speculative decoding was new, this posed a tradeoff betwixt latency and throughput: speculation was expensive, series lengths were short, and acceptance rates were low, meaning speculative decoding was only feasible astatine mini batch sizes.

Today, techniques for illustration EAGLE-3, DSpark, and DFlash still compete pinch the main exemplary loop for resources, somewhat limiting maximum batch sizes. However, acknowledgment to the beardown capacity of these techniques, particularly connected codification procreation wherever output token sequences are comparatively predictable, they output ratio gains from skipped guardant passes successful summation to the earthy simplification successful latency successful the shape of much tokens per 2nd per user.

Disaggregation

P/D disaggregation, aliases separating prefill and decode onto dedicated workers, is simply a strategy for optimizing high-volume deployments of LLMs. Running prefill and decode independently intends that workers tin beryllium optimized for the unsocial characteristics of each shape of inference, and that the ratio betwixt prefill and decode workers tin beryllium adjusted to lucifer the input and output series lengths and cache deed rates from incoming traffic.

This article provided a basal overview of techniques for managing tradeoffs versus techniques for improving systemwide performance. For much item connected each method mentioned successful this article, publication my free book Inference Engineering.