p50 vs p99 Latency: Why Median Benchmarks Mislead AI Agent Workloads

Most published conclusion benchmarks lead pinch 1 number. A median clip to first token. A highest tokens-per-second figure. These numbers are not false. They are measured nether conditions that flatter the provider: debased concurrency, a lukewarm instance, a short prompt. The spread betwixt that information and your accumulation postulation is wherever your latency surprises live.

This article makes 2 arguments, and states up beforehand which 1 is new. For a azygous interactive request, the style of the latency distribution matters much than its center: p95 and p99 picture what a existent fraction of your users feel, and the median describes the petition cipher complains about. That statement is not new, and DigitalOcean has already published bully activity connected it, cited passim this piece.

What is caller present is the 2nd argument: for an supplier that makes a concatenation of sequential exemplary calls to complete 1 task, per-call latency is adjacent to the incorrect portion of analysis. Task completion clip is simply a sum crossed calls, and the probability that astatine slightest 1 telephone successful that concatenation lands successful the tail grows pinch each further call.

In this article I will screen the following: really to publication a percentile correctly, really to decompose latency into the metric that matches your workload, why supplier chains alteration the study entirely, a checklist for reference immoderate published benchmark critically, and a trial protocol that I ran against DigitalOcean’s Serverless Inference pinch each measured number and the nonstop book included, truthful you tin rerun it against your ain postulation style alternatively of trusting anyone’s header number, including mine.

Terms Used successful This Article

If you are caller to latency analysis, skim this erstwhile and mention backmost arsenic needed.

The only look this article uses is P = 1 − (1 − q)^n, the probability that astatine slightest 1 of n calls hits a tail arena erstwhile each telephone has an independent chance q of doing so. It is explained measurement by measurement wherever it first appears, successful the supplier multiplier section.

TL;DR

• A cleanable median pinch a bad tail is not a rounding error. It is simply a fixed fraction of your postulation having a bad experience, each day, by construction. At 10,000 regular requests, a 1 successful 100 tail arena intends 100 users deed it daily, sloppy of really bully the median looks. This is simply a mathematical identity, not a benchmark claim; I verified it by nonstop calculation and show the look successful the conception beneath alternatively than asking you to return it connected faith.
• For a azygous request, 3 different metrics matter for 3 different workloads. Time to first token for streaming chat UX, inter-token latency for sound and agelong streamed output, and full completion clip for agents and batch jobs. Definitions per Artificial Analysis’s published methodology. Naming “the fastest provider” without naming which of these 3 you mean is not a complete claim.
• For supplier pipelines, the mathematics changes. A concatenation of sequential calls sums their latencies, and the probability that astatine slightest 1 telephone lands successful the tail rises pinch concatenation length. I checked a communal declare astir really quickly this reaches near-certainty and recovered the numbers do not support it: astatine a 1 percent per-call tail rate, a 10-call concatenation gives 9.6 percent, not near-certainty, verified by nonstop calculation and shown successful afloat successful the conception below. The corrected mathematics is still a real, worldly risk, conscionable not the consequence arsenic commonly overstated.
• A exemplary that wins the header benchmark tin suffer the task, and I measured this happening alternatively than simulating it. On DigitalOcean Serverless Inference connected July 27, 2026, Llama 4 Maverick won the single-call time-to-first-token comparison against GPT-OSS 120B by 4 times (300 sclerosis median versus 1,202 ms). On the measured 10-call supplier chain, the ranking inverted: GPT-OSS 120B vanished the task successful 20.9 seconds astatine the median against Maverick’s 26.7 seconds, and Maverick’s worst measured concatenation took complete 20 minutes because 1 call’s watercourse stalled mid-generation.
• A repeatable trial protocol thumps trusting immoderate azygous published number, including the ones successful this article. I ran the afloat protocol connected 3 models and a bounded follow-up connected a fourth, much than 1,500 recorded requests from a GPU droplet successful the aforesaid cloud, and each number, each caveat, and the nonstop book are included beneath truthful you tin rerun it against your ain postulation shape. The complete grounds base, harness, earthy per-request data, console logs, study code, and charts, is nationalist successful the serverless-inference-tail-latency-study GitHub repository I created for this research.

What users really feel: the lawsuit for tail metrics

State this plainly earlier thing else. If your median clip to first token is simply a cleanable 300 milliseconds but your p99 is 4 seconds, and you service 10,000 requests a day, astir 100 of those requests each time deed a hold that is much than 10 times longer than the number connected your dashboard. This is not noise. A percentile is simply a definition, not a measurement artifact: the p99 is the worth that 1 percent of your postulation exceeds, by construction, each azygous time you tally astatine that rate.

For a azygous request, this is already a logic to look past the median. For a convention pinch much than 1 request, the vulnerability compounds. If each petition successful a convention has an independent 1 percent chance of landing successful the tail, a 20-request convention has a chance of hitting astatine slightest 1 tail arena of:

P(at slightest 1 tail event) = 1 − (1 − 0.01)^20 ≈ 18.2%

I verified this by nonstop calculation. Close to 1 successful 5 sessions of that magnitude will incorporate astatine slightest 1 bad request, moreover though immoderate azygous petition only has a 1 percent chance of being bad. This is the system this full portion is built on: independent mini risks, repeated capable times, extremity being small.

Why p50 and p99 diverge connected conclusion specifically

The divergence has a mini number of concrete, mechanistic causes, each covered successful extent successful the companion portion connected conclusion silicon and architecture. Briefly:

Queueing nether concurrency intends your request’s commencement clip depends connected really galore different requests are up of it, not conscionable connected really accelerated the exemplary runs. Batching dynamics mean your petition whitethorn hold for a batch model to capable earlier the GPU starts moving connected it astatine all. A long-prompt neighbour sharing your batch aliases your GPU tin widen your prefill clip moreover if your ain punctual is short, since prefill is compute-bound and a ample concurrent punctual consumes the compute your petition needed. Cold starts connected scale-to-zero infrastructure adhd a afloat exemplary aliases adapter load clip to whichever petition happens to beryllium first aft an idle period. DigitalOcean’s ain study of fine-tuned serverless deployments recovered acold starts spike time-to-first-token specifically, not the steady-state procreation rate, and that periodic keep-alive requests are 1 applicable mitigation. Source: Fine-Tuned LLMs connected Serverless Architecture, DigitalOcean Community.

The tail is simply a mini area nether the curve. At existent postulation volumes it is besides a fixed, repeating count of existent requests, not a rounding error.

When the median is the correct number

High-volume, latency-insensitive batch workloads, information suites, offline summarization, synthetic information generation, and contented moderation astatine scale, attraction astir throughput and costs per token, not astir immoderate individual request’s hold time, since nary personification is watching a spinner. DigitalOcean’s ain batch conclusion explicitly isolates this postulation from real-time serving. Its documented limits authorities that batch scheduling does not degrade real-time conclusion p99 latency by much than 5 percent, which is the platform’s ain committedness that these are different workload classes pinch different metrics. Source: DigitalOcean Inference Limits. If your workload looks for illustration this, throughput and costs per successful petition are the correct metrics, and the remainder of this portion does not alteration your evaluation.

TTFT, inter-token latency, and full time: prime the metric for your workload

Latency is not 1 number. It decomposes into 3 measurements, and each 1 dominates a different workload.

1. Time to first token is the hold betwixt sending a petition and receiving the first output token. This is what a chat interface’s personification stares astatine earlier thing appears connected screen, and it is dominated by prefill, the compute-bound measurement of processing your full input punctual earlier procreation starts.

2. Inter-token latency, sometimes called clip per output token, is the spread betwixt each consequent token erstwhile procreation is underway. This is what determines whether a agelong streamed consequence feels soft aliases halting, and it matters astir for sound interfaces and long-form streamed answers, wherever the scholar aliases listener is consuming tokens continuously alternatively than waiting once.

3. Total completion time is TTFT positive the afloat procreation time, and it is the only metric that matters erstwhile cipher is watching the stream, which is precisely the lawsuit for a batch occupation aliases an intermediate telephone wrong an supplier pipeline. Source for these definitions: Artificial Analysis Language Model Benchmarking Methodology.

Three different clocks, 3 different workloads. A supplier optimized for 1 is not automatically beardown connected the different two.

There is nary azygous fastest provider

Asking which supplier has the fastest TTFT is an underspecified mobility until you besides authorities the concurrency, the punctual length, and which output magnitude you are measuring against, because rankings reorder crossed these conditions.

This is not a hedge. It is straight checkable against Artificial Analysis’s ain published supplier comparisons, which explicitly abstracted velocity by input token count and update their default benchmarking workload erstwhile they judge it unrepresentative. Their ain statement connected 1 comparison page states plainly that the “default capacity benchmarking workload has updated to 10k input tokens to amended bespeak accumulation usage cases,” which is Artificial Analysis admitting their ain anterior default was not typical of existent traffic.

The dispersed crossed providers serving the identical exemplary is ample capable to make “fastest provider” a meaningless building without a information attached. On GLM-5.2, Artificial Analysis reports a 963 percent quality betwixt the fastest and slowest provider’s output speed. On gpt-oss-120b, the reported dispersed is 4,149 percent, pinch Cerebras astatine 1,677.5 tokens per 2nd astatine the apical of the range. Sources: Artificial Analysis, GLM-5.2 supplier comparison and Artificial Analysis, gpt-oss-120b supplier comparison. I want to beryllium precise astir what I derived versus what was stated directly: Artificial Analysis states the percent dispersed and the fastest provider’s velocity directly. Back-solving from that percent implies a slowest supplier adjacent 45 tokens per 2nd for GLM-5.2 and adjacent 39 tokens per 2nd for gpt-oss-120b. Those 2 implied figures are my ain arithmetic from Artificial Analysis’s stated spread, not numbers Artificial Analysis published arsenic a standalone figure, and you should dainty them arsenic an estimate alternatively than a confirmed information point.

The exemplary is identical successful some cases. The supplier is not, and the spread betwixt the champion and worst supplier dwarfs immoderate quality you would find betwixt competing models.

The streaming illusion

A supplier tin triumph connected TTFT and suffer connected the metric that really determines whether your personification vanished reference earlier they gave up. A accelerated first token pinch slow inter-token latency feels responsive for the first instant and past drags, which is simply a bad floor plan for a agelong streamed reply but is adjacent to irrelevant for an agent, since an agent’s orchestration furniture is not watching tokens arrive, it is waiting for the complete consequence earlier deciding its adjacent action. This is why a azygous “fastest provider” reply does not beryllium independent of your workload. The correct mobility names the metric, the concurrency, and the token lengths, not conscionable a supplier name.

The supplier multiplier: why task completion clip is the existent metric

This is the conception wherever the mathematics has to beryllium precisely right, truthful I americium going to show my activity alternatively than asseverate a conclusion.

An supplier task seldom completes successful 1 exemplary call. A emblematic agentic workflow, instrumentality call, reasoning step, instrumentality call, synthesis, runs 5 to 20 sequential exemplary calls to decorativeness 1 task. Task latency is not immoderate azygous call’s latency. It is the sum of each telephone successful the chain, because each telephone waits connected the erstwhile 1 to find its adjacent input. This reframes the full question: the metric that predicts your user’s acquisition is not immoderate individual call’s p50 aliases p99, it is the distribution of that sum.

The corrected declare connected tail probability crossed a chain

Here is simply a declare astir this effect that I person seen stated, and want to cheque alternatively than repeat: that a 10-call concatenation against a per-call p99 tail complaint makes a tail arena a near-certainty for the task. I checked this directly, and the numbers do not support it arsenic stated.

Using the aforesaid look arsenic the single-session lawsuit above, pinch q arsenic the per-call probability of a tail arena and n arsenic the number of calls successful the chain:

P(at slightest 1 tail arena successful the chain) = 1 − (1 − q)^n

At a 1 percent per-call tail rate, the complaint implied by a p99 threshold, a 10-call concatenation gives:

1 − (0.99)^10 ≈ 9.6%

That is simply a real, worldly risk, astir 10 times higher than immoderate azygous call’s ain 1 percent rate. It is not “near-certainty” by immoderate reasonable meaning of that phrase. I computed the concatenation magnitude that would really beryllium needed to make a 1 percent per-call arena much than 95 percent apt to hap astatine slightest once: astir 298 calls.

The honorable type of this claim: supplier chains do not make p99 tail events near-certain astatine realistic lengths. They make them respective times much apt than a azygous call’s complaint suggests, and astatine a p95 threshold, the consequence crosses 50 percent by astir 14 calls, which is squarely wrong a emblematic agent’s telephone count. That is still the constituent worthy making. It is simply a smaller, much precise constituent than “near-certainty,” and it is the 1 the mathematics really supports.

Longer chains raise the likelihood of hitting a tail arena location successful the sequence, but the complaint astatine which they do depends heavy connected which percentile you are calling a tail event. Neither statement reaches the near-certainty immoderate framings of this statement declare astatine realistic supplier concatenation lengths.

The objection that matters: the benchmark victor loses the task, measured

The much useful and much original constituent is not astir the probability of a azygous bad call. It is astir what happens to the full clip crossed a concatenation erstwhile the exemplary that wins the single-call benchmark is not the exemplary that finishes the task fastest.

I ran the afloat trial protocol described later successful this article against existent models connected DigitalOcean Serverless Inference.

Two of the models measured make the constituent cleanly. On the benchmark-headline metric, median clip to first token astatine concurrency 1, Llama 4 Maverick hit GPT-OSS 120B by 4 times: 300 milliseconds against 1,202 milliseconds. If you were choosing a exemplary from a TTFT leaderboard, Maverick wins and it is not close. Then some models ran the aforesaid measured supplier workload: 30 independent chains of 10 sequential calls each, each call’s output feeding the adjacent call’s input.

The ranking inverted. Maverick’s first token arrives fast, but its procreation velocity during this model was 21 tokens per 2nd against GPT-OSS 120B’s 86, truthful its full completion clip per telephone was longer, and crossed 10 sequential calls the exemplary that “felt” 4 times faster vanished the task 28 percent slower astatine the median and 84 percent slower astatine p95. TTFT is the metric a leaderboard shows you. Task completion clip is the metric your agent’s personification waits on. They picked other winners connected the aforesaid day, connected the aforesaid platform, nether the aforesaid test.

Same 2 models, aforesaid platform, aforesaid day. The near floor plan is what a benchmark header reports. The correct floor plan is what your agent’s personification really waits for. Measured connected DigitalOcean Serverless Inference, July 27, 2026.

How to publication this chart: each sheet asks the aforesaid question, “which exemplary is faster?”, utilizing a different clock. The near sheet times only the hold for the first token of a azygous reply, successful milliseconds, and the greenish barroom (Maverick) is intelligibly shorter. The correct sheet times a complete 10-step supplier task, successful seconds, and now the greenish barroom is intelligibly taller.

The worst-measured-chain statement deserves its ain sentence, because it is the tail statement of this full portion compressed into 1 event. One of Maverick’s 300 concatenation calls delivered its first token successful 321 milliseconds, a genuinely bully TTFT, and past the watercourse stalled mid-generation and did not complete for 19.8 minutes. Every header metric would person scored that petition favorably astatine the infinitesimal it started. The concatenation containing it took 20.4 minutes alternatively of 27 seconds. That is 1 telephone successful 300, which is simply a 0.3 percent arena rate, and the look from the conception supra tells you what that does crossed chains: astatine 10 calls per chain, astir 1 task successful 30 contains one. That is not a simulated number. It is the measured logic timeouts and retries exist.

Task completion clip compounds pinch concatenation length. Lines are resampled from each model’s 75 measured single-call latencies; stars are the straight measured 10-call concatenation medians. Where the 2 disagree, spot the stars: existent chains transportation increasing punctual history and unrecorded load variety that resampling from a fixed sample cannot capture.

How to publication this chart: moving correct intends your supplier makes much sequential calls per task; moving up intends the full task takes longer. Each colour is 1 model, the coagulated statement is the emblematic (median) task, and the dashed statement supra it is the unlucky (p99) task. The vertical axis is logarithmic, truthful each gridline measurement intends astir multiplying the wait, not adding to it. Two things to return away: the spread betwixt the colors is tremendous compared to the spread betwixt immoderate 1 model’s median and its ain tail, truthful exemplary and serving prime dominate; and the stars, which are existent measured 10-call chains alternatively than projections, corroborate the ordering the lines predict.

Tail events do not conscionable slow tasks, they multiply spend

A tail arena wrong an supplier concatenation has a 2nd costs beyond the hold itself. Orchestration layers commonly instrumentality a timeout, and a petition that exceeds it triggers a retry. A retried telephone intends you salary for the tokens of the original effort and the tokens of the retry, and you sorb the hold of both. Tail latency wrong an supplier pipeline converts straight into copy spend, not conscionable a slower task.

The stalled watercourse successful the measured information supra is precisely the petition a accumulation timeout exists to catch: a customer that gave up astatine 30 seconds and retried would person paid for the copy telephone and still vanished astir 19 minutes sooner than 1 that waited. The mechanics of routing retries to a faster fallback exemplary alternatively than retrying the aforesaid slow path.

The short type is that DigitalOcean’s Inference Router tin neglect complete to an alternate exemplary automatically alternatively than resending an identical petition into the aforesaid queue. The aforesaid routing furniture is besides the costs lever this measured information points at: the concatenation results supra show that astir calls successful an supplier pipeline do not request the astir costly exemplary connected the endpoint, truthful matching each petition to the cheapest exemplary that meets its latency and value bar, and reserving heavier models for the calls that genuinely request them, is really you support some the latency fund and the per-token bill nether power astatine once.

How to publication a latency benchmark critically

Use these 7 questions to publication immoderate published latency declare critically.

• What percentile is reported?
  A median-only declare tells you astir the emblematic petition and thing astir the tail.

• What concurrency was the trial tally at?
  A number measured astatine concurrency 1 describes a champion lawsuit that queueing nether existent load will not preserve.

• What punctual and output lengths were used?
  TTFT scales pinch punctual magnitude because prefill is compute-bound, and full clip scales pinch output length.

• Was the lawsuit lukewarm aliases cold?
  A cold-start-inclusive benchmark and a warm-instance-only benchmark reply different questions.

• What clip model and long was measured?
  A azygous burst of requests during off-peak hours tin look thing for illustration sustained accumulation traffic; DigitalOcean’s ain consistency testing recovered that benchmarks tally during business hours tin look artificially bully because different users person kept a exemplary warm.

• Was streaming aliases non-streaming measurement used?
  TTFT is undefined without streaming.

• Was the measurement taken from a location that includes aliases excludes existent web path?
  A benchmark tally from infrastructure adjacent to the supplier will understate the latency a existent customer experiences.

A latency declare missing immoderate 1 of these 7 specifications is simply a declare you cannot enactment connected yet. It is not needfully a mendacious claim, conscionable an incomplete one.

Applying the checklist without the cynicism

A 2026 measurement-bias insubstantial from researchers astatine Google goes further and identifies a specific, method nonaccomplishment mode successful really galore benchmarking clients are built: single-process, asyncio-driven load generators tin deed their ain client-side queueing bottleneck nether precocious concurrency, which the authors exemplary arsenic an M/G/1 queue, and this client-side bottleneck tin inflate the very TTFT and per-token latency numbers the benchmark is trying to measure. Source: Identifying and Mitigating Systemic Measurement Bias successful Production LLM Inference Benchmarks, arXiv. The applicable accusation for you arsenic a reader: moreover a benchmark tally pinch existent intentions tin beryllium measuring its ain customer arsenic overmuch arsenic the server it targets, which is 1 much logic a documented, reproducible methodology is worthy much than a header number.

DigitalOcean’s ain organization contented has already tally this critique connected itself. Its study of supplier consistency for the aforesaid exemplary reports a coefficient of variation, modular deviation divided by mean, arsenic debased arsenic 21 percent connected a well-supported supplier and arsenic precocious arsenic 710 percent for the identical exemplary connected a different provider, a 34 times quality successful consistency for zero quality successful model. Source: Why Serverless Inference Consistency Varies connected the Same Model, DigitalOcean Community.

Source: Why Serverless Inference Consistency Varies connected the Same Model, DigitalOcean Community. That portion defines a coefficient of variety supra 100 percent arsenic the signature of acold starts and queueing variance, and supra 300 percent arsenic not production-ready for latency-sensitive work.

Predictable latency, defined properly

“Predictable latency” does not mean a debased median. It intends debased variance astir immoderate the median is, and the cleanest azygous number for this is the ratio of your p99 to your p50. A supplier pinch a p50 of 300 milliseconds and a p99 of 350 milliseconds is much predictable than a supplier pinch a p50 of 200 milliseconds and a p99 of 2,000 milliseconds, moreover though the 2nd provider’s emblematic lawsuit is faster. DigitalOcean’s ain published study of its serverless behaviour for 2 of its ain models recovered emblematic and worst-case first-token times wrong a fewer 100 milliseconds of each other, pinch 1 exemplary moving only from 0.29 to 0.35 seconds betwixt emblematic and worst case. Source: Metrics that Matter pinch Serverless Inference, DigitalOcean Community. A p99:p50 ratio adjacent to 1 is what that uncovering describes successful the vocabulary this article uses; the coefficient of variety from the consistency article supra is measuring the intimately related amount of variance astir the mean alternatively than the ratio betwixt 2 named percentiles, and either 1 is simply a morganatic measurement to quantify the aforesaid underlying property.

Where an architecture is built specifically to make this ratio adjacent to 1 by creation alternatively than by luck, the deterministic scheduling attack utilized by Groq’s LPU is the clearest example, and it is covered successful extent successful the related article connected inference silicon.

Tail latency is besides not an world metric to the companies that unrecorded pinch it. Among DigitalOcean’s published customer results, Hippocratic AI, which runs safety-critical healthcare agents, reports 2 times higher accumulation throughput and a 40 percent little p99 latency crossed much than 20 cardinal diligent interactions connected the platform. Note what the metric successful that condemnation is: p99, not median. Teams whose products break erstwhile the tail misbehaves discuss and measurement connected the tail, which is worthy remembering the adjacent clip a benchmark hands you a median and calls it speed.

The DIY latency trial protocol, executed

This conception describes the trial design, past reports the results of really moving it. I executed this trial connected July 27, 2026, against DigitalOcean Serverless Inference, from a GPU Droplet (gpu-h200x1-141gb) successful DigitalOcean’s NYC2 region, utilizing the nonstop book printed beneath for the afloat protocol runs. Everything the tally produced, the harness, the earthy per-request JSON for each 1,590 recorded requests, the console logs, the study code, and the charts, has been added to this nationalist Github repository: github.com/anishsingh20/serverless-inference-tail-latency-study.

Test design

Run against a azygous model, held constant, truthful the only adaptable nether trial is postulation style and concurrency:

• Single-call baseline, 3 concurrency levels. At slightest 75 sequential requests per compartment astatine concurrency 1, past repetition the aforesaid petition shape astatine concurrency 5 and concurrency 20, to observe really queueing changes the distribution arsenic load rises. Seventy-five requests is the period DigitalOcean’s ain consistency testing uses to reliably aboveground a acold start; less requests consequence missing the tail entirely. Source for this circumstantial period and the highest versus off-peak timing proposal below: Why Serverless Inference Consistency Varies connected the Same Model, DigitalOcean Community.
• Two clip windows. Run the afloat concurrency expanse erstwhile during business hours and erstwhile overnight aliases connected a weekend, since cold-start behaviour is worst erstwhile inheritance postulation to a exemplary is lowest and a daytime-only trial tin look artificially good.
• A chained multi-call test. Construct a 10-call sequential concatenation that mimics an supplier loop, each call’s output feeding the adjacent call’s input, and tally astatine slightest 30 independent chains done it. Record the full completion clip for each afloat chain, not conscionable each call, and compute the chain-level p50, p95, and p99 straight from those 30 totals, alternatively than trying to reconstruct them from the per-call numbers.

What to measurement and compute

For each cell: median TTFT, p95 TTFT, p99 TTFT, and the p99:p50 ratio. For the chained trial specifically: the distribution of full concatenation completion time, not immoderate individual call’s distribution, since that full is the number that predicts what your personification aliases your downstream strategy really waits for.

Two complementary tests. The concurrency expanse shows really queueing degrades a azygous telephone nether load. The chained trial shows what a realistic supplier workload really experiences extremity to end.

Setup checklist to tally this against DigitalOcean’s Serverless Inference

Every measurement present is the measurement I really performed for the results below, documented against DigitalOcean’s ain existent pages truthful you tin verify each 1 arsenic you go.

1. Create a exemplary entree key. In the DigitalOcean Cloud Console, spell to the Inference conception and create a exemplary entree key. You tin scope it to a circumstantial exemplary aliases time off it unscoped. See Manage Model Access Keys.
2. Confirm your relationship has a affirmative prepaid balance. Serverless Inference is prepaid and billed per token, and requests neglect if your equilibrium is astatine zero. See Manage Serverless Inference Prepayment and the Serverless Inference pricing page for existent per-model token rates.
3. Pick a exemplary and corroborate its nonstop API exemplary ID. Retrieve the ID pinch a GET petition to https://inference.do-ai.run/v1/models utilizing your exemplary entree key, aliases from the Model Catalog successful the Cloud Console, and usage immoderate drawstring that telephone returns. For the tally below, that telephone returned the IDs mistral-3-14B, openai-gpt-oss-120b, llama-4-maverick, and llama3.3-70b-instruct, and those nonstop strings were used. See Retrieve Available Models.
4. Install the 1 dependency the book needs. The book beneath uses only the Python modular library, truthful location is thing to instal beyond Python 3.
5. Export your cardinal arsenic an situation variable and tally the book erstwhile during business hours and erstwhile overnight aliases connected a weekend, redeeming each tally to a different output file. You tin transcript the harness from the codification artifact below, aliases clone it together pinch the earthy information and study codification from this github repository.
6. Compare the resulting JSON against this article’s tables, and statement the model, region, and dates you tested. The numbers beneath are 1 model connected 1 day; yours are the ones that matter for your workload.

The harness

#!/usr/bin/env python3 """ Latency benchmark for DigitalOcean Serverless Inference. Measures, per request: - TTFT (time to first streamed token), timestamped client-side - full completion time - punctual and completion token counts, from the usage object Workloads: 1. baseline_sweep - N requests astatine concurrency 1, 5, and 20 2. chained_session - M independent chains of `chain_length` sequential calls, each call's output feeding the adjacent call's input, pinch full concatenation clip recorded per chain Concurrency is implemented pinch a existent OS-thread excavation (ThreadPoolExecutor), not a single-process asyncio loop. This is simply a deliberate choice: a 2026 measurement-bias insubstantial (Chandrasekar and Kramberger, arXiv, cited successful this article) shows that single-process asyncio benchmarking clients tin hit their ain client-side queueing bottleneck nether concurrency, which inflates the very TTFT numbers you are trying to measure. Real OS threads release the GIL during blocking web I/O, which avoids that specific failure mode astatine the humble concurrency levels utilized here. Run: export DO_MODEL_ACCESS_KEY=your_key_here python3 do_latency_bench.py --model <model-id> --out results_daytime.json python3 do_latency_bench.py --model <model-id> --out results_overnight.json """ import argparse import concurrent.futures import json import os import time import urllib.request def percentile(sorted_vals, pct): """Linear-interpolation percentile, nary outer dependency required.""" if not sorted_vals: return None k = (len(sorted_vals) - 1) * (pct / 100) f = int(k) c = min(f + 1, len(sorted_vals) - 1) if f == c: return sorted_vals[f] return sorted_vals[f] * (c - k) + sorted_vals[c] * (k - f) def summarize(ttfts_ms): s = sorted(t for t in ttfts_ms if t is not None) if not s: return {"n": 0} p50 = percentile(s, 50) p95 = percentile(s, 95) p99 = percentile(s, 99) return { "n": len(s), "p50_ms": round(p50, 1), "p95_ms": round(p95, 1), "p99_ms": round(p99, 1), "p99_to_p50_ratio": round(p99 / p50, 2) if p50 else None, } def streamed_request(base_url, api_key, model, messages, max_tokens=64): """Send 1 streaming chat completion. Return TTFT and full clip successful ms.""" payload = { "model": model, "messages": messages, "max_tokens": max_tokens, "temperature": 0, "stream": True, "stream_options": {"include_usage": True}, } req = urllib.request.Request( base_url + "/chat/completions", data=json.dumps(payload).encode(), headers={ "Content-Type": "application/json", "Authorization": f"Bearer {api_key}", }, ) commencement = time.perf_counter() ttft = None content_parts = [] usage = {} try: with urllib.request.urlopen(req, timeout=120) as r: for earthy in r: statement = raw.decode(errors="ignore").strip() if not line.startswith("data:"): continue assemblage = line[5:].strip() if assemblage == "[DONE]": break chunk = json.loads(body) if chunk.get("usage"): usage = chunk["usage"] for prime in chunk.get("choices", []): delta = choice.get("delta", {}) if delta.get("content"): if ttft is None: ttft = (time.perf_counter() - start) * 1000 content_parts.append(delta["content"]) except Exception as e: return {"error": str(e), "ttft_ms": None, "total_ms": None} full = (time.perf_counter() - start) * 1000 return { "ttft_ms": round(ttft, 1) if ttft is not None else None, "total_ms": round(total, 1), "prompt_tokens": usage.get("prompt_tokens"), "completion_tokens": usage.get("completion_tokens"), "content": "".join(content_parts), } def baseline_sweep(base_url, api_key, model, n_requests, concurrency_levels, prompt): results = {} messages = [{"role": "user", "content": prompt}] for c in concurrency_levels: print(f"[baseline] concurrency={c}, {n_requests} requests", flush=True) records = [] with concurrent.futures.ThreadPoolExecutor(max_workers=c) as pool: futures = [ pool.submit(streamed_request, base_url, api_key, model, messages) for _ in range(n_requests) ] for f in concurrent.futures.as_completed(futures): records.append(f.result()) ttfts = [r.get("ttft_ms") for r in records] results[f"concurrency_{c}"] = { "summary": summarize(ttfts), "records": records, } return results def chained_session(base_url, api_key, model, n_chains, chain_length, prompt_prefix): chains = [] for one in range(n_chains): history = [{"role": "user", "content": f"{prompt_prefix} Chain {i}, measurement 0."}] commencement = time.perf_counter() calls = [] for measurement in range(chain_length): r = streamed_request(base_url, api_key, model, history) calls.append(r) history.append({"role": "assistant", "content": r.get("content") or "ok"}) history.append( {"role": "user", "content": f"Chain {i}, measurement {step + 1}. Continue."} ) total_chain_ms = (time.perf_counter() - start) * 1000 chains.append({"chain_index": i, "total_chain_ms": round(total_chain_ms, 1), "calls": calls}) print(f"[chained] concatenation {i} done: {total_chain_ms:.0f} sclerosis total", flush=True) chain_totals = sorted(c["total_chain_ms"] for c in chains) return { "chain_summary": { "n_chains": len(chain_totals), "p50_ms": round(percentile(chain_totals, 50), 1), "p95_ms": round(percentile(chain_totals, 95), 1), "p99_ms": round(percentile(chain_totals, 99), 1), }, "chains": chains, } def main(): ap = argparse.ArgumentParser() ap.add_argument("--base-url", default="https://inference.do-ai.run/v1") ap.add_argument("--model", required=True, help="Exact exemplary ID from GET /v1/models") ap.add_argument("--n-requests", type=int, default=75) ap.add_argument("--n-chains", type=int, default=30) ap.add_argument("--chain-length", type=int, default=10) ap.add_argument("--out", default="latency_bench_results.json") args = ap.parse_args() api_key = os.environ.get("DO_MODEL_ACCESS_KEY") if not api_key: raise SystemExit("Set DO_MODEL_ACCESS_KEY earlier running.") results = { "model": args.model, "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), } print("== Baseline expanse ==", flush=True) results["baseline_sweep"] = baseline_sweep( args.base_url, api_key, args.model, args.n_requests, concurrency_levels=[1, 5, 20], prompt="Reply pinch a two-sentence summary of why caching matters for LLM inference.", ) print("== Chained convention ==", flush=True) results["chained_session"] = chained_session( args.base_url, api_key, args.model, args.n_chains, args.chain_length, prompt_prefix="You are debugging a accumulation incident.", ) with open(args.out, "w") as f: json.dump(results, f, indent=2) print(f"Wrote {args.out}", flush=True) if __name__ == "__main__": main()

Run it precisely arsenic shown successful the docstring, erstwhile successful each clip window. The book prints advancement arsenic it runs and writes each earthy grounds to the output file.

What was really run, and nether what conditions

Everything beneath comes from 1 execution of this protocol, and these are the conditions, stated earlier the numbers truthful you tin measurement them:

• Client: a azygous GPU Droplet (gpu-h200x1-141gb, 24 vCPUs) successful DigitalOcean’s NYC2 region. The web way is cloud-internal and short, which intends these numbers could somewhat understate what a customer connected the nationalist net would see. But the flip broadside is that this is besides the realistic lawsuit for 1 people of reader: if your exertion already runs connected DigitalOcean adjacent to the conclusion endpoint, the short way measured present is the way your accumulation postulation really takes, pinch nary cross-cloud hop aliases egress successful between. The Droplet’s GPU is irrelevant to the test; it was simply the instrumentality available.
• Endpoint and workload: https://inference.do-ai.run/v1/chat/completions, streaming enabled, somesthesia 0, max_tokens 64, the aforesaid short punctual for each baseline request, precisely arsenic the book supra sends it.
• Volume: the afloat protocol per exemplary is 225 baseline requests (75 each astatine concurrency 1, 5, and 20) positive 30 independent 10-call chains, aliases 525 requests per model. Three models completed it: Mistral 3 14B, GPT-OSS 120B, and Llama 4 Maverick, for 1,575 recorded requests, pinch zero grounded requests.
• Time window: I performend the trial only for 1 window, 08:39 to 10:53 UTC connected Monday, July 27, 2026, which is 4:39 to 6:53 a.m. US Eastern, a low-traffic period. The protocol calls for a second, contrasting window, and this tally does not person one; dainty each number present arsenic a single-window measurement.
• One measurement caveat: the harness defines TTFT arsenic clip to first content token. A mini number of GPT-OSS 120B responses (3 of 75 astatine concurrency 1, 10 of 75 astatine concurrency 5, 1 of 75 astatine concurrency 20) spent their full 64-token fund connected soul reasoning tokens and produced nary content, truthful they person a full clip but nary TTFT, and the TTFT rows for that exemplary are computed connected the remaining requests.

The measured results

Time to first token, by concurrency level, 75 requests per cell:

Total completion clip for a azygous telephone astatine concurrency 1, pinch the consistency metrics this portion has been arguing for:

The measured 10-call supplier chains, 30 chains per model, full task clip per chain:

Excluding Maverick’s 1 stalled-stream chain, described successful the supplier multiplier conception above, its remaining 29 chains still measurement p50 26.6 s, p95 47.2 s, worst 57.7 s: slower and wider than GPT-OSS 120B astatine each percentile moreover pinch the catastrophic arena removed.

Each statement is 75 existent requests. The shape, not the starting point, is the story: Maverick starts accelerated and spreads wide; GPT-OSS starts slow and stays put.

How to publication this chart: Pick immoderate constituent connected a line, and it says “this percent of requests (vertical axis) sewage their first token wrong this overmuch clip (horizontal axis).” A statement that shoots up almost vertically intends a predictable model: astir each petition waits astir the aforesaid amount. A statement that rises accelerated and past bends into a long, shallow slope to the correct intends a exemplary pinch a tail: astir requests are quick, but the past fewer percent hold acold longer. That crook is precisely wherever the p95 and p99 live, and it is invisible if each you people is the constituent wherever the statement crosses 50 percent.

In this chart, Mistral 3 14B “wins” for some emblematic and worst-case latency: it not only has the lowest median clip to first token, but it besides keeps astir each requests tightly clustered together—even astatine the tail. Llama 4 Maverick comes adjacent astatine the median, but its tail is overmuch wider, meaning a fewer requests took overmuch longer. GPT-OSS 120B starts slower but maintains a short and accordant tail. Mistral’s operation of accelerated commencement and tight dispersed makes it the clear victor erstwhile you attraction astir some emblematic and worst-case latency.

Medians hardly move arsenic concurrency rises from 1 to 20. Tails move more, which is precisely why a concurrency-1 median is the slightest informative number a benchmark tin report.

How to publication this chart: each sheet is 1 model, near to correct on the bottommost is really galore requests were successful formation astatine once, and the 3 lines successful each sheet are the emblematic petition (p50, solid), the slow petition (p95, dashed), and the very slow petition (p99, dotted). A level coagulated statement intends the emblematic acquisition holds up arsenic load rises. What to watch is the vertical dispersed betwixt the coagulated statement and the dotted line: erstwhile that dispersed widens arsenic you move right, arsenic it does for GPT-OSS 120B astatine concurrency 20, queueing is starting to agelong the tail moreover though the median still looks untouched. This is the system by which a provider’s benchmark number and your accumulation acquisition softly portion ways.

Which exemplary won and why: Mistral 3 14B, and it is not close. It was the fastest to first token astatine each percentile and each concurrency level, pinch a median of astir 172 to 182 sclerosis whether 1 aliases 20 requests were successful flight, and a p99 that ne'er exceeded 628 ms; moreover its worst measured percentile was astir half of GPT-OSS 120B’s champion median. Llama 4 Maverick came 2nd pinch accelerated medians (214 to 320 ms) but a p99 4 to 5 times higher, astir 1.4 to 1.6 seconds. GPT-OSS 120B was past connected some counts: the slowest to commencement (medians of 1.1 to 1.3 seconds) and the only exemplary whose tail visibly degraded nether load, pinch p95 stretching from 2.0 to 3.1 seconds and p99 from 2.1 to 3.6 seconds arsenic concurrency roseate from 1 to 20.

The apt logic is mostly size and what happens earlier the first token: Mistral 3 14B is by acold the smallest exemplary here, truthful the prefill activity each petition needs earlier it tin emit thing is cheap, and during this model the shared excavation absorbed 20 concurrent requests without measurable queueing. GPT-OSS 120B is astir an bid of magnitude larger and, successful this deployment, spends portion of its fund connected soul reasoning earlier the first contented token appears, truthful its TTFT starts precocious and is the first to consciousness queueing pressure. The caveat from the remainder of this article still applies, though: winning clip to first token is winning the streaming-chat metric. On full completion clip and the measured 10-call chain, Mistral 3 14B did besides decorativeness first, but GPT-OSS 120B hit Llama 4 Maverick contempt losing to it connected each sheet of this chart.

The 4th exemplary is its ain finding

I intended to tally the identical afloat protocol connected llama3.3-70b-instruct. I ran 15 sequential requests utilizing the aforesaid streamed_request usability from the harness above, wrapped successful a watchdog that enforces a difficult 150-second wall-clock headdress per petition and writes each consequence to disk the infinitesimal it completes, truthful a stalled watercourse tin costs astatine astir 150 seconds alternatively of stalling the full run.

All 15 completed wrong the cap. Median TTFT was 2.1 seconds. Median full clip for 64 tokens was 43.5 seconds. Two of the 15 requests waited 37 and 68 seconds respectively for their first token, the signature of queueing aliases a acold path, while the different 13 sewage their first token wrong 1.6 to 3.4 seconds. At these speeds, a 10-call supplier concatenation connected this exemplary would return complete 7 minutes astatine the median, truthful the chained trial was not run, and this model’s numbers are from 15 requests alternatively than 525 and should beryllium publication pinch that weight.

This is the sharpest statement successful the article and it was not planned: the astir recognizable exemplary sanction successful the catalog was, during this circumstantial window, 1 to 2 orders of magnitude slower per completed petition than 3 little celebrated alternatives connected the aforesaid endpoint, and nary header TTFT fig would person warned you, because its median TTFT was a reasonable-looking 2.1 seconds. The number that told the truth was the 1 almost cipher publishes: full completion clip and its spread.

Interpreting what you find

A fat correct tail pinch an different normal-looking assemblage suggests queueing aliases batching contention: the emblematic petition is fine, but a subset waits down different traffic. Maverick’s measured TTFT is this shape exactly: a 300 millisecond median pinch a p99 5 times higher and a coefficient of variety of 75 percent, against GPT-OSS 120B’s 27 percent. A bimodal distribution, a cluster of accelerated requests and a abstracted cluster of slow ones pinch a spread betwixt them, is the signature of acold starts aliases a congested path: immoderate requests deed a lukewarm replica and immoderate do not, and location is adjacent to thing successful between. Llama 3.3 70B’s bounded tally showed this shape, pinch 13 requests getting a first token wrong 3.4 seconds and 2 requests waiting 37 and 68 seconds.

For the fat-tail pattern, the nonstop mitigation is simply a dedicated endpoint aliases reserved capacity, since the underlying origin is title pinch different tenants’ traffic. On DigitalOcean specifically, serverless and dedicated conclusion are portion of the aforesaid platform, and the move from shared to dedicated capacity is positioned arsenic a capacity determination alternatively than a migration project, which matters present because the correct clip to make that determination is aft a measurement for illustration this one, not before. For the bimodal pattern, the nonstop mitigation is simply a lukewarm excavation aliases scheduled keep-alive postulation that prevents the replica count from dropping to zero successful the first place. If you way crossed aggregate models aliases providers, session pinning is what keeps a concatenation of calls connected the aforesaid lukewarm way alternatively than re-triggering a acold commencement partway done a task; this is covered straight successful 1 of the related article I wrote connected prompt caching and convention pinning.

Decision framework

Evaluate connected p95 and p99 erstwhile your workload is interactive, session-based, aliases bound by a latency SLA. A azygous user-facing petition aliases a short convention is precisely the lawsuit wherever the tail is what a existent personification feels.

Evaluate connected task completion clip erstwhile your workload is an supplier pipeline of 3 aliases much sequential calls. Derive your required per-call p99 from your task-time target and activity backward, alternatively than picking a per-call target first and hoping the full comes retired acceptable. If your task fund is 5 seconds crossed 10 calls, your existent per-call fund is adjacent to 500 milliseconds astatine the tail, not astatine the median, and the measured results earlier successful this portion show why a header single-call metric is the incorrect number to cheque that fund against: the exemplary that won measured TTFT by 4 times mislaid the measured 10-call task by 28 percent.

Evaluate connected throughput and cost, and group tail latency aside, erstwhile your workload is offline batch aliases an asynchronous pipeline pinch nary personification aliases downstream strategy waiting connected immoderate azygous request’s completion.

If you way requests crossed models by costs versus latency, way your latency-critical paths by their measured p99 floor plan and way your cost-tolerant paths by price, a determination this portion states concisely since the actual routing mechanics are covered successful the Inference Router documentation.

Three workload shapes, 3 correct metrics. Using the incorrect 1 does not conscionable mislead you, it tin constituent you astatine the incorrect exemplary entirely, arsenic the measured TTFT-versus-task-time inversion supra shows directly.

What this portion does and does not verify

The tail-compounding look and each worked number derived from it are my ain nonstop calculation, checked pinch a elemental script, and you tin rerun the aforesaid look against your ain per-call tail complaint and concatenation length. The measured latency numbers successful this portion travel from 1 execution of the published trial protocol, tally by the writer against DigitalOcean Serverless Inference connected July 27, 2026, from a GPU droplet successful the aforesaid cloud, utilizing the nonstop book printed successful the harness section; the earthy per-request JSON was retained, each array was computed from it, and the complete grounds guidelines is published successful the GitHub repository truthful anyone tin recompute each array and floor plan from the earthy records. Those numbers are real, and they are besides 1 clip window, 1 customer location, and 1 day. They picture what that endpoint did during that window, not what it will do during yours, which is the full logic the book is included. The resampled chain-length curves are explicitly branded arsenic bootstrap extensions of measured single-call information and are shown alongside straight measured chains; wherever the 2 disagree, the measured chains are the crushed truth. The Artificial Analysis provider-spread figures and the DigitalOcean consistency and metrics information are real, cited, and linked directly; the 2 “implied slowest provider” figures successful the TTFT-ranking conception are my ain arithmetic from Artificial Analysis’s stated percent spread, not a number Artificial Analysis published directly, and I person flagged that favoritism astatine the constituent it appears. If you tally the protocol and your results disagree from thing reported here, including connected DigitalOcean’s ain serverless inference, that correction is worthy much than this article, and I would alternatively you people it than presume this portion already covered your case.

Common Questions connected this topic?

1. Is a debased median latency ever a bad sign?

Not connected its own. A low median pinch a low p99:p50 ratio is the champion imaginable outcome. A low median pinch a high p99:p50 ratio intends the emblematic lawsuit is accelerated but a meaningful fraction of your postulation is not, which the median unsocial will ne'er show you.

2. Why does a 1 percent per-call tail complaint matter if it sounds truthful small?

Because “small” is comparative to really galore times you return the risk. A azygous petition has a 1 percent chance of hitting it. A 20-call supplier concatenation has an 18 percent chance that astatine slightest 1 telephone successful that concatenation hits it, a number this portion verified straight alternatively than assumed.

3. Does this mean I should ever prime the supplier pinch the tightest tail complete the 1 pinch the champion median?

Not automatically, and it depends wholly connected your workload. For a azygous interactive petition successful a streaming chat interface, a debased TTFT pinch an acceptable p95 tin beryllium the correct call. For immoderate workload that chains sequential calls into 1 task, this piece’s measured results show the TTFT victor losing the 10-call task by 28 percent astatine the median and 84 percent astatine p95, which is why the correct information metric depends connected your telephone count and what your personification is really waiting for, not connected a fixed penchant for 1 header number.

4. Do these results mean GPT-OSS 120B is “better” than Llama 4 Maverick?

No, and this portion should not beryllium quoted arsenic saying so. The measurements are 1 clip model connected 1 day, pinch a fixed 64-token output headdress and nary information of reply quality, which is usually the deciding facet betwixt models. What the results do show is that the ranking depends connected the metric: Maverick genuinely won TTFT and genuinely mislaid task completion clip during this window. The durable takeaway is the method, not the leaderboard: measurement the metric your workload really feels, connected your ain postulation shape, earlier you choose.

Conclusion

The benchmark ecosystem reports what is easy to measurement and flattering to publish. A cleanable median is easy to measurement and ever flattering. Production symptom lives successful the style of the distribution for a azygous request, and for an agent, successful the sum crossed each telephone successful its chain. This article made 2 changes to really you should publication a latency number. Look astatine the distribution, not the center, for immoderate azygous interactive request. Look astatine the full task time, not immoderate individual call’s percentile, for immoderate workload that chains calls together, and deduce your per-call request backward from your task-level target alternatively than the reverse.

Please tally the trial protocol successful this article against your ain postulation shape, astatine your ain concurrency, connected your ain concatenation length. This is the only measurement to cognize what your ain accumulation acquisition will look like.

References

1. GitHub Repository to tally the trial protocol yourself

• serverless-inference-tail-latency-study connected GitHub: the afloat grounds guidelines for each measured number successful this article — the measurement harness, orchestration and bounded-sweep scripts, earthy per-request JSON for each 1,590 recorded requests, unedited console logs, the study script, and the 4 charts, nether an MIT license.

2. DigitalOcean archiving and engineering content

• Prompt Caching and Cost Break-Even
• Inferentia2, TPU, Groq LPU, vs. GPU for LLM Serving
• Serverless Inference Metrics reference
• Inference Features, observability section
• Inference Limits
• Why Serverless Inference Consistency Varies connected the Same Model
• Metrics that Matter pinch Serverless Inference
• DigitalOcean Inference Mode Comparison for Your Each Use Case
• Fine-Tuned LLMs connected Serverless Architecture
• How to Use Inference Router
• DigitalOcean Serverless Inference: A Deep Dive
• Why is p99 Time to First Token (TTFT) High When Everything Else Looks Normal? Debugging Tail Latency successful LLM Inference

3. External sources

• Artificial Analysis, Language Model Benchmarking Methodology
• Artificial Analysis, GLM-5.2 (max) supplier comparison
• Artificial Analysis, gpt-oss-120b (high) supplier comparison
• Ashok Chandrasekar and Jason Kramberger, “Identifying and Mitigating Systemic Measurement Bias successful Production LLM Inference Benchmarks,” arXiv, 2026.