Long-context LLM serving: the real tradeoffs in memory, latency, cost, and accuracy

What “long context” means, and why supported is not the aforesaid arsenic served

Context is everything you nonstop to a exemplary successful 1 request: the prompt, documents, code, speech history. It’s measured successful tokens, which are connection pieces, astir 750 words per 1,000 tokens. A 128K-token model (about a novel’s worthy of text) is the communal modular today. Llama 3.1 and GPT-OSS-120B some vessel pinch 128K windows. Frontier closed models spell further. Claude Sonnet 4.6, 4.5, and 4 judge up to 1M input tokens, astir 10 novels, per the exemplary listings connected DigitalOcean’s pricing page.

The number connected the spec expanse tells you what the exemplary tin accept. It tells you thing astir what happens erstwhile galore users really nonstop inputs that large to the aforesaid infrastructure astatine the aforesaid time. Support is simply a exemplary capability. Performance is simply a serving question, and the 2 travel isolated successful 4 circumstantial places. This article walks done them, pinch the numbers.

You request 2 pieces of inheritance first.

Prefill and decode. Inference has 2 phases. In prefill, the exemplary sounds your full input successful 1 parallel walk earlier it writes anything. Prefill is compute-bound, constricted by really accelerated the GPU tin do math. In decode, the exemplary generates its reply 1 token astatine a time. Decode is memory-bound, constricted by really accelerated information moves retired of GPU memory, not by arithmetic. Long inputs mostly accent prefill. Long outputs mostly accent decode.

The KV cache. For each token the exemplary reads, it computes and stores 2 vectors, a cardinal and a value, truthful it doesn’t person to reread the full input for each caller token it generates. That shop is the KV cache. It sits successful GPU representation for the full life of the request, and it grows successful nonstop proportionality to really overmuch you sent.

Tradeoff 1: memory

The KV cache footprint follows a elemental formula:

2 × layers × KV heads × head magnitude × series magnitude × bytes per element

The 2 counts the 2 vectors stored per token, 1 cardinal and 1 value. This is the modular cache accounting from the serving literature; the PagedAttention paper (Kwon et al., SOSP 2023), the activity down vLLM, derives the aforesaid per-token calculation. The look present uses KV heads alternatively than full attraction heads because models for illustration Llama 3 usage grouped-query attention, wherever galore query heads stock a fewer KV heads, which shrinks the cache considerably.

Here it is worked out, pursuing the DigitalOcean tutorial, for Llama 3 70B successful BF16 precision (80 layers, 8 KV heads, caput magnitude 128) pinch a 128K-token input:

2 × 80 × 8 × 128 × 131,072 × 2 bytes ≈ 43 GB

Sit pinch that number for a second. 43 GB of GPU representation for 1 user’s context, astir what a heavy compressed transcript of the exemplary itself would take. An NVIDIA A100 aliases H100 has 80 GB total, and the exemplary weights already declare astir of it. Push to a 1M-token discourse and the cache outgrows the model.

Run the aforesaid look crossed the Llama 3 family (all 3 usage 8 KV heads and a caput magnitude of 128; the furniture counts are 32, 80, and 126 per their published configurations) and you tin spot really the cache scales pinch exemplary size and discourse magnitude together. Cache sizes only; weights travel connected top:

Read the 128K file against an 80 GB card. One petition takes 21% of the card’s representation connected the 8B model, 54% connected the 70B, 85% connected the 405B, and that’s earlier the weights. Now look astatine the 1M column. Every number successful it is bigger than 80 GB, which intends a azygous million-token petition cannot fresh connected 1 GPU nary matter what. The cache has to beryllium divided crossed respective cards, and during procreation those cards perpetually walk attraction information backmost and distant complete the links betwixt them. Those links are galore times slower than a GPU reference its ain memory, truthful serving gets slower and much analyzable astatine precisely the infinitesimal it’s already astir expensive. (One honorable statement astir the table: Llama 3.1 itself only accepts 128K tokens, truthful the 1M file is not thing you tin really tally connected these models. It applies the aforesaid look astatine 1M to show what the mathematics looks for illustration for the frontier models that do advertise windows that size.)

Where this really hurts is batching. GPUs enactment affordable because galore requests get processed together and stock the fixed costs. Long requests break that successful 2 ways. A azygous immense cache leaves nary headroom for different requests to subordinate the batch, and a operation of mini and ample requests fragments GPU representation truthful that moreover the free abstraction can’t beryllium packed well.

PagedAttention, the method vLLM uses, allocates cache representation successful pages alternatively of 1 continuous artifact and recovers immoderate of the waste. But a 128K petition still needs 128K worthy of memory. So arsenic mean discourse magnitude creeps up crossed a fleet, batch sizes shrink, utilization falls, and costs per token climbs, each without thing being incorrect pinch the hardware.

Tradeoff 2: latency

Latency is waiting time: really agelong betwixt sending a petition and getting thing back. For a connection exemplary it has 2 parts that users consciousness differently. The first is the region earlier immoderate output appears astatine all, called clip to first token (TTFT). The 2nd is really quickly the remainder of the reply streams successful erstwhile it has started. People tolerate a slow watercourse acold amended than a agelong silent pause, which makes TTFT the number that decides whether an app feels responsive. And TTFT is precisely what agelong discourse inflates.

Here’s why. Attention, the cognition astatine the halfway of each transformer, compares each token successful the input against each different token. Compute grows pinch the quadrate of input length. Double the context, quadruple the work. From 32K to 128K is 4x the magnitude and 16x the attraction compute.

All of that happens successful prefill, earlier the first connection of the reply appears. Which is why TTFT goes from milliseconds astatine short contexts to full seconds astatine agelong ones. Optimizations for illustration FlashAttention do thief here. They trim down the clip the GPU wastes moving information backmost and distant betwixt its accelerated and slow memory, and the speedup is noticeable. What they can’t do is alteration the underlying math. The activity still grows pinch the quadrate of the input; it conscionable grows from a little starting point. As agelong arsenic modular attraction is quadratic, each token you adhd makes the adjacent 1 much expensive.

The 2nd latency costs is easier to miss: what your agelong petition does to everyone else. A 128K-token prefill holds the GPU for seconds, and each petition queued down it waits. The group wounded astir aren’t the ones who sent agelong requests, since they expect to wait. They’re the ones who sent short requests that had the bad luck to onshore down a agelong one. On a dashboard, this looks for illustration patient mean latency while p95 and p99 softly rot, and that’s the shape that does the astir harm to interactive products for illustration agents and copilots.

Tradeoff 3: throughput and the bandwidth ceiling

Even pinch unlimited compute, decode hits a wall that has thing to do pinch arithmetic. To make each output token, the GPU has to publication the full KV cache retired of high-bandwidth representation (HBM). Flagship GPUs move information astatine astir 3 to 4 TB/s, and that number is fixed. Once the cache is tens of gigabytes, reference it erstwhile per generated token becomes the constraint that matters.

The narration is astir linear. Double the context, double the cache, double the information publication per decode step, and your tokens-per-second astir halves. A cluster tin person compute sitting idle and still service agelong contexts slowly, because the bottleneck has moved disconnected the compute units and onto the representation bus. If you bargain and measurement GPU capacity successful FLOPS, you’re measuring the incorrect constraint for this workload.

You tin put an precocious bound connected this yourself pinch 1 division. For a azygous request, each decode measurement sounds the exemplary weights positive the KV cache, truthful the champion imaginable velocity is bandwidth divided by bytes read. For Llama 3 70B successful BF16 (141 GB of weights) connected an H100 SXM (3.35 TB/s):

Two caveats. This is simply a theoretical ceiling for 1 unbatched request; thing exceeds it and astir systems beryllium good beneath it. And for a azygous request, the weights predominate the publication until the cache grows to a comparable size, which is why the ceiling falls mildly astatine first and past steeply. In accumulation the effect is worse than the array suggests, because batching shares 1 weight publication crossed galore requests while each petition brings its ain cache traffic. Once cache sounds predominate the bus, you get the double-the-context, halve-the-throughput behavior. The constituent of the arithmetic isn’t precision. It’s that you tin sanity-check immoderate provider’s throughput declare successful 30 seconds.

Cost follows. You mightiness presume a 128K-token petition costs astir 128 times a 1K request, successful proportionality to tokens. It usually costs more, erstwhile quadratic prefill, shrunken batches, saturated bandwidth, and the blocked queue down it are counted. Some pricing already admits this. Anthropic’s Claude Sonnet models costs $3.00 per cardinal input tokens up to 200K tokens of input and $6.00 per cardinal past it, pinch output rising from $15.00 to $22.50, arsenic listed connected DigitalOcean’s exemplary pricing page (checked July 2026). Prompt caching exists mostly to soften this: you salary erstwhile to process a long, unchangeable discourse ($3.75 per cardinal tokens to constitute the cache, pinch a 5-minute lifetime, for Claude Sonnet), past $0.30 per cardinal each clip you reuse it.

The cleanest measurement to spot what this intends is to value a convention alternatively of a token. Take a 10-turn supplier speech carrying a unchangeable 128K-token context, opportunity a codebase aliases a archive set, wherever each move adds a 500-token mobility and gets a 500-token answer, astatine the Claude Sonnet rates above:

Cumulative costs of a 10-turn supplier convention pinch a 128K-token context, compared crossed 3 strategies: resending the afloat discourse each move reaches $3.93, caching it reaches $0.95, and retrieving an 8K portion reaches $0.33

The crippled makes the style of each strategy visible. Resending grows steeply and ne'er stops. Caching starts higher than retrieval because of the one-time cache constitute connected move 1 (you tin spot it transverse the resend statement betwixt turns 1 and two), past flattens to almost thing per turn. Retrieval stays debased and level throughout.

Same conversation, aforesaid model. Caching cuts the measure 4x and retrieval cuts it 12x. At thousands of sessions a day, the architecture prime matters much than the per-token rate. Retrieval isn’t free either; it needs its ain infrastructure, which brings america to the adjacent section.

Tradeoff 4: accuracy, and why accuracy is besides a velocity problem

The first 3 tradeoffs are infrastructure problems. The 4th starts arsenic a exemplary problem and ends up arsenic an infrastructure problem.

Models reply questions astir agelong inputs little reliably than short ones. This is good documented. RULER showed that the effective discourse magnitude of galore models, the magnitude astatine which they still execute well, falls acold short of the advertised maximum. Lost successful the Middle showed accuracy dropping sharply erstwhile the applicable accusation sits successful the mediate of a agelong input alternatively of astatine the commencement aliases end.

A 2026 measurement study from IBM Research and TU Delft, Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Serving (Yoshimura, van de Beek, and Chiba, EuroMLSys '26), adds 2 findings worthy knowing if you tally long-context workloads successful production.

The first is that accuracy degrades successful ways that break intuition. The authors tested 5 models connected key-value lookup tasks astatine discourse lengths from 4K to 64K tokens, successful English, Japanese, and Chinese. Model size didn’t foretell long-context accuracy. Phi3-mini, the smallest exemplary tested, was often the astir accurate, and hit the larger Phi3-medium. A 2B exemplary outperformed its 8B related astatine 32K and 64K tokens. One exemplary held up good to 16K and past collapsed astatine 32K. Language mattered too: the champion exemplary successful English wasn’t needfully the champion successful Japanese aliases Chinese. Latency rankings crossed models stayed stable. Accuracy rankings didn’t.

The 2nd uncovering is the paper’s halfway argument: accuracy failures move into latency. When a exemplary answers wrong, the personification aliases the strategy retries, and the timepiece keeps running. The insubstantial gives this a name, Time-to-Correct-Answer (TTCA): the full wall-clock clip from the first effort until the first correct answer. A accelerated exemplary that’s incorrect half the clip tin consciousness worse than a slower exemplary that’s correct the first time, because the retries adhd up. Under long-context serving, arsenic the authors put it, accuracy becomes speed.

They besides show you tin enactment connected this. Their routing design, LAAR (Lightweight Accuracy-Aware Routing), scores each campaigner exemplary by expected latency divided by expected occurrence probability, utilizing thing fancier than input magnitude and connection arsenic features, and it won’t resend a retry to a exemplary that already failed. Across 5 models connected A100 GPUs, this trim mean TTCA by up to 31% against load-aware routing and up to 49% against session-affinity routing. The circumstantial numbers matter little than the principle. If you service agelong contexts crossed much than 1 model, discourse magnitude should thief determine which exemplary gets the request, because the champion prime changes pinch it.

Is RAG still applicable erstwhile discourse windows are this big?

Fair question. If models judge a cardinal tokens, why support a vector database, a chunking pipeline, and a retrieval measurement around? Why not paste the full corpus into the prompt?

Because everything supra is the value of that paste. Stuffing the model intends paying the quadratic prefill cost, the multi-second clip to first token, and the gradual per-token pricing connected each request, moreover erstwhile the reply needed 3 paragraphs of the input. It besides intends moving successful the region wherever models are slightest reliable. RULER recovered effective discourse routinely falling short of the advertised window, and Lost successful the Middle recovered models weakest precisely erstwhile the applicable item is buried mid-context, which is the normal lawsuit erstwhile a full corpus goes in. Retrieval sidesteps each 4 tradeoffs the aforesaid way: it keeps requests short. The exemplary sounds only the portion that matters, truthful prefill is cheap, the cache is small, the reply starts fast, and the exemplary useful astatine lengths wherever it’s dependable.

So agelong discourse didn’t make retrieval obsolete. It changed erstwhile each attack wins, and the be aware runs some ways: activity connected combining the two, specified arsenic Long-Context LLMs Meet RAG, recovered that retrieving much passages into a large model doesn’t reliably amended answers and tin degrade them arsenic irrelevant matter piles up. A bigger model is not a licence to retrieve carelessly.

The costs of skipping retrieval

The 4 tradeoffs are the visible price. Teams that driblet retrieval and spell all-in connected the model besides deed a group of costs that don’t show up until the strategy is successful production. Five travel up again and again.

The measure scales pinch the window, not pinch the question. With retrieval, you salary for the fewer 1000 tokens that were really relevant. Without it, you salary for the full model connected each request, whether the exemplary needed it aliases not. At the Claude Sonnet complaint of $3.00 per cardinal input tokens, 1 128K-token petition costs astir $0.38 successful input alone. At 10,000 requests a month, that’s $3,840 a period of input walk earlier a azygous output token is billed. The aforesaid postulation done an 8K retrieved discourse is astir $240. Nothing is wrong, thing is broken, and the measure is 16x higher.

Caches expire, and idle clip bills you again. Prompt caching looks for illustration the hole for the costs above, and it often is, but the inexpensive cache tier connected Claude Sonnet has a 5-minute lifetime. A personification who sounds thing for six minutes and past asks a follow-up triggers a afloat rewrite of the cache, different $0.48 for a 128K discourse astatine the $3.75 per cardinal constitute rate. Systems pinch bursty, human-paced postulation re-pay this constantly, and it seldom appears successful the original costs model.

Quality drops silently, successful the middle. Lost successful the Middle measured double-digit percentage-point accuracy drops erstwhile the applicable accusation sits mid-context alternatively than astatine the commencement aliases end. Nothing errors and thing logs. The answers conscionable get worse depending connected wherever successful your corpus the truth happened to live, which is simply a nonaccomplishment mode you can’t spot without information sets that alteration reply position.

Requests tin neglect wrong the advertised limit. The model connected the spec expanse assumes the representation to service it is free astatine that moment. Under concurrent load, the KV cache and activations for a very agelong petition tin transcend what’s available, and the petition fails moreover though it was wrong the model’s stated limit. Retrieval-shaped traffic, thousands of mini requests alternatively of a fewer tremendous ones, is simply easier for a serving strategy to support promises about.

Interactive latency budgets don’t past the window. A well-built retrieval measurement adds tens to a mates 100 milliseconds. Prefill connected a very ample discourse costs seconds, and nary magnitude of output streaming hides a slow start. If your product’s responsiveness target is simply a mates of seconds, the architecture determination is efficaciously made for you.

None of this says retrieval is free. It says the window’s costs are back-loaded: inexpensive to demo, costly to operate. Retrieval’s costs are front-loaded successful infrastructure and spell down per query. That asymmetry is why the demo and the accumulation measure truthful often disagree.

A applicable measurement to choose

Long discourse positive caching wins erstwhile the moving group is unchangeable and reused. Many requests against the aforesaid archive set, and the group fits successful the window? Cache it erstwhile and skip retrieval infrastructure entirely. A day-long supplier convention that keeps accumulating history is the aforesaid shape: the discourse is the state, and there’s thing to retrieve from.

RAG wins erstwhile the corpus is large, changing, aliases mostly irrelevant per query. A knowledge guidelines bigger than immoderate window, documents that update daily, aliases questions that each touch a mini fraction of the data. Resending the unneeded mostly connected each petition buys thing but costs and latency.

Precision lookups favour RAG moreover erstwhile the corpus would fit. If the occupation is uncovering 1 circumstantial fact, retrieval keeps the exemplary astatine short lengths wherever the accuracy measurements supra opportunity it holds up.

Most accumulation systems extremity up utilizing both: retrieval to determine what enters the context, a agelong model to clasp a generous magnitude of it, and caching wherever it repeats.

What providers and teams do astir it

Every known mitigation buys thing and gives thing up:

None of these gets you long-context capacity astatine short-context cost. What they adhd up to is simply a short playbook:

Don’t nonstop what you don’t need. Retrieval, summarizing aged turns, and pruning dormant discourse hit each GPU-side fix, because the cheapest token is the 1 you ne'er send.

Cache what repeats. A long, unchangeable prefix shared crossed requests (a strategy prompt, a document, a codebase) is precisely what punctual caching is for. Pay the prefill once, reuse it cheaply.

Match the exemplary to the length. The caller research results opportunity the astir meticulous exemplary astatine 8K whitethorn not beryllium the astir meticulous astatine 64K, and bigger isn’t automatically better. If you power routing, provender discourse magnitude into it. If you don’t, trial your exemplary astatine your lengths alternatively than trusting the advertised window. RULER exists because the 2 routinely differ.

Watch tail latency and discourse mix, not averages. The harm hides successful p95 and p99, and successful autoscaling signals that count requests alternatively of tokens. Ten 128K requests and a 1000 1K requests tin look identical to a request-rate dashboard correct up until latency collapses.

Before committing to a supplier for long-context work, these are the 4 questions worthy asking. How is long-context throughput measured, unsocial aliases nether concurrent load? What happens to different tenants’ latency erstwhile a agelong petition arrives? Does pricing standard linearly pinch tokens, aliases does it bespeak the existent non-linear cost? And tin they show you the prefill versus decode latency split? That past 1 is simply a decent maturity test. A supplier that has instrumented that divided knows wherever its clip goes.

Getting started

If you’re evaluating really your ain stack holds up arsenic discourse grows, DigitalOcean’s Inference Engine serves long-context models (including Claude Sonnet’s 1M-token input window) pinch per-token pricing published per model, punctual caching rates listed alongside, and Dedicated Inference for workloads that request isolation from noisy neighbors. Test pinch your existent discourse lengths and existent concurrency, not a azygous petition successful a demo. That’s the fastest measurement to study whether a discourse model is thing your merchandise tin trust connected aliases conscionable thing it technically supports.

Sources

• Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Serving, Yoshimura, van de Beek, and Chiba, EuroMLSys '26 (accuracy variability measurements, TTCA, LAAR results)
• Long-Context Inference astatine Scale: The Hidden Infrastructure Cost, DigitalOcean, May 2026 (KV cache look and worked example, prefill/decode analysis, bandwidth ceiling, supplier questions)
• Efficient Memory Management for Large Language Model Serving pinch PagedAttention, Kwon et al., SOSP 2023 (KV cache accounting, paged cache allocation)
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, Ainslie et al., 2023 (grouped-query attention, the KV-heads word successful the cache formula)
• RULER: What’s the Real Context Size of Your Long-Context Language Models? (effective vs. advertised discourse length)
• Lost successful the Middle: How Language Models Use Long Contexts (positional accuracy loss)
• Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs successful RAG (retrieval value successful agelong windows)
• RAG vs Large Context Window: The Real Trade-offs for AI Apps, Redis, February 2026 (operational costs categories referenced successful the hidden-costs subsection)
• DigitalOcean Gradient AI Platform Pricing (tiered long-context and punctual caching rates, checked July 2026)