How Does Prompt Caching Work and When Does It Actually Cut LLM Costs?

Prompt caching is marketed arsenic a discount you get for free. It is not automatic and it is not free connected each provider. The redeeming depends connected 3 things astir your traffic: really overmuch of each petition repeats, really galore requests stock that repeated part, and really overmuch clip passes betwixt them. Get those 3 correct and caching tin trim your input token costs by 80 to 90 percent connected a agelong supplier session. Get them incorrect and connected astatine slightest 1 awesome provider, caching tin costs you much than not caching astatine all.

This article gives you the formula, not conscionable the claim. It shows the nonstop break-even mathematics derived from each provider’s ain published pricing, past gives you a DIY trial harness you tin tally against DigitalOcean’s Serverless Inference to measurement your ain workload alternatively of trusting a vendor’s header percentage. It besides includes firsthand measured results I ran against Llama 3.3 70B (FP8) served by vLLM 0.24.0 connected a DigitalOcean GPU Droplet pinch a azygous NVIDIA H200, and each deed complaint and latency number successful the measured-results conception beneath comes from that test.

TL;DR

• Prompt caching is workload-dependent, not provider-dependent. The aforesaid provider’s caching tin beryllium a clear triumph aliases a nett nonaccomplishment depending connected really your postulation is shaped.
• Three variables determine the outcome: prefix stability, convention depth, and interarrival time. This article defines each 1 precisely and shows really to measurement it.
• Anthropic charges a constitute premium; astir OpenAI models connected DigitalOcean do not, though OpenAI’s ain archiving says the newest ones (GPT-5.6) now do. That quality changes the math, and this article derives the nonstop break-even constituent for each case. The Anthropic and most-OpenAI rates beneath are confirmed connected DigitalOcean’s ain pricing page; the GPT-5.6 cache-write complaint is confirmed connected OpenAI’s pricing page but not yet independently verified connected DigitalOcean’s, since that exemplary launched connected DigitalOcean aft the pricing page was past verified.
• A structural gross matters much than immoderate percentage: for OpenAI models, DigitalOcean will not cache a punctual beneath 1,024 tokens. Open-source models connected DigitalOcean person nary specified floor, and neither does a self-hosted engine, a quality that this article measures directly.
• Open-source models connected DigitalOcean’s Serverless Inference support punctual caching automatically. It is successful public preview and Prompt caching is automatic (no cache_control aliases prompt_cache_retention needed) for open-source models connected DigitalOcean, per-account isolated, and disposable connected models for illustration DeepSeek V3.2. Use punctual caching pinch the serverless conclusion chat completion and responses APIs to cache discourse and usage it successful early requests. If portion of your petition is already cached, you are charged a little value for those cached tokens and the modular value for the remaining input tokens. This importantly reduces the costs for inference. Open-source models cache discourse automatically. For a database of models that support punctual caching, spot Foundation Models. For much details, usage Prompt Caching successful the serverless inference documentation.
• Measured connected a DigitalOcean GPU Droplet (H200, vLLM 0.24.0, Llama 3.3 70B FP8): a cache deed trim time-to-first-token from 462 sclerosis to 39 sclerosis connected a 3,110-token prompt, a 92 percent reduction. A 12-turn supplier convention pinch a unchangeable 2,080-token prefix sustained a 98.7 percent token deed complaint aft the first turn.
• One statement of move contented destroys everything, and this portion measured precisely really much: placing a timestamp and petition ID astatine the apical of the strategy punctual dropped the measured token deed complaint from 98.7 percent to 0.7 percent and returned each move to full-price prefill latency.
• Self-hosting connected a GPU Droplet is the fallback for immoderate unfastened exemplary without managed caching. vLLM enables automatic prefix caching by default, pinch nary constitute premium and nary 1,024-token minimum. The measured tally beneath confirms caching connected a punctual arsenic short arsenic 155 tokens connected Llama 3.3 70B, which does not person a managed serverless caching rate.

Table of Key Terms successful Prompt Caching

If you’re caller to punctual caching, present are immoderate elemental definitions for position utilized successful this article:

How punctual caching really works, and what it costs

When you nonstop a petition to a model, the level has to person each input token into an soul practice earlier it tin make a response. This measurement is called prefill. For a agelong strategy prompt, a group of instrumentality definitions, aliases a ample retrieved document, this is the astir costly portion of a petition successful some costs and latency, because the level reprocesses that aforesaid contented from scratch each azygous time, moreover if it saw the identical matter 1 2nd ago.

Prompt caching stores the processed representation, the key-value authorities the attraction layers use, truthful that a later petition pinch the aforesaid prefix tin skip reprocessing and publication the stored authorities instead. This only applies to input tokens. Output tokens are ever generated fresh, since the consequence depends connected what you are asking this time, not what you asked past time.

The system depends connected nonstop matching. Providers cache based connected a hash of the punctual prefix. If a azygous characteristic changes anyplace earlier the cache boundary, including a timestamp, a petition ID, aliases a reordered instrumentality list, the full prefix fails to lucifer and the petition falls backmost to full-price processing. This is why punctual building matters arsenic overmuch arsenic punctual content.

A cache deed skips the costly prefill measurement for the matched prefix. A cache miss processes the afloat punctual and, connected providers pinch definitive caching, writes a caller introduction for adjacent time.

The costs structure, supplier by provider

Every awesome supplier prices caching differently, and the differences are not cosmetic. They alteration which workloads caching helps.

Anthropic models connected DigitalOcean usage definitive caching. You people a bound successful your petition pinch a cache_control section of type: ephemeral and a ttl of either 5m (default) aliases 1h. The first petition that writes that bound is billed astatine a premium complete the modular input rate, and DigitalOcean’s ain pricing page confirms the multipliers: for Claude Haiku 4.5, modular input is $1.00 per cardinal tokens, 5-minute cache creation is $1.25 (1.25x), 1-hour cache creation is $2.00 (2.0x), and cache publication is $0.10 (0.10x, a 90 percent discount). Those aforesaid 1.25x / 2.0x / 0.10x ratios clasp crossed the Claude family connected DigitalOcean. The consequence usage entity reports cache_creation_input_tokens connected the penning petition and cache_read_input_tokens connected consequent hits.

Confirm this against the existent docs for your circumstantial model. Sources: DigitalOcean Prompt Caching how-to and Inference pricing, past verified connected July 14, 2026; Anthropic punctual caching documentation.

OpenAI models connected DigitalOcean cache prompts of 1,024 tokens aliases more, and connected DigitalOcean you opt successful per petition pinch a prompt_cache_retention parameter group to either in_memory aliases 24h. Caching is best-effort: it applies erstwhile a request’s input tokens lucifer the prefix of a erstwhile response, but a deed is not guaranteed. The cache-read discount is not a azygous number, and this is the item worthy getting right, because DigitalOcean’s ain pricing page shows it varies sharply by model. On the GPT-4o family the cached-read complaint is precisely half the input complaint (GPT-4o: $2.50 input, $1.25 cached read), a 50 percent discount. On GPT-4.1 and o3 it is 75 percent off. On the GPT-5 family it is 90 percent disconnected (GPT-5: $1.25 input, $0.125 cached read). These figures are confirmed straight connected DigitalOcean’s ain Inference Pricing page, verified July 14, 2026. Separately, OpenAI’s ain pricing archiving states that the newest GPT-5.6 models (Sol, Terra, Luna) adhd a cache-write complaint of 1.25 times the input price, which would make them behave much for illustration Anthropic’s definitive exemplary than older OpenAI models. I could not corroborate this circumstantial statement point connected DigitalOcean’s ain pricing page, since GPT-5.6 became disposable connected DigitalOcean aft that page’s past verification date. Verify the existent complaint for your circumstantial exemplary connected the Inference pricing page alternatively than assuming a level percentage. Source: DigitalOcean Prompt Caching how-to and Inference pricing, verified July 14, 2026.

Open-source models connected DigitalOcean’s Serverless Inference Support punctual caching. As per DigitalOcean’s existent documentation, punctual caching for open-source models is successful nationalist preview, and models specified arsenic DeepSeek V3.2 cache automatically. You do not group cache_control aliases prompt_cache_retention astatine all; caching is applied best-effort whenever a request’s tokens lucifer the prefix of a erstwhile request. Caches are isolated per customer account, derived from a per-account key, truthful 1 relationship tin ne'er publication aliases infer another’s cached content. Two properties make the open-source way behave otherwise from the commercialized models above. First, location is nary 1,024-token minimum: DigitalOcean’s ain documented illustration shows a 214-token DeepSeek petition serving 128 tokens from cache. Second, location is nary fixed TTL; the docs authorities caches are held for an unbounded play but should beryllium treated arsenic opportunistic, truthful you should not dangle connected a cache deed for predictable cost. Cached tokens are reported successful 2 places connected the usage object, cache_read_input_tokens and prompt_tokens_details.cached_tokens, and are billed astatine the model’s discounted cache-read rate. That discount, from DigitalOcean’s pricing page, runs astir 46 to 80 percent disconnected input depending connected the exemplary (DeepSeek V3.2: $0.425 input, $0.15 cached read; Qwen3 Coder Flash: $0.45 input, $0.09 cached read), pinch nary abstracted constitute premium. Sources: DigitalOcean Prompt Caching how-to, Inference Features reference, and Inference pricing, verified July 14, 2026.

Note that not each open-source exemplary carries a managed caching rate. DigitalOcean’s pricing page lists a prompt-caching complaint for models for illustration DeepSeek V3.2, the Qwen 3 series, GLM, Kimi, and respective NVIDIA and MiniMax models, but not for others, including Meta’s Llama 3.3 70B, which is the exemplary this portion benchmarks. For the afloat existent list, spot Foundation Models.

Self-hosted models connected a DigitalOcean GPU Droplet are the fallback for immoderate unfastened exemplary without a managed caching rate, and for afloat power complete the caching layer. Modern conclusion engines, vLLM, SGLang, and TensorRT-LLM, each support automatic prefix caching wrong a replica, matching incoming prompts against antecedently cached KV authorities pinch nary personification configuration, arsenic DigitalOcean’s Advanced Prompt Caching astatine Scale engineering station explains successful detail. vLLM enables it by default. There is nary constitute premium, nary per-token discount to track, and, arsenic the measured tally later successful this portion confirms, nary 1,024-token minimum: vLLM caches successful 16-token blocks, and a 155-token punctual deed the cache connected its 2nd occurrence. You salary for the GPU by the hr ($3.44 per hr for the single-H200 Droplet utilized successful this piece’s test, per GPU Droplet pricing) whether the cache hits aliases not, truthful the use shows up arsenic latency and capacity alternatively than a line-item discount: each cached prefill is GPU clip freed for serving different requests.

All DigitalOcean figures beneath are from DigitalOcean’s ain prompt caching how-to and Inference pricing pages, verified arsenic of July, 2026.

Implementing caching connected DigitalOcean, supplier by provider

All 3 managed paths usage the aforesaid OpenAI-compatible endpoint, https://inference.do-ai.run/v1, and the aforesaid model entree key successful the Authorization header. What changes per supplier is really you petition caching and wherever the cache counts show up successful the usage object. The examples beneath usage the Python modular room truthful they tally pinch nary dependencies, and they travel DigitalOcean’s Prompt Caching how-to exactly. Set your cardinal once:

export DO_MODEL_ACCESS_KEY="your-model-access-key"

A mini shared helper sounds the caching fields, which disagree somewhat crossed providers:

import os import json import urllib.request BASE_URL = "https://inference.do-ai.run/v1" HEADERS = { "Content-Type": "application/json", "Authorization": f"Bearer {os.environ['DO_MODEL_ACCESS_KEY']}", } def post_chat(body): req = urllib.request.Request( BASE_URL + "/chat/completions", data=json.dumps(body).encode(), headers=HEADERS, ) with urllib.request.urlopen(req, timeout=120) as r: return json.loads(r.read()) def cache_report(usage): """Normalize the cache fields DigitalOcean returns crossed exemplary families.""" specifications = usage.get("prompt_tokens_details") or {} return { "prompt_tokens": usage.get("prompt_tokens"), "cache_created": usage.get("cache_created_input_tokens", 0), "cache_read": usage.get("cache_read_input_tokens", 0), # open-source models besides expose this field: "cached_tokens": details.get("cached_tokens", 0), }

Anthropic models: definitive cache_control. You people the extremity of the unchangeable artifact pinch a cache_control breakpoint of type: ephemeral and a ttl of 5m (default) aliases 1h. Everything up to and including that artifact is cached. Put the large, unchangeable contented (system instructions, instrumentality definitions, reference documents) first, people it, past append the volatile personification turn:

STABLE_SYSTEM_PROMPT = "..." # your long, unchangeable instructions and reference material def anthropic_turn(user_message, ttl="1h"): assemblage = { "model": "claude-haiku-4.5", "messages": [ { "role": "system", "content": [ { "type": "text", "text": STABLE_SYSTEM_PROMPT, "cache_control": {"type": "ephemeral", "ttl": ttl}, } ], }, {"role": "user", "content": user_message}, ], } consequence = post_chat(body) return cache_report(response["usage"]) # First telephone writes the cache (cache_created > 0); the 2nd sounds it (cache_read > 0). print(anthropic_turn("Summarize the attached argumentation successful 3 bullets.")) print(anthropic_turn("Now database immoderate compliance risks."))

On the first petition the usage entity reports cache_created_input_tokens greater than 0 and cache_read_input_tokens astatine 0; connected the repetition wrong the TTL model it flips, cache_read_input_tokens becomes the cached count and cache_created_input_tokens returns to 0. That publication is billed astatine 0.10x the modular input rate.

OpenAI models: opt successful pinch prompt_cache_retention. No cache_control markers; you group 1 top-level parameter to in_memory aliases 24h, and caching applies best-effort to immoderate punctual of 1,024 tokens aliases more. Keep the unchangeable contented astatine the beforehand of the connection database truthful the matching prefix is arsenic agelong arsenic possible:

def openai_turn(messages, retention="24h"): assemblage = { "model": "gpt-5", "prompt_cache_retention": retention, # "in_memory" aliases "24h" "messages": messages, "temperature": 0.2, } consequence = post_chat(body) return cache_report(response["usage"]) history = [ {"role": "system", "content": STABLE_SYSTEM_PROMPT}, # >= 1,024 tokens to beryllium eligible {"role": "user", "content": "Summarize the attached argumentation successful 3 bullets."}, ] print(openai_turn(history)) # cache_created_input_tokens connected first eligible call history.append({"role": "user", "content": "Now database immoderate compliance risks."}) print(openai_turn(history)) # cache_read_input_tokens connected the repeat

The cached-read discount varies by exemplary (50% disconnected connected GPT-4o, 75% connected GPT-4.1 and o3, 90% connected GPT-5), truthful publication the returned cache_read_input_tokens against your model’s complaint alternatively than assuming a level percentage.

Open-source models: thing to configure. DeepSeek V3.2 and the different open-source models pinch a managed caching complaint cache automatically. You nonstop an mean request, and cached tokens look successful some cache_read_input_tokens and prompt_tokens_details.cached_tokens:

def open_source_turn(messages): assemblage = { "model": "deepseek-3.2", # caches automatically, nary cache params "messages": messages, "max_tokens": 512, } consequence = post_chat(body) return cache_report(response["usage"]) history = [ {"role": "system", "content": STABLE_SYSTEM_PROMPT}, {"role": "user", "content": "Summarize the attached argumentation successful 3 bullets."}, ] print(open_source_turn(history)) history.append({"role": "user", "content": "Now database immoderate compliance risks."}) print(open_source_turn(history)) # cached_tokens > 0 erstwhile the prefix matches

The balanced earthy request, consecutive from DigitalOcean’s documentation, is simply a plain curl pinch nary caching parameters astatine all:

curl https://inference.do-ai.run/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $DO_MODEL_ACCESS_KEY" \ -d '{ "model": "deepseek-3.2", "messages": [ {"role": "user", "content": "Summarize this: ..."} ], "max_tokens": 512 }'

On a hit, the consequence usage artifact reports some fields, for illustration "cache_read_input_tokens": 128 and "prompt_tokens_details": {"cached_tokens": 128} against a prompt_tokens of 214, meaning 128 of the 214 input tokens were served from cache astatine the discounted complaint and the remaining 86 were billed astatine the modular rate.

For immoderate unfastened exemplary without a managed caching complaint (such arsenic the Llama 3.3 70B this portion benchmarks), self-host it connected a GPU Droplet, wherever vLLM applies prefix caching automatically pinch nary parameters and nary minimum. That path, and really to measurement it directly, is the taxable of the firsthand trial later successful this piece.

Prompt caching is not acold commencement mitigation

Prompt caching and acold commencement mitigation are different mechanisms solving different problems, and supplier trading sometimes blurs them, truthful it is worthy separating them explicitly.

Prompt caching addresses repeated input contented wrong an already-running exemplary deployment. It has thing to do pinch whether the exemplary itself is loaded into GPU memory. Cold starts reside the other problem: a exemplary deployment that scaled to zero during an idle play must reload earlier it tin service the first petition again. That reload clip shows up wholly successful time-to-first-token, the hold earlier the first output token streams back, and does not impact time-per-output-token, the steady-state procreation velocity erstwhile the consequence is underway.

On DigitalOcean’s Serverless Inference, acold starts tin hap aft an idle period, and the first petition pursuing that idle clip tin beryllium slower while capacity spins backmost up. This is the tradeoff for scaling to zero and paying thing while idle, and DigitalOcean’s ain guidance is that steady, latency-sensitive workloads are a amended fresh for a dedicated deployment pinch reserved capacity alternatively than serverless. Source: Where to Host Your Open-Source Model, DigitalOcean Community. For fine-tuned adapters specifically, DigitalOcean’s ain study recovered that acold starts spike time-to-first-token by astir a fewer 100 milliseconds erstwhile only a lightweight adapter needs reloading, and that this tin beryllium mitigated pinch periodic keep-alive requests. Source: Fine-Tuned LLMs connected Serverless Architecture, DigitalOcean Community.

What really reduces acold commencement latency is exemplary pre-loading, lukewarm pools of fresh capacity, aliases a dedicated deployment that ne'er scales to zero. What really reduces costs connected repeated input contented is punctual caching. Neither substitutes for the other, and a workload pinch some problems needs some solutions.

Caching skips reprocessing of repeated contented connected an already-running deployment. Cold commencement mitigation keeps the deployment itself fresh to serve. They lick different problems and neither fixes the other.

The break-even formula, derived from each provider’s ain numbers

Here is the portion astir explainers skip: an existent look you tin rerun pinch your ain provider’s published rates, not conscionable a header percentage.

Define the constitute multiplier w arsenic the costs of the first petition that establishes a cached prefix, comparative to modular input price, and the publication multiplier r arsenic the costs of a consequent petition that hits the aforesaid cached prefix, besides comparative to modular price. The number of cache hits needed earlier caching becomes cheaper than ne'er caching astatine all, which I will telephone the break-even constituent h*, is:

h* = (w − 1) / (1 − r)

Apply Anthropic’s ain published numbers. On the 5-minute tier, w = 1.25 and r = 0.10, giving h* = 0.25 / 0.90 ≈ 0.28. Since you cannot person a fractional hit, this rounds up to 1: aft a azygous cache hit, the 2nd petition successful a two-request sequence, the mixed costs of constitute positive that 1 publication is already little than paying afloat value twice. I verified this directly: 2 full-price requests costs 2.0 times the modular rate, while 1 constitute positive 1 publication costs 1.25 positive 0.10, aliases 1.35, which is cheaper. On the 1-hour tier, w = 2.0, giving h* = 1.0 / 0.90 ≈ 1.11, rounding up to 2 hits needed earlier the longer, much costly constitute pays for itself.

Apply the numbers for a no-write-premium model, which connected DigitalOcean covers astir OpenAI models (GPT-4o, GPT-4.1, GPT-5, the o-series) and each open-source model. Here w = 1, and the look gives h* = 0. Every azygous cache deed is axenic savings from the very first one, because location was ne'er an other costs to recover. This is the applicable quality betwixt definitive and automatic caching: Anthropic’s exemplary tin suffer money connected a workload pinch excessively fewer hits, while a zero-premium exemplary structurally cannot. One caveat worthy flagging directly: OpenAI’s ain pricing archiving states that GPT-5.6 (Sol, Terra, Luna) introduces a cache-write complaint of 1.25 times the uncached input rate, a alteration from earlier OpenAI models. I could not corroborate this statement point connected DigitalOcean’s ain Inference Pricing page arsenic of this writing, since that page was past verified July 1, 2026, earlier GPT-5.6 became disposable connected DigitalOcean’s platform. If you are utilizing GPT-5.6 done DigitalOcean, cheque the unrecorded pricing page for the existent cache-write complaint earlier assuming h* = 0.

Each statement shows the constituent wherever caching becomes cheaper than paying modular value each time. A constitute premium intends the first petition costs more, and you person to gain that backmost earlier caching pays disconnected astatine all.

What this look does and does not show you

This look tells you, erstwhile you cognize a provider’s rates, precisely really galore hits you need. It does not show you whether your postulation will really nutrient that galore hits. That 2nd mobility depends connected the 3 variables named successful this piece’s thesis, and each maps to a specific, measurable nonaccomplishment mode.

Prefix stability beneath the minimum cacheable magnitude is simply a difficult gross for commercialized models, not a matter of degree. If your full punctual is nether 1,024 tokens, Anthropic and OpenAI models (on DigitalOcean and elsewhere) will not cache immoderate of it, nary matter really galore times you repetition it aliases really quickly requests arrive. This is worthy naming straight against the chatbot workload this portion uses arsenic an illustration later: a 400-token strategy punctual is beneath that period connected both. That workload cannot cache astatine each connected those models, independent of convention extent aliases timing. The gross does not use everywhere, though: DigitalOcean’s open-source models person nary documented minimum (the docs show caching astatine 214 tokens), and the measured short-prompt trial later successful this portion shows a self-hosted motor caches astatine 155 tokens. The gross is simply a commercial-model policy, not a spot of the mechanism, which matters if a small-prompt workload is pushing you toward an open-source aliases self-hosted model.

Session depth beneath h* for your provider’s tier intends you ne'er retrieve the constitute premium, if 1 exists. A azygous one-shot petition that writes a cache introduction and is ne'er followed by a matching petition pays the premium and gets thing back. This is simply a consecutive nonaccomplishment connected immoderate supplier that charges one.

Interarrival time exceeding the cache life intends each petition is treated arsenic caller sloppy of convention depth. If your requests are 7 minutes isolated and you are connected Anthropic’s 5-minute tier, each azygous petition rewrites the cache and nary of them publication it, guaranteeing you salary the constitute premium each clip and ne'er person the discount. The hole present is not to wantonness caching, it is to usage the 1-hour tier if your postulation is sparse but yet repeats, accepting the higher h* of 2 hits successful speech for a wider model to scope them.

A worked example, pinch the deed complaint measured

To show really the look scales pinch existent convention depth, return a 20-turn supplier convention connected Anthropic’s 5-minute tier, wherever a unchangeable 3,000-token strategy punctual and instrumentality meaning artifact is reused connected each move and each move arrives good wrong 5 minutes of the last. The pricing multipliers present are Anthropic’s published rates. The deed rate, 1 miss connected the first move and hits connected each move after, is not an presumption successful this piece: the measured 12-turn convention successful the firsthand trial below, tally against Llama 3.3 70B connected a DigitalOcean GPU Droplet pinch precisely this postulation shape, produced a cache deed connected 11 of 11 consequent turns pinch a 98.7 percent token-level deed rate. A unchangeable prefix reused wrong the cache model hits fundamentally each time, connected each caching implementation this portion examined.

Apply that deed shape to Anthropic’s pricing. The tokens successful the unchangeable artifact are billed arsenic 1 constitute (1.25x) positive 19 sounds (0.10x each), for a full of 1.25 + 1.9 = 3.15 times the modular complaint crossed 20 requests, versus 20.0 times the modular complaint pinch nary caching astatine all. That is astir an 84 percent simplification connected that information of the input, good supra the 1-hit break-even point, and accordant pinch the “50 to 90 percent” scope reported crossed the manufacture sources I reviewed for agelong supplier sessions. Contrast a 3-turn chatbot convention pinch a 4,000-token strategy prompt, comfortably supra the minimum, but only 2 opportunities to hit. At 2 hits: 1.25 + 0.20 = 1.45 times modular crossed 3 requests, versus 3.0 times modular uncached, a 52 percent simplification connected that portion, smaller successful absolute position because location are less requests to dispersed the constitute costs across, but still past break-even. The chatbot illustration that fails outright is not this one. It is the 400-token punctual from the earlier gate, which ne'er qualifies to cache connected a managed supplier successful the first place.

The 1 caveat the measured tally adds: the near-perfect deed complaint holds only erstwhile the prefix is byte-for-byte stable. The aforesaid measured convention rerun pinch a timestamp and petition ID astatine the apical of the strategy prompt, everything other identical, dropped from a 98.7 percent token deed complaint to 0.7 percent. If your worked illustration assumes 19 of 20 hits but your punctual template injects thing move up of the unchangeable block, your existent number is the 2nd one. Measure your ain postulation pinch the harness successful the adjacent conception earlier you scheme a fund astir either figure.

The firsthand test: measuring cache behaviour connected your ain traffic

This is the conception that matters astir successful this article, because it replaces assumptions pinch a tally you tin reproduce. The trial harness beneath was executed against a unrecorded exemplary serving endpoint, and each number successful the tables that travel came straight from that trial run. The managed-platform DIY trial harness is included astatine the extremity of this conception for anyone pinch a key. The caching system being measured, exact-prefix KV reuse, is the aforesaid 1 each supplier implements, truthful the hit-rate behaviour transfers moreover wherever the billing exemplary differs.

The measured environment

• Infrastructure: DigitalOcean GPU Droplet, gpu-h200x1-141gb (1x NVIDIA H200, 141 GB VRAM, 24 vCPUs, 240 GB RAM), $3.44 per hour, NYC2 region, created from the 1-Click Inference Ready image. See GPU Droplet pricing.
• Serving stack: vLLM 0.24.0 (vllm/vllm-openai:v0.24.0 Docker image), which enables automatic prefix caching by default (enable_prefix_caching=True successful the motor startup log).
• Model: RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic, FP8 KV cache, 4,096-token context, azygous GPU.

The nonstop serving command, truthful the situation is reproducible:

docker tally -d --name vllm-bench --gpus each \ -v /root/.cache/huggingface:/root/.cache/huggingface \ -p 8000:8000 \ vllm/vllm-openai:v0.24.0 \ RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic \ --kv-cache-dtype fp8 \ --max-model-len 4096 \ --max-num-seqs 256 \ --gpu-memory-utilization 0.92 \ --tensor-parallel-size 1 \ --port 8000

nvidia-smi connected the Droplet astatine trial time, confirming the hardware and the loaded motor (output trimmed to the applicable lines):

+-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 12.9 | |-----------------------------------------+------------------------+----------------------+ | 0 NVIDIA H200 On | 00000000:83:00.0 Off | 0 | | N/A 43C P0 125W / 700W | 134095MiB / 143771MiB | 0% Default | +-----------------------------------------+------------------------+----------------------+ | 0 N/A N/A 1644641 C VLLM::EngineCore 13408... | +-----------------------------------------------------------------------------------------+

And the engine’s ain startup log statement confirming prefix caching is progressive pinch nary configuration connected our part:

Initializing a V1 LLM motor (v0.24.0) pinch config: model='RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic', ... kv_cache_dtype=fp8, ... enable_prefix_caching=True, enable_chunked_prefill=True, ...

Test design

Four synthetic workloads against the aforesaid model, holding everything changeless isolated from postulation shape:

• TTFT sweep. Unique prefixes of astir 512, 1,024, 2,048, and 3,072 tokens, each sent erstwhile acold and past 3 times warm, to measurement the latency worth of a deed arsenic prefix size grows.
• Agent workload. A unchangeable 2,048-token strategy punctual followed by 12 sequential turns pinch increasing speech history, each move arriving astir 1 2nd aft the last.
• Prefix-buster workload. The identical supplier workload, pinch 1 change: a petition ID and timestamp injected astatine the top of the strategy punctual connected each turn, the azygous astir communal caching correction successful production.
• Short-prompt workload. A 117-token strategy punctual repeated 4 times, to trial whether the 1,024-token minimum that commercialized models enforce exists astatine the motor level.

What was measured, and how

vLLM does not return per-request cached-token counts successful the OpenAI-compatible usage entity the measurement managed providers do, but it exposes engine-level counters connected its Prometheus /metrics endpoint: vllm:prefix_cache_queries_total (tokens checked against the cache) and vllm:prefix_cache_hits_total (tokens served from it). One naming note: vLLM’s ain metrics archiving lists these arsenic vllm:prefix_cache_queries and vllm:prefix_cache_hits without the _total suffix. Prometheus customer libraries conventionally append _total to Counter-type metrics successful the existent exposed matter output, which is accordant pinch the curl output below, but if your type differs, database your endpoint’s existent metric names pinch curl localhost:8000/metrics | grep prefix_cache earlier relying connected the nonstop string. The harness issues requests serially and diffs those counters astir each request, which attributes deed tokens to individual requests exactly. Time-to-first-token is measured client-side pinch streaming enabled, timestamping the first contented chunk.

This is what those counters look like, queried straight connected the Droplet:

curl -s localhost:8000/metrics | grep -E "^vllm:prefix_cache_(queries|hits)_total" vllm:prefix_cache_queries_total{engine="0",model_name="RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic"} 2.99845e+06 vllm:prefix_cache_hits_total{engine="0",model_name="RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic"} 194672.0

The trial harness

This is the complete book I wrote to nutrient each measured number successful this article, cache_bench.py, runnable against immoderate vLLM endpoint pinch nary limitations beyond the Python modular library. Save it, constituent --base-url astatine your endpoint, and it runs each 4 workloads and writes the earthy per-request records to a JSON file.

The measurement shape is worthy stealing moreover if you alteration the workloads: measured_turn diffs the engine’s prefix-cache counters astir each serial request, which attributes cached tokens to individual requests exactly, and make_prefix calibrates punctual sizes against vLLM’s /tokenize endpoint truthful “a 2,048-token prefix” intends what it says.

cache_bench.py

#!/usr/bin/env python3 """ Prefix-cache benchmark for a vLLM OpenAI-compatible endpoint. Measures, per request: - TTFT (time to first streamed token) - full latency - punctual tokens (from the usage object) - cached prefix tokens (by diffing vLLM's /metrics prefix-cache counters astir each request, truthful requests are issued serially) Workloads: 1. ttft_sweep - acold vs lukewarm TTFT crossed prefix sizes 2. agent_session - unchangeable strategy punctual + increasing history, N turns 3. prefix_buster - identical convention but a timestamp astatine the TOP of the strategy prompt, which breaks exact-prefix matching 4. short_prompt - a ~120-token punctual repeated, to observe block-level behaviour beneath emblematic managed-provider minimums Run connected the droplet (or anyplace that tin scope the endpoint): python3 cache_bench.py --base-url http://localhost:8000 --out results.json """ import argparse import json import time import urllib.request import uuid WORD = "inference " # 1 connection ~= 1.2 tokens for this tokenizer; calibrated below def http_json(url, payload=None, timeout=120): information = json.dumps(payload).encode() if payload is not None else None req = urllib.request.Request( url, data=data, headers={"Content-Type": "application/json"} ) with urllib.request.urlopen(req, timeout=timeout) as r: return json.loads(r.read()) def get_cache_counters(base_url): with urllib.request.urlopen(base_url + "/metrics", timeout=30) as r: matter = r.read().decode() queries = hits = 0.0 for statement in text.splitlines(): if line.startswith("vllm:prefix_cache_queries_total"): queries = float(line.rsplit(" ", 1)[1]) elif line.startswith("vllm:prefix_cache_hits_total"): hits = float(line.rsplit(" ", 1)[1]) return queries, hits def streamed_request(base_url, model, messages, max_tokens=32): """Send a streaming chat completion; return TTFT, full time, usage.""" payload = { "model": model, "messages": messages, "max_tokens": max_tokens, "temperature": 0, "stream": True, "stream_options": {"include_usage": True}, } req = urllib.request.Request( base_url + "/v1/chat/completions", data=json.dumps(payload).encode(), headers={"Content-Type": "application/json"}, ) commencement = time.perf_counter() ttft = None usage = {} content_parts = [] with urllib.request.urlopen(req, timeout=300) as r: for earthy in r: statement = raw.decode().strip() if not line.startswith("data:"): continue assemblage = line[5:].strip() if assemblage == "[DONE]": break chunk = json.loads(body) if chunk.get("usage"): usage = chunk["usage"] for prime in chunk.get("choices", []): delta = choice.get("delta", {}) if delta.get("content"): if ttft is None: ttft = time.perf_counter() - start content_parts.append(delta["content"]) full = time.perf_counter() - start return { "ttft_s": ttft, "total_s": total, "prompt_tokens": usage.get("prompt_tokens"), "completion_tokens": usage.get("completion_tokens"), "content": "".join(content_parts), } def measured_turn(base_url, model, messages, max_tokens=32): q0, h0 = get_cache_counters(base_url) consequence = streamed_request(base_url, model, messages, max_tokens) q1, h1 = get_cache_counters(base_url) result["cache_query_tokens"] = int(q1 - q0) result["cache_hit_tokens"] = int(h1 - h0) return result def make_prefix(base_url, model, target_tokens): """Build a unsocial prefix of astir target_tokens, calibrated via tokenize API.""" tag = f"run-{uuid.uuid4().hex[:8]} " words = int(target_tokens * 0.9) matter = tag + (WORD * words) # calibrate pinch the /tokenize endpoint for _ in range(6): n = http_json( base_url + "/tokenize", {"model": model, "prompt": text} )["count"] if abs(n - target_tokens) <= 8: break delta_words = int((target_tokens - n) * 0.8) if delta_words > 0: matter += WORD * delta_words else: matter = text[: delta_words * len(WORD)] return text, n def ttft_sweep(base_url, model, sizes, warm_repeats=3): retired = [] for size in sizes: prefix, existent = make_prefix(base_url, model, size) messages = [ {"role": "system", "content": prefix}, {"role": "user", "content": "Reply pinch the azygous word: ready."}, ] acold = measured_turn(base_url, model, messages, max_tokens=8) warms = [ measured_turn(base_url, model, messages, max_tokens=8) for _ in range(warm_repeats) ] out.append( { "target_prefix_tokens": size, "actual_prefix_tokens": actual, "cold": cold, "warm": warms, } ) print(f"[ttft_sweep] {size} tokens done", flush=True) return out def agent_session(base_url, model, prefix_tokens=2048, turns=12, bust_prefix=False): prefix, existent = make_prefix(base_url, model, prefix_tokens) results = [] history = [] for one in range(turns): strategy = prefix if bust_prefix: # per-request adaptable contented astatine the TOP: the classical mistake strategy = f"[request-id: {uuid.uuid4()}] [ts: {time.time()}]\n" + prefix messages = ( [{"role": "system", "content": system}] + history + [{"role": "user", "content": f"Turn {i}: reply successful 5 words aliases fewer."}] ) r = measured_turn(base_url, model, messages, max_tokens=16) r["turn"] = i results.append(r) history.append({"role": "user", "content": f"Turn {i}: reply successful 5 words aliases fewer."}) history.append({"role": "assistant", "content": r["content"] or "ok"}) time.sleep(1) return {"prefix_tokens": actual, "turns": results} def short_prompt(base_url, model, repeats=4): prefix, existent = make_prefix(base_url, model, 120) messages = [ {"role": "system", "content": prefix}, {"role": "user", "content": "Reply pinch 1 word."}, ] return { "prefix_tokens": actual, "requests": [ measured_turn(base_url, model, messages, max_tokens=8) for _ in range(repeats) ], } def main(): ap = argparse.ArgumentParser() ap.add_argument("--base-url", default="http://localhost:8000") ap.add_argument("--out", default="cache_bench_results.json") args = ap.parse_args() exemplary = http_json(args.base_url + "/v1/models")["data"][0]["id"] print(f"Model: {model}", flush=True) results = {"model": model, "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())} print("== TTFT expanse ==", flush=True) results["ttft_sweep"] = ttft_sweep(args.base_url, model, [512, 1024, 2048, 3072]) print("== Agent convention (stable prefix) ==", flush=True) results["agent_session"] = agent_session(args.base_url, model, 2048, 12) print("== Prefix-buster convention (timestamp first) ==", flush=True) results["prefix_buster"] = agent_session(args.base_url, model, 2048, 12, bust_prefix=True) print("== Short punctual ==", flush=True) results["short_prompt"] = short_prompt(args.base_url, model) with open(args.out, "w") as f: json.dump(results, f, indent=2) print(f"Wrote {args.out}", flush=True) if __name__ == "__main__": main()

The afloat run, executed connected the Droplet:

python3 cache_bench.py --base-url http://localhost:8000 --out cache_bench_results.json Model: RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic == TTFT expanse == [ttft_sweep] 512 tokens done [ttft_sweep] 1024 tokens done [ttft_sweep] 2048 tokens done [ttft_sweep] 3072 tokens done == Agent convention (stable prefix) == == Prefix-buster convention (timestamp first) == == Short punctual == Wrote cache_bench_results.json

Here is really the results look like:

cache_bench_results.json

{ "model": "RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic", "timestamp": "2026-07-14T13:45:35Z", "ttft_sweep": [ { "target_prefix_tokens": 512, "actual_prefix_tokens": 510, "cold": { "ttft_s": 0.11955002718605101, "total_s": 0.14034194918349385, "prompt_tokens": 551, "completion_tokens": 2, "content": "ready", "cache_query_tokens": 551, "cache_hit_tokens": 16 }, "warm": [ { "ttft_s": 0.03656787099316716, "total_s": 0.05752997612580657, "prompt_tokens": 551, "completion_tokens": 2, "content": "ready", "cache_query_tokens": 551, "cache_hit_tokens": 544 }, { "ttft_s": 0.03680372005328536, "total_s": 0.05818751105107367, "prompt_tokens": 551, "completion_tokens": 2, "content": "ready", "cache_query_tokens": 551, "cache_hit_tokens": 544 },

Every array beneath is publication straight retired of that results file, and the earthy per-request records are shown alongside each one.

Measured consequence 1: a cache deed trim TTFT by 69 to 92 percent, scaling pinch prefix size

Each statement is 1 unsocial prefix sent acold (first occurrence, nary cache entry) and past 3 times lukewarm (identical prefix, cache populated). Warm TTFT is the mean of the 3 repeats, which varied by little than 2 ms.

Cold TTFT scales pinch prefix size because prefill does. Warm TTFT does not, because a cache deed skips prefill for the matched information entirely.

The earthy records for the 3,072-token case, precisely arsenic the harness logged them:

{"cold": {"ttft_s": 0.4615, "prompt_tokens": 3110, "cache_query_tokens": 3110, "cache_hit_tokens": 16}, "warm": [ {"ttft_s": 0.0389, "prompt_tokens": 3110, "cache_query_tokens": 3110, "cache_hit_tokens": 3104}, {"ttft_s": 0.0384, "prompt_tokens": 3110, "cache_query_tokens": 3110, "cache_hit_tokens": 3104}, {"ttft_s": 0.0383, "prompt_tokens": 3110, "cache_query_tokens": 3110, "cache_hit_tokens": 3104}]}

Two things are worthy noting successful this table. First, acold TTFT grows linearly pinch punctual size, astatine astir 134 sclerosis per 1,000 punctual tokens connected this hardware and model, which useful retired to a prefill throughput of astir 7,500 tokens per second.

That slope is the point caching deletes. Second, lukewarm TTFT is level astatine 37 to 39 sclerosis regardless of prefix size, because a deed skips prefill for the matched information entirely. The longer your unchangeable prefix, the much a deed is worth, which is the measured confirmation of why the determination model successful this portion weights prefix size truthful heavily. The 92 percent simplification astatine 3,110 tokens lands astatine the apical of the “up to 80 percent” scope OpenAI advertises for its ain managed caching, measured present connected unfastened root infrastructure anyone tin rent by the hour.

Cached-token counts get successful multiples of 16 because vLLM caches KV authorities successful 16-token blocks; the fewer uncached tokens connected each lukewarm petition are the last partial block. The blocks holding the chat template header are shared crossed each requests to the endpoint, which is why moreover a “cold” petition shows 16 cached tokens.

Measured consequence 2: a 12-turn supplier convention sustained a 98.7 percent deed complaint aft move one

This is the workload the worked illustration earlier successful this article models: a unchangeable 2,080-token strategy prompt, past 12 turns of accumulating speech history, 1 2nd apart.

One acold turn, past eleven flat, accelerated ones. Each move extends the erstwhile turn’s cached prefix, truthful the deed complaint compounds alternatively of resetting.

The earthy records for the first 4 turns and the last:

{"turn": 0, "prompt_tokens": 2084, "cache_query_tokens": 2084, "cache_hit_tokens": 16, "ttft_s": 0.3267} {"turn": 1, "prompt_tokens": 2110, "cache_query_tokens": 2110, "cache_hit_tokens": 2080, "ttft_s": 0.0416} {"turn": 2, "prompt_tokens": 2136, "cache_query_tokens": 2136, "cache_hit_tokens": 2112, "ttft_s": 0.0425} {"turn": 3, "prompt_tokens": 2163, "cache_query_tokens": 2163, "cache_hit_tokens": 2128, "ttft_s": 0.0394} {"turn": 11, "prompt_tokens": 2372, "cache_query_tokens": 2372, "cache_hit_tokens": 2336, "ttft_s": 0.0377}

Across turns 1 done 11, the token-level deed complaint was 98.7 percent and mean TTFT was 40 ms, versus 327 sclerosis connected the acold first turn. Note what is being cached by move 3: not conscionable the strategy punctual but the accumulated speech history too, because each turn’s punctual is simply a strict prefix hold of the erstwhile turn’s. This is the append-only spot that makes supplier loops the best-case caching workload, and it is why the earlier worked example’s “hit connected each move aft the first” shape is simply a measurement here, not an assumption.

Measured consequence 3: 1 timestamp astatine the apical of the punctual collapsed the deed complaint from 98.7 to 0.7 percent

The aforesaid 12-turn session, rerun pinch precisely 1 change: each turn’s strategy punctual began pinch [request-id: <uuid>] [ts: <time>] up of the unchangeable 2,080-token block.

Same session, aforesaid 2,080-token unchangeable block, 1 difference. Exact-prefix matching intends thing aft the first changed byte is ever reached, truthful the harm is total, not partial.

The earthy records for the first 4 turns, broadside by broadside pinch the stable-prefix records above. The only cached tokens connected immoderate move are the 16-token chat-template block; the 2,080-token unchangeable assemblage down the timestamp ne'er hits:

{"turn": 0, "prompt_tokens": 2122, "cache_query_tokens": 2122, "cache_hit_tokens": 16, "ttft_s": 0.3150} {"turn": 1, "prompt_tokens": 2152, "cache_query_tokens": 2152, "cache_hit_tokens": 16, "ttft_s": 0.3137} {"turn": 2, "prompt_tokens": 2176, "cache_query_tokens": 2176, "cache_hit_tokens": 16, "ttft_s": 0.3133} {"turn": 3, "prompt_tokens": 2202, "cache_query_tokens": 2202, "cache_hit_tokens": 16, "ttft_s": 0.3048}

Every move paid afloat prefill, astir 8x the lukewarm TTFT, and connected a supplier pinch per-token pricing each move would person paid afloat input value too. Nothing astir the workload’s convention depth, interarrival time, aliases prefix size changed. The cache was defeated purely by punctual structure, and the harm was total, not partial, because exact-prefix matching intends thing aft the first changed byte is ever reached. This is the azygous astir communal accumulation caching mistake, and this is what it measures as.

Measured consequence 4: the 1,024-token minimum is simply a commercial-model policy, not an motor limit

A 117-token strategy prompt, 155 punctual tokens successful total, acold beneath the commercial-model minimum, repeated 4 times:

Anthropic and OpenAI will not cache this punctual astatine all, since it is beneath their 1,024-token floor. A self-hosted motor has nary specified level and cached it astatine vLLM’s 16-token artifact granularity instead.

The earthy records:

{"prompt_tokens": 155, "cache_query_tokens": 155, "cache_hit_tokens": 16, "ttft_s": 0.0491} {"prompt_tokens": 155, "cache_query_tokens": 155, "cache_hit_tokens": 144, "ttft_s": 0.0368} {"prompt_tokens": 155, "cache_query_tokens": 155, "cache_hit_tokens": 144, "ttft_s": 0.0352} {"prompt_tokens": 155, "cache_query_tokens": 155, "cache_hit_tokens": 144, "ttft_s": 0.0367}

vLLM cached it without complaint, astatine its 16-token artifact granularity. The 1,024-token minimum that Anthropic and OpenAI models enforce is simply a argumentation prime astir erstwhile caching is worthy the bookkeeping, not a spot of the mechanism. Two paths connected DigitalOcean flight that floor: its managed open-source models, which the docs show caching astatine 214 tokens, and a self-hosted motor for illustration the 1 measured here, caching astatine 155. So a small-prompt workload that is structurally ineligible connected Anthropic aliases OpenAI models is not retired of options; it conscionable needs an open-source aliases self-hosted model. That changes the earlier difficult gross from “cannot cache astatine all” to “cannot cache connected a commercialized model.”

What the measured numbers mean for costs connected self-hosted infrastructure

On a per-token provider, a cache deed is simply a line-item discount. On an hourly GPU, the accounting is different: the H200 costs $3.44 per hr whether the cache hits aliases not, truthful the redeeming arrives arsenic reclaimed capacity. At the measured acold prefill complaint of astir 7,500 tokens per second, each cardinal cached punctual tokens is astir 134 GPU-seconds of prefill activity the paper does not repeat, which is GPU clip disposable for decoding different requests instead. For a latency-sensitive workload, the much nonstop reference is the TTFT array above: the quality betwixt 462 sclerosis and 39 sclerosis to first token is the quality users really feel, connected infrastructure whose caching behaviour you afloat control.

One scaling caveat that matters arsenic soon arsenic you outgrow 1 GPU: these deed rates were measured connected a azygous replica, wherever each petition lands connected the aforesaid cache. Under round-robin load balancing crossed N replicas, an identical prefix has astir a 1-in-N chance of landing wherever its cache introduction lives, and the deed complaint measured present degrades accordingly. Session-affinity routing, pinning a session’s requests to 1 replica, is the modular fix, and DigitalOcean’s Advanced Prompt Caching astatine Scale blog covers that architecture, gradual prefix routing for multi-task deployments, and erstwhile a shared cross-replica cache is worthy its transportation latency.

Running the aforesaid trial against Serverless Inference

The DIY trial harness beneath targets DigitalOcean’s managed Serverless Inference endpoint, wherever caching shows up successful the consequence usage entity alternatively than successful motor metrics. The default exemplary present is deepseek-3.2, an open-source exemplary that, per DigitalOcean’s docs, caches automatically pinch nary cache_control aliases prompt_cache_retention parameter. The section to watch for open-source and Anthropic models is cache_read_input_tokens; open-source models additionally study prompt_tokens_details.cached_tokens. For OpenAI models, group prompt_cache_retention (shown commented out) and expect caching only connected prompts of 1,024 tokens aliases more. Source: DigitalOcean Prompt Caching how-to.

import time import os from openai import OpenAI client = OpenAI( api_key=os.environ["DO_MODEL_ACCESS_KEY"], base_url="https://inference.do-ai.run/v1", ) STABLE_PREFIX = "..." # your unchangeable strategy punctual and instrumentality definitions spell here MODEL = "deepseek-3.2" # open-source model; caches automatically, nary parameters needed def cached_tokens(usage): """Read cached-token count crossed the fields DO populates per exemplary family.""" nonstop = getattr(usage, "cache_read_input_tokens", 0) or 0 specifications = getattr(usage, "prompt_tokens_details", None) nested = getattr(details, "cached_tokens", 0) if specifications else 0 return max(direct, nested or 0) def run_turn(messages): commencement = time.perf_counter() consequence = client.chat.completions.create( model=MODEL, messages=messages, stream=False, # For OpenAI models, opt successful to caching (1,024-token minimum applies): # extra_body={"prompt_cache_retention": "24h"}, ) elapsed = time.perf_counter() - start usage = response.usage return { "elapsed_seconds": elapsed, "prompt_tokens": usage.prompt_tokens, "cache_read_input_tokens": getattr(usage, "cache_read_input_tokens", 0), "cached_tokens": cached_tokens(usage), } def agent_workload(turns=20, delay_seconds=2): history = [{"role": "system", "content": STABLE_PREFIX}] results = [] for one in range(turns): history.append({"role": "user", "content": f"Turn {i}: proceed the task."}) consequence = run_turn(history) results.append(result) # seizure the existent adjunct consequence truthful the prefix extends append-only: history.append({"role": "assistant", "content": "..."}) time.sleep(delay_seconds) return results def long_gap_workload(turns=5, delay_seconds=400): # For OpenAI models pinch prompt_cache_retention, a hold beyond the retention # model tests the interarrival nonaccomplishment mode. Open-source caches are unbounded # but best-effort, truthful a miss aft a agelong spread is imaginable but not guaranteed. return agent_workload(turns=turns, delay_seconds=delay_seconds) if __name__ == "__main__": print("Agent workload:") for r in agent_workload(): print(r) print("Long-gap workload:") for r in long_gap_workload(): print(r)

In the supra codification block, switch MODEL for an Anthropic exemplary (and adhd a cache_control marker) aliases an OpenAI exemplary (and uncomment prompt_cache_retention) to comparison mechanics crossed different exemplary families connected the aforesaid endpoint.

The determination framework

Cache when:

• Your stable, repeated contented is supra the model’s minimum cacheable length, 1,024 tokens for Anthropic and OpenAI models. Open-source models connected DigitalOcean and self-hosted engines person nary specified minimum.
• Your convention extent clears the break-even constituent h* for your model’s pricing, 1 deed connected Anthropic’s 5-minute tier, 2 hits connected its 1-hour tier, and efficaciously 0 hits connected immoderate exemplary pinch nary cache-write complaint (most OpenAI models and each open-source models connected DigitalOcean).
• Your requests get wrong the cache’s TTL of each other. For OpenAI models you tin group prompt_cache_retention up to 24 hours; open-source caches are unbounded but best-effort.
• You are moving supplier aliases coding workloads pinch fixed instrumentality definitions and large, unchangeable context, the script wherever convention extent and prefix size some activity successful your favour astatine once. The measured 12-turn supplier convention supra sustained a 98.7 percent token deed complaint nether precisely these conditions.
• You usage an open-source exemplary pinch a managed caching complaint (DeepSeek V3.2, Qwen 3, GLM, and others), wherever caching is automatic and needs nary parameters, aliases you self-host immoderate unfastened exemplary connected a GPU Droplet pinch vLLM, wherever prefix caching is connected by default pinch nary minimum prefix length.

Do not fuss when:

• Your punctual is smaller than 1,024 tokens and you are connected an Anthropic aliases OpenAI model. No magnitude of convention extent aliases timing fixes this there, though an open-source aliases self-hosted exemplary has nary specified floor.
• Your sessions are shallow enough, aliases your postulation sparse enough, that you will not clear h* earlier the cache expires, connected a exemplary that charges a cache-write premium (Anthropic and GPT-5.6).
• Your prompts alteration connected each petition successful ways that break exact-prefix matching, defeating the system sloppy of size aliases frequency.

Design your prompts for caching:

Put unchangeable contented first and volatile contented last. System prompt, past fixed instrumentality definitions, past agelong fixed reference material, past slowly-changing speech history, past the existent personification message, successful that order. Never spot a timestamp, a petition ID, aliases immoderate per-request adaptable up of the contented you want cached, since exact-prefix matching intends thing aft the first alteration is ne'er reached.

This ordering rule is accordant crossed each provider’s archiving I reviewed and is the azygous astir communal implementation correction reported successful accumulation lawsuit studies of teams recovering debased deed rates. It is besides nary longer conscionable a reported correction successful this piece: the measured prefix-buster tally supra put 1 timestamp statement astatine the apical of an different perfectly cacheable convention and watched the deed complaint autumn from 98.7 percent to 0.7 percent.

Three gates, checked successful order. A workload only needs to neglect 1 to suffer the economics, which is why convention extent alone, without checking prefix size and timing, is not capable to decide.

Prompt caching and exemplary routing are coupled

If you tally postulation done DigitalOcean’s Inference Router alternatively than a azygous fixed model, location is simply a system relationship worthy knowing directly, because it tin silently destruct the cache economics described successful this full piece.

The router tin prime a different exemplary for each petition successful a convention based connected what that circumstantial petition looks like, which is useful for costs governance crossed mixed workloads. But punctual caches are scoped to a circumstantial model. If your agent’s first move routes to 1 exemplary and its 2nd move routes to another, the cache written by the first move is ne'er publication by the second, and each move pays afloat constitute costs pinch nary accumulated savings, sloppy of really good your prefix and convention extent different fulfill the break-even look above.

DigitalOcean’s router exposes an X-Model-Affinity header specifically to forestall this. Setting it to a convention identifier connected the first petition causes the router to way usually once, past pin each consequent petition pinch the aforesaid identifier to that aforesaid model, skipping routing and preserving the cache. According to DigitalOcean’s ain documentation, successful a 15-turn agentic loop wherever 90 percent of the input is simply a cached, unchangeable prefix, this convention pinning behaviour is associated pinch 45 to 80 percent savings connected input token cost. Source: How to Use Inference Router, DigitalOcean Documentation.

The applicable rule: if you route, pin. Session pinning is not a abstracted optimization from punctual caching, it is the system that keeps caching’s economics intact erstwhile much than 1 exemplary is successful play.

Routing without pinning tin conclusion caching entirely, moreover connected a workload that would different clear each gross successful the determination model above.

Common questions connected this topic?

1. Does punctual caching alteration the model’s output?

No. Caching only affects really the input prefix is processed internally. The generated consequence is computed the aforesaid measurement whether the prefix was publication from cache aliases processed fresh, and providers authorities the output is unaffected by whether a cache deed occurred.

2. Can I harvester punctual caching pinch batch processing for other savings?

On providers that connection both, yes, and the discounts person been reported arsenic stacking. Confirm the existent mixed complaint connected your provider’s unrecorded pricing page, since stacked discounts are precisely the benignant of fig that changes betwixt pricing updates.

3. Does a shorter cache TTL ever mean little risk?

No, it intends a narrower model to cod hits earlier the cache expires and you person to constitute again. A shorter TTL fits tightly-spaced, high-frequency traffic. A longer TTL fits postulation that is sparse but still repeats wrong an hour, astatine the costs of a higher constitute premium and a higher break-even point, arsenic this piece’s look shows directly.

4. Why does my measured deed complaint not lucifer the discount percent a supplier advertises?

Advertised percentages picture the discount connected a successful cache read. Your effective savings crossed an full workload dangle connected what fraction of your full requests really hit, which is simply a spot of your traffic, not of the provider’s pricing table. This is the full logic this portion leads pinch a look and a trial harness alternatively of a percentage.

5. Do open-source models support punctual caching connected DigitalOcean’s Serverless Inference?

Yes. Per DigitalOcean’s documentation, punctual caching for open-source models is successful nationalist preview, and models specified arsenic DeepSeek V3.2, the Qwen 3 series, and GLM cache automatically pinch nary cache_control aliases prompt_cache_retention parameter. Caches are isolated per account, location is nary 1,024-token minimum, and cached tokens are billed astatine a discounted publication complaint (roughly 46 to 80 percent disconnected input, depending connected the model) pinch nary constitute premium. Not each open-source exemplary carries a managed caching rate, though; cheque the Foundation Models list.

6. What if my open-source exemplary does not person a managed caching rate?

Self-host it. vLLM, SGLang, and TensorRT-LLM each support automatic prefix caching wrong a replica, and vLLM enables it by default. The measured tally successful this portion confirms it moving connected a DigitalOcean GPU Droplet pinch nary configuration, nary constitute premium, and nary minimum prefix length, including connected a 155-token prompt, utilizing Llama 3.3 70B, which is 1 of the models without a managed serverless caching rate. The tradeoff is that you run the serving stack and salary for the GPU by the hr whether postulation arrives aliases not.

7. Do the deed rates measured present clasp erstwhile I standard past 1 replica?

Not automatically. These numbers travel from a azygous replica, wherever each petition sees the aforesaid cache. Under round-robin load balancing crossed N replicas, an identical prefix has astir a 1-in-N chance of landing connected the replica holding its cache entry. Session-affinity routing restores astir of the benefit, and DigitalOcean’s Advanced Prompt Caching astatine Scale station covers the multi-replica architectures successful detail.

8. How do I cheque whether my requests are really hitting the cache?

On DigitalOcean’s Serverless Inference, publication the usage entity connected each response: a nonzero cache_read_input_tokens confirms a hit, and open-source models besides study prompt_tokens_details.cached_tokens. On self-hosted vLLM, the per-request usage entity does not study cached tokens, but the /metrics endpoint exposes vllm:prefix_cache_queries_total and vllm:prefix_cache_hits_total, and diffing those counters astir serial requests attributes hits exactly, which is really the harness successful this portion measures them.

Conclusion

Prompt caching is simply a workload decision. The 3 variables that determine it, prefix stability, convention depth, and interarrival time, are each measurable, and this portion has fixed you the look to cheque the first 2 against immoderate provider’s published rates, a harness to measurement the 3rd against your ain existent traffic, and 1 completed tally of that harness: connected an H200 GPU Droplet serving Llama 3.3 70B, a unchangeable prefix earned a 98.7 percent deed complaint and a 92 percent TTFT reduction, and a azygous timestamp astatine the apical of the punctual erased astir each of it.

Caching addresses input token costs specifically. It has nary effect connected output token cost, which needs its ain controls and is covered successful a companion portion connected output pricing. If you way requests crossed aggregate models, convention pinning done DigitalOcean’s Inference Router is the system that keeps a cache lukewarm crossed a routed session, and the 2 systems should beryllium treated arsenic coupled, not separate, precisely arsenic this piece’s convention pinning conception describes.

Run the harness connected your ain postulation shape. Your measured deed rate, not immoderate vendor’s header percent and not moreover the numbers measured here, is the 1 that should determine whether caching pencils retired for you.

References

DigitalOcean archiving and engineering content

• Use Prompt Caching successful Chat Completions and Responses API
• Advanced Prompt Caching astatine Scale (DigitalOcean Engineering blog)
• Inference Features reference
• Inference Pricing
• Foundation Models (which models support punctual caching)
• Serverless Inference API reference
• How to Use Inference Router
• Multi-Model API Cost Governance pinch the Inference Router
• DigitalOcean Serverless Inference: A Deep Dive
• Fine-Tuned LLMs connected Serverless Architecture
• Where to Host Your Open-Source Model (Under 10B Parameters)
• What’s New connected DigitalOcean’s Inference Engine

Provider superior sources

• Anthropic: Prompt caching documentation
• Anthropic: Pricing documentation
• OpenAI: Prompt Caching successful the API (announcement)
• OpenAI: Prompt Caching 201 cookbook
• OpenAI: Prompt caching guide
• Google: Context caching, Gemini API documentation

Measured trial environment

• vLLM: Automatic Prefix Caching documentation
• vLLM: Production Metrics (/metrics endpoint)
• Model: RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic connected Hugging Face