Prompt Caching in Practice: From 7% to 74% Hit Rate(Inference in Production Series)

Prompt-caching mechanics positive a first-hand measurement connected DigitalOcean Serverless (Anthropic-style definitive cache_control). The mechanics use crossed providers; DigitalOcean numbers travel from a documented benchmark, not marketing.

Prompt caching is the highest-leverage costs lever astir accumulation LLM teams person configured but aren’t really benefiting from. A documented accumulation lawsuit from ProjectDiscovery shows some the nonaccomplishment mode and the fix: a security-intelligence supplier sent the aforesaid 4,000-token strategy instruction, instrumentality definitions, and study model connected each request, followed by the circumstantial threat information to measure — and still ran a 7% cache deed rate. One structural change, moving a azygous move identifier from the mediate of the punctual to the end, took the deed complaint to 74% and trim the monthly conclusion measure 59%.

The shape is common: caching is configured correctly successful principle, but 1 move section successful the incorrect position breaks the full prefix. This article covers really punctual caching really works, why astir implementations get it wrong, and the actual steps from single-digit deed rates to the 60–80% scope achievable successful production.

TL;DR

• Prompt caching is the azygous highest-leverage costs lever for agentic and multi-turn workloads — if the prefix is system correctly. One accumulation lawsuit (ProjectDiscovery) took cache deed complaint from 7% to 74% pinch a azygous reordering fix, cutting conclusion costs 59%.
• Three different caches lick 3 different problems: KV cache (automatic, wrong a azygous request), prefix cache (cross-request, the taxable of this piece), and semantic cache (application-layer, for semantically akin repetition queries, not nonstop matches).
• The cache is keyed connected exact, byte-identical prefix content. One move section — a timestamp, a convention ID, per-user information — placed before the cache bound invalidates the full prefix down it, not conscionable that field.
• Anthropic-style definitive caching breaks moreover aft astir 1 cache hit astatine modular rates (1.25x write, 0.10x read); each deed aft that is axenic savings. OpenAI’s cached-read discount varies by exemplary procreation — historically 50% off, moving toward 90% disconnected connected its newest models — truthful cheque the complaint for your circumstantial exemplary alternatively than assuming 1 number.
• Minimum cacheable prefix magnitude varies by exemplary and generation. Older Claude 3.x models cache prefixes arsenic short arsenic 1,024–2,048 tokens; newer models for illustration Claude Haiku 4.5 require 4,096 tokens. Don’t presume a azygous period applies crossed each exemplary version.
• Firsthand connected DigitalOcean Serverless (Claude Haiku 4.5, ~6,000-token prefix): moving 1 move section retired of the cached prefix took the deed complaint from 0% to 99.3% and trim input costs by astir 90%.
• Google’s Vertex AI squad doubled prefix cache deed complaint from 35% to 70% pinch cache-aware routing — cutting TTFT 35% connected context-heavy coding-agent workloads and P95 tail latency 52% connected bursty chat workloads (two chopped postulation profiles, not the aforesaid 1 measured twice).

KV, prefix, and semantic caching lick 3 different problems

“Caching” intends 3 different things successful the LLM conclusion context, and confusing them leads to misplaced optimization effort:

1. KV cache — automatic, wrong a azygous request. When a exemplary generates a response, it computes key-value attraction states for each input token. Without a KV cache, each decode measurement would recompute attraction complete each erstwhile tokens. The KV cache stores these states truthful decoding is incremental. It is ever on, requires nary configuration, and speeds up procreation wrong a azygous request. You don’t power it; you use from it automatically.

2. Prompt cache (prefix cache) — cross-request, controlled by you. Prefix caching extends the KV cache conception crossed aggregate independent requests. When requests stock the aforesaid starring tokens — the aforesaid strategy prompt, the aforesaid few-shot examples, the aforesaid reference documents — the exemplary reuses the computed KV states from the first petition alternatively of recomputing them connected each call. This is the cache the remainder of this article is about. The unchangeable starring tokens are the cacheable prefix; the per-request contented is the move tail.

3. Semantic cache — exertion layer, abstracted system. Semantic caching stores complete input-output pairs and retrieves them erstwhile a caller query is semantically akin to a erstwhile one, based connected embedding similarity. This is not a model-level characteristic — it’s a shape you instrumentality successful your exertion furniture utilizing a vector store. It is complementary to punctual caching, not a replacement.

You tin publication much connected Prompt Caching.

Three different caches astatine 3 different layers: KV (automatic, wrong a request), prefix (cross-request, controlled by punctual structure), and semantic (application-layer, for repetition queries).

Cache sounds costs 10% of input; the constitute premium pays backmost aft astir 1 hit

The economics are what make prefix caching the ascendant costs optimization.

On Anthropic (Claude), a cache constitute costs 1.25× the guidelines input rate (5-minute TTL) aliases 2× the guidelines input rate (1-hour TTL); a cache publication costs 0.10× the guidelines input complaint — a 90% discount. On Claude Sonnet 4.6 astatine $3.00/M input tokens, you salary $3.75/M to constitute a cache introduction for a 5-minute model and $0.30/M each clip you publication it. If a cached prefix is publication 10 times earlier it expires, you’ve paid $3.75 erstwhile and $0.30 × 10 = $3.00 for sounds — versus $3.00 × 11 = $33.00 without caching. That is an 80% simplification connected that prefix for a humble deed count.

OpenAI’s cached-input discount is not a fixed number crossed its catalogue — it varies by exemplary generation. Historically it sat astatine 50% disconnected for the GPT-4o family; connected OpenAI’s newest models the discount has moved up toward 90% off, the aforesaid 0.10x publication complaint Anthropic charges. Check the complaint for the circumstantial exemplary you’re calling alternatively than assuming 1 fig applies crossed each OpenAI model. DigitalOcean’s companion piece, When Prompt Caching Actually Pencils Out, has a afloat per-model complaint array for Anthropic, OpenAI, and open-source models connected DigitalOcean’s ain pricing page, and is worthy reference alongside this one.

The constitute costs is the adaptable to understand. Caching isn’t free — you salary a premium connected the first petition that populates the cache, and the mathematics only useful if consequent requests deed it often enough. The wide break-even constituent is h* = (w − 1) / (1 − r), wherever w is the constitute multiplier and r is the publication multiplier. At Anthropic’s 5-minute-tier rates (w = 1.25, r = 0.10), that gives h* = 0.25 / 0.90 ≈ 0.28, which rounds up to just 1 cache hit — aft that azygous hit, each consequent deed is axenic savings. You tin cheque this straight against the worked numbers above: astatine precisely 1 deed (2 full requests), caching costs $3.75 + $0.30 = $4.05, against $3.00 × 2 = $6.00 without caching — already cheaper. For a strategy punctual sent pinch each request, payback is efficaciously instant. For a ample archive referenced only a fewer times, cipher whether the publication savings warrant the constitute cost. For the afloat derivation and worked examples crossed each complaint tier DigitalOcean supports, spot the companion portion linked above.

When prefix caching helps — and erstwhile it doesn’t:

• Helps: a ample unchangeable prefix (system punctual + instrumentality definitions + few-shot + RAG template) reused crossed galore requests wrong the TTL window. The bigger and more-reused the prefix, the larger the win.
• Doesn’t: prefixes beneath the provider’s minimum cacheable length. On DigitalOcean serverless, prefixes nether ~4,000 tokens didn’t cache for america — accordant pinch Anthropic’s existent 4,096-token minimum for newer models for illustration Claude Haiku 4.5. That minimum isn’t cosmopolitan crossed each Claude generation, though: older Claude 3.x models cache prefixes arsenic short arsenic 1,024–2,048 tokens. Check the minimum for your circumstantial exemplary type — don’t presume 1 period applies everyplace — and spot the companion portion supra for a per-model reference table. Caching besides doesn’t thief connected low-reuse prefixes wherever the 1.25–2× constitute premium ne'er amortizes, aliases low-traffic endpoints whose petition inter-arrival clip exceeds the TTL.

A move section successful the prefix drops deed complaint to azygous digits

This is the correction made by the ProjectDiscovery squad above, and by a ample mostly of teams implementing caching for the first time.

The prefix cache requires bit-for-bit identical contented from the opening of the prompt. The cache is keyed connected the nonstop token series starting from the first token. Any alteration — immoderate character, immoderate section that varies betwixt requests — resets the cache bound astatine that point.

Take a template wherever the first portion is fixed (system instructions, instrumentality definitions) and location successful the mediate there’s a session_id, a timestamp, aliases a per-user configuration section that changes connected each request. Even though 90% of the punctual is identical, the cache only covers the tokens up to the first difference. Everything aft that constituent is recomputed from scratch connected each request.

Same content, 1 section moved. Stable layout: the move section sits astatine the tail, truthful the full prefix stays cacheable. Broken layout: the section sits mid-prefix, and everything aft it is recomputed connected each request.

That is why the information team’s deed complaint was 7%: a move threat identifier sat successful the mediate of an different unchangeable template, truthful the strategy prompt, study instructions, and instrumentality definitions were recomputed each time. Moving the identifier to the extremity — aft each the unchangeable contented — restored the afloat prefix for caching and took the deed complaint from 7% to 74% successful a azygous deployment.

Four steps from single-digit to 60–80% deed rate

Step 1: Identify your unchangeable prefix. Map your punctual building and categorize each component:

Your cacheable prefix is the longest starring series that stays identical crossed the mostly of requests — typically the strategy prompt, instrumentality definitions, and fixed few-shot examples, aliases 1,000–8,000 tokens of unchangeable content.

Step 2: Move each move contented to the end. Restructure truthful the unchangeable prefix comes first, uninterrupted, and each move contented follows astatine the end. Production prompts accumulate move fields organically — a convention ID added here, a personification penchant injected location — and each insertion silently kills caching. The target structure:

Concretely, successful the Anthropic-style cache_control API that DigitalOcean serverless exposes, the trap is simply a worth that changes each petition (a convention id, a timestamp) sitting wrong the marked prefix, and the hole is to move it retired of strategy into the personification turn:

Step 3: Monitor cache deed complaint arsenic a first-class metric. Cache deed complaint belongs connected your LLM operations dashboard alongside latency and cost. It’s a starring indicator: a driblet often precedes a costs spike, and it surfaces prompt-structure regressions earlier they go billing surprises. Most supplier APIs return cache usage successful consequence metadata — log it, alert connected it.

Step 4: Match cache TTL to your postulation interval. Anthropic’s TTL options (5 minutes aliases 1 hour) are operational constraints, not conscionable pricing tiers. A deployment pinch requests each 10 minutes will ne'er deed the 5-minute tier — take TTL based connected your expected petition interval astatine p50, not the burst peak. For bursty workloads (a regular batch that processes thousands of requests successful an hour, past goes quiet), the 1-hour TTL wins moreover astatine 2× constitute cost: the constitute is paid erstwhile per hour, and publication savings accumulate crossed the burst.

Cache-aware routing trim TTFT 35% and P95 latency 52% connected 2 different Vertex AI workloads

The costs savings from punctual caching are well-documented. The latency effect is little discussed and tin beryllium arsenic significant.

Google’s Vertex AI squad documented that intelligent routing — sending requests pinch shared prefixes to the aforesaid conclusion server that already has the prefix cached — doubled their prefix cache deed complaint from 35% to 70%. Google reports this trim TTFT by 35% connected context-heavy coding-agent workloads (Qwen3-Coder) and improved P95 tail latency by 52% connected bursty chat workloads (DeepSeek V3.1). Those are 2 chopped postulation profiles pinch different bottlenecks — context-heavy workloads are bound by re-computation overhead, bursty workloads by queue congestion — not the aforesaid workload measured 2 different ways.

The mechanism: cache hits destruct the prefill computation for the unchangeable information of the prompt. For a punctual wherever 4,000 of 4,500 tokens are cached, the exemplary runs prefill connected only 500 caller tokens — astir an 89% simplification successful prefill work. Prefill is compute-intensive; skipping it is simply a meaningful latency win, not conscionable a costs saving.

First-hand connected DigitalOcean Serverless: 0% to 99.3% deed rate, ~90% little input cost

To put numbers to this alternatively than quote a vendor figure, we benchmarked DigitalOcean Serverless Inference straight (Claude Haiku 4.5 astatine $1.00/M input, $5.00/M output, a ~6,000-token shared prefix, 1,500 requests per layout crossed 3 runs). Two findings worthy internalizing:

First, caching present is explicit, not automatic. DigitalOcean serverless uses the Anthropic-style model: you people the unchangeable prefix pinch an ephemeral cache_control breakpoint. Without that marker, the deed complaint is 0% nary matter really cleanly the punctual is system — and there’s a minimum cacheable magnitude (a ~2,300-token prefix didn’t cache; ~6,000 did, accordant pinch the 4,096-token minimum for this exemplary noted above). This is the 5-minute / 1-hour TTL system described earlier, not vLLM’s automatic prefix caching.

Second, pinch the prefix correctly marked, the layout effect is dramatic. Holding the exemplary and token count changeless and only moving the per-request move section from wrong the prefix to the tail, the measured cache deed complaint went from 0% to 99.3%, and estimated input costs dropped from $6.72 to $0.57 per 1,000 requests — astir a 90% reduction. That’s successful the scope the 90%-off cache-read economics would predict: erstwhile astir each input is served from cache astatine 0.1× the rate, the input measure falls by astir an bid of magnitude. (Our measured simplification runs a fewer points supra the simplest back-of-envelope estimate from deed complaint and discount alone; if you reproduce this benchmark, dainty the nonstop percent arsenic directional alternatively than a number to scheme a fund around, and measurement your ain postulation style instead.)

First-hand connected DigitalOcean serverless (Claude Haiku 4.5, ~6,000-token prefix, 1,500 requests per layout): marking the prefix pinch cache_control and moving the move section to the tail took the deed complaint from 0% to 99.3% and trim input costs astir 90% ($6.72 → $0.57 per 1,000 requests).

One honorable caveat from the data: connected shared serverless, the reproducible triumph is cost, not throughput. Across runs, TTFT and throughput fluctuated pinch queue conditions while the costs simplification held steady. Client-observed throughput gains from caching — the “more GPU cycles freed for decode” communicative show up connected a dedicated endpoint, wherever you ain the GPU. DigitalOcean’s Inference Optimized Image for GPU Droplets layers automatic prefix caching into a vLLM serving stack for precisely that case; measuring its throughput delta connected a dedicated Droplet is the earthy follow-up benchmark. For a afloat reproducible, engine-level benchmark harness against a self-hosted GPU Droplet (including earthy hit-rate and TTFT logs), spot the companion piece, When Prompt Caching Actually Pencils Out.

Semantic caching eliminates conclusion wholly for repetition queries

For workloads pinch precocious query repetition — customer support, FAQ bots, knowledge-base hunt — semantic caching complements prefix caching by eliminating afloat conclusion for communal queries. The pattern:

1. On each request, compute an embedding of the incoming query.
2. Check the vector shop for semantically akin erstwhile queries (typically ≥95% cosine similarity).
3. If a lucifer is found, return the cached consequence instantly — nary exemplary call.
4. If nary match, tally conclusion and shop the result.

The economics are compelling: a cache deed has near-zero marginal costs (one embedding telephone + 1 vector lookup). For a support bot wherever 40% of questions are variations of “how do I reset my password,” semantic caching tin destruct astir half of each conclusion calls.

When to usage it and what to manage:

• Staleness: cached responses request a TTL aliases invalidation mechanism. A consequence meticulous past period whitethorn beryllium incorrect aft a merchandise update.
• Threshold tuning: excessively precocious a similarity period yields fewer hits; excessively debased returns incorrect responses for different queries. 95% cosine similarity is simply a reasonable starting constituent — tune it against your query distribution.
• Security: successful multi-tenant deployments, 1 user’s query tin return different user’s cached consequence if queries are akin but contexts differ. Namespace the cache by tenant.

Pitfalls checklist earlier you ship

• [ ] No move fields (timestamps, convention IDs, personification IDs, per-request metadata) are embedded wrong the unchangeable prefix.
• [ ] Retrieved documents travel the unchangeable punctual template but look aft the fixed prefix.
• [ ] Cache deed complaint is logged per deployment and monitored utilizing alerting.
• [ ] Cache TTL is matched to your postulation interval — not group to 5 minutes for a workload pinch 10-minute inter-arrival times.
• [ ] For multi-tenant deployments, semantic cache entries are namespaced by tenant.
• [ ] Cache constitute costs are tracked separately — they partially offset publication savings.
• [ ] Your prefix meets the minimum cacheable magnitude for your circumstantial exemplary version, not a generic “1,024 tokens” presumption — newer models tin require more.
• [ ] Prompt building is version-controlled — accidental punctual changes invalidate the cache silently.

Done right, each 3 layers together service 60–80% of conclusion costs and latency from cache. Only prefix caching depends connected punctual structure, which is why it’s wherever astir of the wins — and astir of the mistakes — live.

Common Questions connected this topic?

1. What is punctual caching, and really is it different from a KV cache?

A KV cache is automatic and scoped to a azygous petition — it’s what lets a exemplary make tokens incrementally without recomputing attraction complete the full series from scratch astatine each decode step. Prompt (prefix) caching is the cross-request hold of that idea: erstwhile a later petition shares the aforesaid starring tokens arsenic an earlier one, the level reuses the already-computed KV authorities alternatively of reprocessing it. You don’t configure a KV cache; you do configure prefix caching, by structuring your punctual truthful the unchangeable contented comes first.

2. Why is my cache deed complaint debased moreover though I’ve enabled caching correctly?

The astir communal cause, by a wide margin, is simply a move section — a timestamp, convention ID, aliases per-request identifier — sitting location earlier the extremity of the cacheable prefix. The cache matches connected exact, byte-identical contented from the first token onward, truthful a azygous changed characteristic anyplace earlier the cache bound invalidates everything aft it. Move each per-request contented to the very extremity of the prompt, aft the marked cache boundary.

3. Does punctual caching alteration the model’s output?

No. Caching only affects really the input prefix is processed internally — whether the level reuses antecedently computed attraction states aliases recomputes them. The consequence is generated the aforesaid measurement either way, and providers authorities that output is unaffected by whether a cache deed occurred.

4. What’s the minimum punctual magnitude that tin really beryllium cached?

It depends connected the supplier and, connected Anthropic, the circumstantial exemplary procreation — location is nary azygous cosmopolitan number. Older Claude 3.x models cache prefixes arsenic short arsenic 1,024–2,048 tokens; newer models specified arsenic Claude Haiku 4.5 require astatine slightest 4,096 tokens. Below that floor, a punctual will not cache astatine all, nary matter really galore times it’s repeated aliases really a cache_control marker is placed. Check the existent minimum for your circumstantial exemplary earlier assuming a fig from a different exemplary aliases an older Claude procreation still applies.

5. How galore cache hits does it return earlier caching really saves money?

On Anthropic’s modular 5-minute-tier rates (1.25× write, 0.10× read), the break-even constituent is astir 1 cache deed — aft that azygous hit, each further deed is axenic savings. The wide look is h* = (w − 1) / (1 − r), wherever w and r are your provider’s constitute and publication multipliers comparative to modular input price; plug successful your ain provider’s rates to get your ain break-even point.

6. Is OpenAI’s cached-input discount the aforesaid arsenic Anthropic’s?

Not uniformly. Anthropic’s publication discount is simply a accordant 90% disconnected (0.10×) crossed the Claude family, pinch a constitute premium of 1.25× to 2× depending connected TTL. OpenAI’s cached-read discount varies by exemplary procreation — historically 50% disconnected connected the GPT-4o family, moving up toward 90% disconnected connected newer models. Check your circumstantial model’s existent complaint alternatively than assuming 1 number applies crossed OpenAI’s full catalogue.

7. Can punctual caching and semantic caching beryllium utilized together?

Yes, and they lick different problems, truthful they stack alternatively than compete. Prefix caching cuts the costs of reprocessing a shared punctual building connected each request. Semantic caching skips conclusion wholly erstwhile an incoming query is semantically akin capable to 1 already answered. A support bot, for example, tin usage prefix caching connected its strategy punctual and instrumentality definitions for each request, while besides utilizing semantic caching to short-circuit conclusion wholly connected the subset of queries that are near-duplicates of erstwhile ones.

8. Does moving a move section to the extremity of the punctual ever hole a debased deed rate?

It’s the azygous highest-leverage hole and resolves the mostly of cases, including the ProjectDiscovery illustration this portion opens with. It won’t help, though, if your prefix is beneath the provider’s minimum cacheable length, if your postulation is excessively sparse to nutrient repetition hits wrong the TTL window, aliases if the “stable” contented isn’t really identical crossed requests for immoderate different logic (a template that includes a subtly varying section you haven’t noticed, for instance). Reordering fixes a layout problem; it doesn’t hole a traffic-shape aliases prefix-length problem.

Conclusion

Prompt caching is simply a powerful instrumentality for reducing LLM costs and latency. It is simply a elemental method that tin beryllium implemented pinch minimal effort, but it tin person a important effect connected your bottommost line.

By pursuing the steps outlined successful this article, you tin instrumentality punctual caching and commencement seeing costs savings and latency improvements successful your LLM workloads.

You tin mention to the pursuing tutorials from this Inference successful Production bid to get started:

1. Why Your LLM Bill Is 3× What You Expected
2. How to Choose the Right LLM Model for Inference Use Case

Next Steps

• Implement punctual caching successful your LLM workloads.
• Monitor your cache deed complaint and set your TTL arsenic needed.
• Experiment pinch different punctual structures to spot what useful champion for your workload.

References

• How Does Prompt Caching Work and When Does It Actually Cut LLM Costs? — DigitalOcean (full break-even derivation, per-model complaint tables, and a reproducible engine-level benchmark harness connected a self-hosted GPU Droplet)
• How We Cut LLM Costs by 59% With Prompt Caching — ProjectDiscovery (production case)
• How GKE Inference Gateway improved latency for Vertex AI — Google Cloud
• Anthropic Prompt Caching documentation — Anthropic
• Prefix Caching — LLM Inference Handbook — BentoML
• Advanced Prompt Caching — DigitalOcean
• LLM Inference Optimization — DigitalOcean
• Serverless vs. Dedicated vs. Self-Hosted LLM Inference: When Self-Hosting Actually Gets Cheaper — DigitalOcean