The Inference Cost Model Nobody Has Published: Token Economics Across Traffic Profiles on Dedicated GPUs

Introduction

LLM conclusion costs connected dedicated hardware is wished by 2 numbers: GPU hourly complaint and really galore tokens that GPU delivers per hr aft accounting for existent utilization. Published benchmarks measurement the 2nd number astatine highest saturation, erstwhile the GPU is afloat loaded. Operators salary the first number whether the GPU is saturated aliases not.

A DigitalOcean H200 GPU Droplet costs $3.44 per hr (current GPU Droplet pricing). That complaint runs whether the GPU is producing 2,036 tokens per 2nd astatine saturating batch sizes aliases sitting astatine 10% utilization astatine 3 AM. Effective costs per cardinal tokens depends connected your postulation profile, not connected hardware specs alone.

This article provides a replicable costs model for llama3.3-70b-instruct FP8 connected a DigitalOcean H200 GPU Droplet pinch vLLM. It derives the halfway formula, applies it to 3 postulation profiles, locates the crossover betwixt dedicated and serverless inference, and models really batch size, KV cache efficiency, and quantization impact effective costs per cardinal tokens. Every calculation is shown measurement by measurement truthful you tin use the look to your ain throughput and utilization.

Key Takeaways

• LLM conclusion costs connected a dedicated GPU scales pinch GPU utilization, not hardware specs alone. At 73% sustained mean utilization, a DigitalOcean H200 GPU Droplet moving llama3.3-70b-instruct FP8 pinch vLLM (2,036 tok/s saturated-batch throughput, measured) costs $0.643/1M tokens, astir break-even pinch DO Serverless astatine $0.65/1M. At 40% mean utilization connected identical hardware, effective costs rises to $1.173/1M, astir 80% much costly than serverless.

• The crossover utilization period betwixt an H200 GPU Droplet ($3.44/hr) and DO Serverless ($0.65/1M tokens for llama3.3-70b-instruct) is 72.2%. Below that sustained average, serverless is cheaper per token. Above it, dedicated is cheaper, and the separator grows pinch each further constituent of utilization.

• Traffic style determines costs result connected dedicated GPUs. A sustained-traffic workload averaging 73% utilization runs astatine $0.643/1M. A bursty workload connected identical hardware astatine the identical hourly rate, averaging 40% utilization, runs astatine $1.173/1M. The $0.53 costs dispersed betwixt these 2 profiles, astir 82% higher astatine 40% than astatine 73%, comes wholly from really postulation fills the GPU crossed 24 hours.

• Batch size is the serving configuration adaptable pinch the largest effect connected costs per token, and it is independent of hardware selection. At batch size 1, GPU SM utilization falls beneath 5%. At batch sizes 64 to 128 pinch vLLM continuous batching nether a 2-second P95 latency target, utilization reaches the scope wherever dedicated infrastructure is cost-competitive. A batch-size expanse tally for this article connected a azygous H200 shows a 44x cost-per-token quality betwixt batch-1 cognition ($20.32/1M) and saturated cognition ($0.46/1M), and astir 6x betwixt batch-8 ($2.73/1M) and saturation ($0.46/1M).

• FP8 quantization’s superior economical use for llama3.3-70b-instruct is hardware footprint. FP8 weights (~70 GB) fresh connected a azygous H100, but time off excessively small room for KV cache to service 70B astatine throughput; the applicable single-GPU level for saturated serving is the H200 (141 GB). FP16 weights (~140 GB) do not fresh connected 1 80 GB H100 astatine each and require 2 GPUs. FP8 halves the representation footprint and the GPU count for the aforesaid model.

The Core Cost Formula for Dedicated GPU Inference

Dedicated GPU conclusion has 1 costs equation.

effective_cpm = (hourly_rate ÷ (peak_tps × 3600 × utilization)) × 1,000,000

effective_cpm is costs per cardinal tokens delivered. hourly_rate is the GPU-hour price. peak_tps is the saturated token throughput of the exemplary connected that hardware. utilization is the fraction of the GPU’s capacity doing billable activity during the billing period. The only adaptable that changes pinch postulation is utilization.

Define Your Inputs: GPU Rate, Throughput, and Utilization

Three inputs provender the formula. The GPU complaint and serverless comparison value are documented; throughput and utilization are what operators measurement and control.

GPU rate. H200 GPU Droplet on-demand: $3.44/hr. H200 Dedicated Inference endpoint: $4.47/hr (managed serving stack included). See GPU Droplet pricing and Dedicated Inference pricing, aliases estimate a afloat configuration pinch the pricing calculator.

Peak throughput. For llama3.3-70b-instruct FP8 connected a azygous H200 pinch vLLM continuous batching, the benchmark described successful the methodology conception records 2,036 tok/s astatine a saturating batch. Saturating batch intends adding much concurrent requests produces nary further output throughput. The 2,036 tok/s is simply a saturating-batch figure; batch-1 throughput is astir 47 tok/s, an bid of magnitude lower. This article uses the measured 2,036 tok/s arsenic the guidelines case. The aforesaid benchmark connected a azygous H100 reaches only 236 to 335 tok/s because the 80 GB paper is KV-cache-bound, which the methodology conception explains.

Utilization. Utilization is the fraction of highest throughput the GPU sustains complete a billing period. A GPU astatine 40% is producing astir 814 tok/s connected mean while billing $3.44/hr for the remaining 1,222 tok/s worthy of idle capacity. Published accumulation cluster telemetry shows sustained utilization ranging from nether 5% connected un-optimized mixed fleets (per the Cast AI State of Kubernetes Optimization Report, 2026, based connected telemetry from astir 23,000 clusters) to 60-70% connected tuned dedicated conclusion stacks.

Measure your ain utilization straight alternatively than estimating it. The charismatic method is the aforesaid ratio this article’s look uses: utilization = actual_tokens_served_in_period / (peak_tps * seconds_in_period). The vLLM benchmark successful the methodology conception gives you peak_tps; your ain petition logs aliases vLLM’s built-in Prometheus metrics endpoint springiness you existent tokens served.

nvidia-smi tin supplement this but does not measurement the aforesaid thing. nvidia-smi --query-gpu=utilization.gpu reports SM activity, the percent of clip a kernel was executing, not the fraction of highest token throughput delivered:

nvidia-smi --query-gpu=utilization.gpu --format=csv -l 10

Because decode is memory-bandwidth-bound, this SM-activity reference tin tally high, 90% aliases more, moreover erstwhile the GPU is delivering a mini fraction of its highest token throughput. Treat nvidia-smi utilization arsenic a unsmooth upper-bound proxy, not a substitute for the tokens-served calculation above. For a billing-period mean utilizing either method, log complete a afloat diurnal rhythm (24 hours minimum, a afloat week if your postulation has play patterns) and return the time-weighted mean.

Measure Token Throughput astatine Target Utilization

Effective throughput astatine a fixed utilization is the merchandise of highest throughput and the utilization fraction:

effective_tps = peak_tps × utilization

At 73% utilization:

effective_tps = 2,036 × 0.73 = 1,486 tok/s

At 40% utilization:

effective_tps = 2,036 × 0.40 = 814 tok/s

Derive Cost per Token and Cost per Million Tokens

The worked illustration beneath uses each inputs defined above. Every multiplication and section is connected its ain line.

Inputs:

• GPU rate: $3.44/hr (H200 GPU Droplet; spot GPU Droplet pricing)
• Model: llama3.3-70b-instruct, FP8 quantization, vLLM continuous batching
• Throughput: 2,036 tok/s (saturated batch, not batch-1; measured connected a azygous H200, spot methodology)
• Target utilization: 73% (Profile 1, sustained-traffic workload)

Step 1, effective tokens per 2nd astatine target utilization:

effective_tps = peak_tps × utilization = 2,036 × 0.73 = 1,486 tok/s

Step 2, effective tokens per hour:

tokens_per_hour = peak_tps × seconds_per_hour × utilization = 2,036 × 3,600 × 0.73 = 7,329,600 × 0.73 = 5,350,608 tokens/hr

Step 3, costs per token:

cost_per_token = hourly_rate ÷ tokens_per_hour = $3.44 ÷ 5,350,608 = $0.000000643

Step 4, costs per cardinal tokens:

cost_per_million = cost_per_token × 1,000,000 = $0.000000643 × 1,000,000 = $0.643

At 73% sustained mean utilization, a azygous H200 GPU Droplet moving llama3.3-70b-instruct FP8 pinch vLLM delivers an effective costs of $0.643 per cardinal tokens (measured throughput). DO Serverless charges $0.65/1M for the aforesaid exemplary (Serverless conclusion pricing). Dedicated is roughly break-even astatine this utilization level, astir 1% cheaper, and only wins by a meaningful separator supra astir 85% sustained utilization.

Applying the afloat look straight arsenic a azygous expression:

effective_cpm = ($3.44 ÷ (2,036 × 3,600 × 0.73)) × 1,000,000 = ($3.44 ÷ 5,350,608) × 1,000,000 = $0.643

GPU Utilization Is the Variable Operators Control

GPU utilization is the superior costs adaptable successful dedicated conclusion and the 1 astir consistently absent from published benchmarks. A benchmark tally astatine saturating batch measures 1 operating constituent connected the costs curve: the minimum costs per token the hardware tin achieve.

Every conclusion workload separates into 2 compute phases. Prefill processes the afloat input punctual successful a azygous parallel pass; it is compute-bound and costs scales pinch punctual length. Decode generates output tokens sequentially, bounded by HBM representation bandwidth; decode is slower per token and dominates the GPU’s clip successful astir workloads. Both phases measure astatine the aforesaid hourly rate.

What Happens to Cost per Token Below the Crossover Threshold

Below 72.2% sustained mean utilization, effective costs per cardinal tokens connected a dedicated H200 GPU Droplet exceeds the DO Serverless complaint of $0.65/1M for llama3.3-70b-instruct. The crossover is the utilization astatine which some options costs the aforesaid per token.

Crossover calculation for GPU Droplet vs. DO Serverless:

crossover_U = (hourly_rate × 1,000,000) ÷ (peak_tps × 3,600 × serverless_rate) = ($3.44 × 1,000,000) ÷ (2,036 × 3,600 × $0.65) = $3,440,000 ÷ $4,764,240 = 0.722 = 72.2%

Below 72.2%, serverless is cheaper per token. Above it, dedicated is cheaper. The Dedicated Inference endpoint astatine $4.47/hr reaches parity astatine 93.8%:

crossover_U (Dedicated) = ($4.47 × 1,000,000) ÷ (2,036 × 3,600 × $0.65) = $4,470,000 ÷ $4,764,240 = 0.938 = 93.8%

The Dedicated Inference endpoint only wins connected costs erstwhile sustained utilization exceeds 94%. Below that, the $1.03/hr guidance premium erodes the per-token advantage comparative to serverless.

The Utilization-Latency Tradeoff astatine High Batch Sizes

Higher utilization requires larger concurrent batch sizes. Larger batches origin requests to hold longer for a decode slot earlier they statesman generating output, which increases per-request latency. At tight latency SLA targets, this queuing clip is the binding constraint that caps achievable utilization.

A deployment enforcing P95 TTFT beneath 500ms cannot capable ample batches: requests request slots to unfastened quickly. For llama3.3-70b-instruct FP8 connected H200, effective batch sizes astatine 500ms P95 are bounded astatine astir 8 to 16 concurrent sequences. A 2-second P95 target opens the scope to 64 to 128. Offline batch jobs pinch nary latency constraint push to 256 aliases more.

Interactive real-time applications are structurally constricted to little utilization operating points. A user-facing chatbot enforcing tight TTFT will run beneath the 72.2% crossover unless postulation measurement keeps moreover a mini batch continuously filled.

Reference Table: Cost per Million Tokens by Utilization Rate

The array beneath applies the costs look crossed the afloat operating range. All figures usage the H200 GPU Droplet astatine $3.44/hr, llama3.3-70b-instruct FP8, vLLM continuous batching, 2,036 tok/s measured saturated throughput. DO Serverless comparison rate: $0.65/1M tokens (Serverless conclusion pricing).

That quality comes wholly from utilization. The hardware and the hourly complaint are identical crossed each statement successful this table; only the postulation style changes.

Traffic Profile Changes Everything

Two operators tin tally the aforesaid GPU astatine the aforesaid hourly complaint and still extremity up pinch very different effective costs, based solely connected postulation shape. Utilization is what your petition volume, batch size, and diurnal rhythm nutrient astatine the GPU level; operators cannot group it directly.

Modeling a Sustained-Traffic Workload

Profile 1 models a workload pinch predictable request crossed business hours and a meaningful off-hours guidelines load: precocious during the day, mean astatine night, ne'er idle.

Profile 1 assumptions:

• Business hours (12 hr/day): 88% GPU utilization
• Off-hours (12 hr/day): 58% GPU utilization

Time-weighted mean utilization:

average_U = ((88% × 12) + (58% × 12)) ÷ 24 = (1,056 + 696) ÷ 24 = 1,752 ÷ 24 = 73%

Effective costs per cardinal tokens astatine 73%:

effective_cpm = ($3.44 ÷ (2,036 × 3,600 × 0.73)) × 1,000,000 = ($3.44 ÷ 5,350,608) × 1,000,000 = $0.643

At $0.643/1M versus $0.65/1M serverless, Profile 1 is astir break-even (modeled utilization, measured throughput), astir 1% cheaper. Business hours astatine 88% utilization tally beneath the crossover-implied costs and prevention money; the 58% off-hours guidelines sits beneath the 72.2% crossover and costs much than serverless during those hours.

Modeling a Bursty-Traffic Workload

Profile 2 models a SaaS backend pinch a beardown daytime highest and a near-zero overnight trough.

Profile 2 assumptions:

• Peak hours (8 hr/day): 80% GPU utilization
• Shoulder hours (8 hr/day): 30% GPU utilization
• Overnight trough (8 hr/day): 10% GPU utilization

Time-weighted mean utilization:

average_U = ((80% × 8) + (30% × 8) + (10% × 8)) ÷ 24 = (640 + 240 + 80) ÷ 24 = 960 ÷ 24 = 40%

Effective costs per cardinal tokens astatine 40%:

effective_cpm = ($3.44 ÷ (2,036 × 3,600 × 0.40)) × 1,000,000 = ($3.44 ÷ 2,931,840) × 1,000,000 = $1.173

At $1.173/1M versus $0.65/1M connected DO Serverless, Profile 2 costs about 80% more connected dedicated infrastructure (modeled). The overnight trough is the structural cause. During those 8 hours astatine 10% utilization, effective costs is:

cpm_trough = ($3.44 ÷ (2,036 × 3,600 × 0.10)) × 1,000,000 = ($3.44 ÷ 732,960) × 1,000,000 = $4.693

The overnight trough runs astatine $4.693/1M, 7.2x the DO Serverless rate. Eight hours of near-idle GPU resistance the blended regular mean from competitory to expensive.

Idle Capacity Cost: The Hidden Line Item successful Dedicated GPU Deployments

The idle taxation is the quality betwixt what you salary for a GPU during periods of debased utilization and what those aforesaid tokens would costs connected serverless. For Profile 2, the overnight trough idle taxation per night:

Tokens delivered during the 8-hour overnight trough:

tokens_trough = 2,036 × 0.10 × 3,600 × 8 = 203.6 × 28,800 = 5,863,680 tokens

Dedicated GPU costs during those 8 hours:

dedicated_cost = $3.44 × 8 = $27.52

What those aforesaid 5.86 cardinal tokens would costs connected DO Serverless:

serverless_cost = (5,863,680 ÷ 1,000,000) × $0.65 = 5.864 × $0.65 = $3.81

Nightly idle taxation and monthly total:

idle_tax_per_night = $27.52 - $3.81 = $23.71 monthly_idle_tax = $23.71 × 30 = $711.30

Eight hours of near-idle cognition each nighttime make $711.30 per period successful idle tax. A hybrid architecture that routes overnight trough postulation to DO Serverless while keeping the GPU for sustained business-hours request is the structural resolution. The Inference Router enables this without changing exertion code.

KV Cache Utilization and Effective Cost

KV cache ratio modifies effective GPU throughput without changing the hourly complaint aliases nominal utilization. When a petition shares a prefix already resident successful the cache, the serving motor skips recomputing those input tokens and uses the cached K and V tensors directly. The effect is balanced to a throughput multiplier proportional to the cache deed complaint and the fraction of overlapping input tokens.

Profile 3 models the Q&A postulation way from a multi-model routing deployment wherever astir 25% of full requests are multi-turn conversations sharing a strategy punctual prefix, arsenic described successful Multi-Model API Cost Governance pinch the Inference Router.

Profile 3: Q&A way astatine 25% nominal utilization, without convention pinning:

effective_cpm = ($3.44 ÷ (2,036 × 3,600 × 0.25)) × 1,000,000 = ($3.44 ÷ 1,832,400) × 1,000,000 = $1.877

At $1.877/1M, this way costs 189% much than DO Serverless. Session pinning routes consequent turns of each speech to the aforesaid serving replica, lifting cache deed complaint from astir 25% nether round-robin to 75% aliases more. Applying the measured 25% throughput summation from prefix-cache routing (llm-d, September 2025) gives an adjusted effective highest throughput:

adjusted_tps = 2,036 × 1.25 = 2,545 tok/s effective_cpm = ($3.44 ÷ (2,545 × 3,600 × 0.25)) × 1,000,000 = ($3.44 ÷ 2,290,500) × 1,000,000 = $1.502

Session pinning reduces the Q&A way costs from $1.877/1M to $1.502/1M, a 20% simplification astatine the aforesaid nominal utilization. The spread versus DO Serverless narrows from a 189% premium to astir 131%. Neither crosses the $0.65 serverless period astatine 25% nominal utilization.

The 25% throughput summation applied successful Profile 3 is simply a measured consequence from a multi-pod, multi-replica prefix-cache routing deployment utilizing 8 vLLM pods. That summation is cross-replica by nature: requests way to whichever pod holds the applicable cached prefix. On a azygous H100 pinch nary 2nd replica to way to, the applicable lever is vLLM engine-level Automatic Prefix Caching (APC), not cross-replica scheduling. Profile 3 is modeled by affinity from the multi-pod consequence and applies erstwhile the serving stack scales to aggregate replicas, arsenic successful a multi-model Inference Router deployment.

The array beneath shows really cache deed complaint affects effective costs per cardinal tokens astatine 40% mean GPU utilization. The 75% hit-rate statement uses the measured throughput gain. All different rows are estimated from that anchor. Assumptions: 50-60% input-to-output token ratio, strategy prompts astir 40% of input tokens.

At 87% cache deed complaint and 40% mean utilization, effective costs ($0.844/1M, modeled) remains supra DO Serverless astatine $0.65/1M. Cache ratio unsocial does not adjacent the gap.

For a deeper curen of KV cache mechanics, spot How KV Caching Slashes LLM Inference Costs astatine Scale. The engineering item down the prefix-cache hit-rate flip from 25% to 75% nether convention pinning is documented successful The Inference Tax: Prefix-Aware Routing.

Batch Size, Latency SLAs, and Their Effect connected Token Cost

Batch size is the astir straight controllable costs lever successful serving configuration, independent of hardware. The aforesaid GPU astatine the aforesaid hourly complaint produces dramatically different costs per cardinal tokens depending connected really galore concurrent requests it services per guardant pass.

At batch size 1, GPU SM utilization falls beneath 5%: the hardware is mostly idle betwixt weight loads, producing fewer output tokens per 2nd comparative to its representation bandwidth ceiling. The batch-size expanse successful this article (H200, vLLM 0.24.0, 1,024 input / 1,024 output tokens, $3.44/hr) quantifies this directly: batch-1 cognition costs $20.32 per cardinal output tokens, batch-8 costs $2.73, and saturated cognition costs $0.46. That is simply a 44x dispersed from batch-1 to saturation, and astir 6x from batch-8 ($2.73/1M) to saturation ($0.46/1M). The system is amortization: astatine precocious batch sizes, the costs of loading exemplary weights from HBM is dispersed crossed much concurrent decode sequences, producing proportionally much output tokens per portion of representation bandwidth. The afloat measured curve is successful the array below.

vLLM continuous batching fills unfastened decode slots pinch caller requests arsenic sequences complete, alternatively than waiting for the afloat batch to finish. At mean concurrency, continuous batching produces 3x to 5x much throughput than fixed batching connected identical hardware (per vLLM continuous batching documentation). max_num_seqs successful vLLM is the astir nonstop way from a poorly utilized GPU to a cost-competitive 1 without a hardware change.

The binding constraint is the latency SLA. Large batches summation queuing clip earlier each request’s decode begins, conflicting pinch tight TTFT targets. For llama3.3-70b-instruct FP8 connected H200, P95 TTFT beneath 500ms limits effective batch sizes to astir 8 to 16 concurrent sequences. A 2-second P95 target opens the scope to 64 to 128. Offline jobs pinch nary latency constraint push to 256 aliases more.

For astir API serving backends, the 64 to 128 scope astatine a 2-second P95 TTFT target is the applicable operating point, delivering 35x to 45x little costs per cardinal tokens than a batch-1 deployment. The expanse besides exposes a ceiling: throughput peaks adjacent concurrency 128 (2,140 tok/s) and does not amended astatine 256 (2,089 tok/s). Past that point, mean TTFT climbs from 4.4 seconds to 16 seconds and per-token latency astir doubles, while output throughput stays flat. Beyond astir 128 concurrent sequences astatine this discourse length, further concurrency buys latency, not throughput, the H200’s effective batching ceiling for llama3.3-70b-instruct FP8 astatine 4K context.

Prefill costs is worthy search separately connected high-context workloads. Prefill scales linearly pinch punctual length; RAG and archive workloads pinch agelong strategy prompts tin spot prefill correspond a meaningful fraction of per-request compute cost. Generation workloads pinch short inputs and agelong outputs are decode-dominated.

Self-Hosted Dedicated GPUs vs. API Consumption: The Crossover Calculation

The crossover betwixt a dedicated H200 GPU Droplet and DO Serverless is simply a utilization threshold. The token breakeven follows from that period erstwhile you hole throughput and regular tally time. At on-demand pricing:

Only person H100 access? A azygous H100 cannot scope the throughput this article uses arsenic its guidelines case; spot the sensitivity array beneath and the single-H100 KV cache ceiling successful the methodology conception for the measured logic why. The short version: alteration FP8 KV-cache quantization to free up headroom, aliases scheme for H200 aliases 2x H100 capacity if you request accumulation discourse lengths.

Monthly GPU Droplet cost:

monthly_cost = $3.44 × 730 = $2,511.20

Monthly token measurement astatine which $2,511.20 equals DO Serverless spend:

breakeven_volume = $2,511.20 ÷ ($0.65 ÷ 1,000,000) = $2,511.20 × 1,538,461 ≈ 3.86 cardinal tokens/month

Below 3.86 cardinal tokens per period astatine afloat on-demand pricing, DO Serverless is cheaper. Above that volume, dedicated is cheaper, and the separator grows pinch each further token because the GPU hourly complaint is fixed while serverless costs scales linearly.

Sensitivity to highest throughput assumption:

The measured guidelines lawsuit (2,036 tok/s) requires supra 72% sustained utilization for dedicated to break even. A higher 3,000 tok/s presumption breaks moreover astatine 49.0%, making dedicated competitory astatine astir half capacity, but the single-H100 measurement (236 to 335 tok/s) ne'er crosses: astatine that throughput dedicated cannot hit serverless astatine immoderate utilization.

Committed-use aliases reserved pricing lowers the hourly complaint and truthful the crossover proportionally; interaction DigitalOcean income for existent reserved H200 terms.

For the latency and capacity tradeoff framing earlier committing to dedicated GPU infrastructure, spot The LLM Inference Trilemma: Throughput, Latency, Cost.

Quantization and Model Optimization Effects connected Inference Cost

Quantization changes the hardware footprint required to service a model, which changes the hourly complaint astatine which conclusion is billed. For llama3.3-70b-instruct, the FP8 versus FP16 determination is simply a hardware action determination earlier it is simply a throughput decision.

FP8 hardware footprint. llama3.3-70b-instruct successful FP16 requires astir 140 GB of GPU VRAM, which does not fresh connected a azygous 80 GB H100 and requires 2 GPUs. In FP8 the exemplary requires astir 70 GB. It fits connected a azygous 80 GB H100, but aft weights and runtime overhead only a fewer GB stay for the KV cache, excessively small to service 70B astatine saturating batch (see the measured H100 consequence successful the methodology section). The applicable single-GPU serving level for 70B FP8 astatine throughput is the H200 (141 GB), which leaves astir 70 GB for KV cache.

At 40% utilization, the costs comparison beneath is simply a modeled hardware-footprint comparison astatine adjacent per-GPU throughput (2,036 tok/s), not the measured single-GPU anchor:

cpm_FP8_one_GPU = ($3.44 ÷ (2,036 × 3,600 × 0.40)) × 1,000,000 = $1.173 cpm_FP16_two_GPU = ($6.88 ÷ (4,072 × 3,600 × 0.40)) × 1,000,000 = ($6.88 ÷ 5,863,680) × 1,000,000 = $1.173

FP16 connected 2 GPUs costs $1.173/1M astatine adjacent per-GPU throughput (modeled), the aforesaid per-token costs arsenic FP8 connected 1 GPU astatine doubly the hardware count and doubly the hourly rate. This assumes adjacent per-GPU saturated throughput, which is conservative. FP8 is faster per GPU astatine debased batch sizes (1.4x to 1.6x connected Hopper-class hardware astatine batch size 1, per vLLM FP8 quantization documentation), truthful FP8’s existent costs advantage is larger than the equal-cost framing implies.

FP8 value costs is astir 0.3 to 0.5 points connected modular benchmarks, acceptable for astir accumulation workloads (per vLLM FP8 quantization documentation and third-party benchmark data). Native Hopper hardware support intends nary civilization kernel overhead.

INT4/AWQ. INT4 quantization drops VRAM by astir 75% and increases throughput beyond FP8. The tradeoff is astir 1.6 points of benchmark regression (per the AWQ quantization research, Lin et al., 2023). INT4 is due erstwhile fitting the exemplary connected less aliases smaller GPUs outweighs the value cost; deploy only aft verifying task-specific output value connected a typical information set.

Speculative decoding. A smaller draught exemplary proposes aggregate tokens per step; the target exemplary verifies them successful a azygous guardant pass. A accumulation benchmark connected code-heavy workloads (SWE-bench evaluation) measured astir 19.4% simplification successful costs per cardinal output tokens utilizing Eagle3-class speculative decoding pinch vLLM (Red Hat Developer, April 2026). Gains are largest astatine debased concurrency, wherever the decode shape leaves GPU SMs idle. At precocious concurrency (saturating batch), the draught exemplary competes pinch the target exemplary for GPU capacity and gains shrink toward zero.

Multi-GPU costs scaling. The costs look applies per node. The measured single-GPU anchor successful this article is the H200 astatine $3.44/hr and 2,036 tok/s. Adding a 2nd GPU doubles the hourly complaint and, astatine adjacent per-GPU throughput, doubles aggregate throughput arsenic well, leaving effective costs per cardinal tokens unchanged astatine immoderate fixed utilization rate. This is simply a modeled relationship; the aggregate throughput of a 2x H100 configuration was not measured here. The economical lawsuit for multi-GPU configurations is KV cache headroom and discourse length, not per-token costs reduction. At accumulation discourse lengths (8K+) wherever a azygous 80 GB H100 holds only a fistful of concurrent sequences, a 2x H100 configuration raises that ceiling while keeping costs per token competitory pinch the single-GPU figure, provided some GPUs are utilized consistently.

For exemplary optimization successful the discourse of mixture-of-experts architectures and their effect connected conclusion billing, spot MoE-ification and Your Inference Bill.

Methodology: What Is Measured vs. What Is Modeled

This conception states precisely what the numbers successful this article travel from and wherever the limits of those numbers are. Any costs fig branded “modeled” is simply a derived calculation utilizing verified pricing and a measured throughput figure, not a measurement from DigitalOcean accumulation infrastructure.

Measured inputs:

• H200 GPU Droplet complaint ($3.44/hr) and Dedicated Inference H200 complaint ($4.47/hr): from GPU Droplet pricing and Dedicated Inference pricing, existent arsenic of publication. The H100 GPU Droplet complaint ($3.39/hr) is referenced arsenic secondary grounds only wherever the single-H100 measurement is discussed.
• DO Serverless llama3.3-70b-instruct complaint ($0.65/1M tokens, input and output, symmetric): from Serverless conclusion pricing, existent arsenic of publication. The 50% batch discount does not use to DO-hosted open-source models.
• Saturated output throughput is measured, not modeled. The benchmark recorded 2,036 tok/s for llama3.3-70b-instruct FP8 connected a azygous H200 (141 GB) pinch vLLM 0.24.0, astatine 1,024 input / 1,024 output tokens, max_model_len 4096, max_num_seqs 256, gpu_memory_utilization 0.92, FP8 KV cache. The aforesaid benchmark connected a azygous H100 (80 GB, vLLM 0.23.0) reached 236 to 335 tok/s depending connected checkpoint, KV-cache-bound. Single-stream (batch 1) throughput was 46 to 47 tok/s connected the H200, a latency reference only.
• The batch-size costs curve is measured, not modeled. A concurrency expanse (1, 8, 64, 128, 256) ran connected the aforesaid azygous H200 pinch the aforesaid serving configuration (vLLM 0.24.0, 1,024/1,024 tokens, petition complaint saturated) connected 2026-07-04, producing 47, 351, 1,656, 2,140, and 2,089 tok/s respectively. The cost-per-million figures successful the batch-size array deduce straight from these measured throughputs astatine the $3.44/hr rate.
• KV cache prefix routing throughput summation (75% hit-rate row, +25-30%): measured consequence from a multi-pod deployment connected 8 vLLM pods. The baseline is an approximate-scheduling baseline, not round-robin. Against round-robin, the measured summation is astir 2x.

Modeled (not measured from DO infrastructure):

• The 40%, 73%, and 25% utilization rates successful the 3 postulation profiles. These are typical scenarios accordant pinch published accumulation cluster telemetry; they are not DigitalOcean customer telemetry.
• All effective costs per cardinal token figures derived from the look utilizing measured inputs above.
• Throughput summation percentages astatine cache deed rates beneath and supra the 75% anchor row, extrapolated from the measured anchor.
• Profile 3 adjusted throughput (2,545 tok/s): the 25% multi-pod summation applied by affinity to a single-path context. Requires aggregate serving replicas pinch a prefix-aware scheduler.

APC baseline disclosure. vLLM V1 enables Automatic Prefix Caching (APC) by default. The H200 benchmark that produced the measured 2,036 tok/s anchor ran pinch vLLM V1 default APC enabled, truthful the KV cache summation rows exemplary betterment supra an already-cached baseline; gains from a acold APC commencement whitethorn differ. Use the summation rows arsenic directional estimates, not guaranteed multipliers.

Single-H100 KV cache ceiling. The azygous H100 cannot clasp capable concurrent sequences successful KV cache to saturate astatine accumulation discourse lengths without FP8 KV-cache quantization, and the measured benchmark confirms it. For llama3.3-70b-instruct (80 layers, 8 GQA KV heads, 128-dimensional heads, FP8 astatine 1 byte per value):

kv_bytes_per_token = 2 × layers × kv_heads × head_dim × bytes_per_value = 2 × 80 × 8 × 128 × 1 = 163,840 bytes ≈ 160 KB per token

Available KV cache fund aft loading FP8 weights (~70 GB) onto an 80 GB H100:

kv_budget = 80 GB - 70 GB = 10 GB = 10,737,418,240 bytes max_kv_tokens = 10,737,418,240 ÷ 163,840 = 65,536 tokens

At 8K discourse per sequence:

max_concurrent_seqs = 65,536 ÷ 8,192 = 8 concurrent sequences

Reaching the measured 2,036 tok/s saturated throughput typically requires 30 to 50 concurrent sequences. A azygous H100 pinch FP8 weights holds only astir 8 concurrent sequences astatine 8K context. The benchmark confirms this ceiling, and astatine an moreover shorter discourse the spread is much pronounced than the nominal fund unsocial would suggest: a azygous 80 GB H100 held less than astir 10 resident sequences astatine 2K discourse (official checkpoint 335 tok/s, RedHatAI checkpoint 236 tok/s, a 42% spread driven purely by really overmuch VRAM each checkpoint leaves for KV cache). Measured KV headroom runs materially beneath the 10 GB nominal figure: weights, activations, CUDA chart buffers, and the gpu_memory_utilization 0.92 ceiling time off only a fewer GB for KV cache successful practice, which is why moreover 2K discourse holds less than 10 sequences alternatively than the astir 32 the nominal fund unsocial would imply. On the memory-rich H200 that checkpoint spread disappears (2,036 vs 2,041 tok/s), confirming the single-H100 consequence is simply a KV-headroom artifact, not a exemplary property. The 2,036 tok/s anchor is the measured H200 single-GPU figure; a azygous H100 cannot scope it astatine immoderate accumulation discourse length. Readers deploying astatine accumulation discourse lengths connected a azygous H100 should alteration FP8 KV-cache quantization aliases scheme for H200 aliases 2x H100 capacity.

Reproduce the measured throughput. Spin up a DigitalOcean H200 GPU Droplet and tally the aforesaid vLLM benchmark harness against llama3.3-70b-instruct FP8 to reproduce the measured 2,036 tok/s anchor; substitute your ain fig if you benchmark a different configuration. The commands beneath ran connected vLLM 0.24.0; verify against your installed version.

meta-llama/Llama-3.3-70B-Instruct is simply a gated exemplary connected Hugging Face. Request entree connected the exemplary page, past authenticate connected the Droplet earlier serving pinch export HF_TOKEN=your_token aliases huggingface-cli login. Without it, the weight download fails pinch an authentication error.

Start the serving endpoint:

vllm service meta-llama/Llama-3.3-70B-Instruct \ --quantization fp8 \ --kv-cache-dtype fp8 \ --max-model-len 4096 \ --max-num-seqs 256 \ --gpu-memory-utilization 0.92 \ --tensor-parallel-size 1 \ --port 8000

Output

INFO: Started server process [12345] INFO: Waiting for exertion startup. INFO: Application startup complete. INFO: Uvicorn moving connected http://0.0.0.0:8000 (Press CTRL+C to quit)

Then tally the saturated throughput benchmark against the moving server. The modern vLLM CLI ships this arsenic vllm chair service (vLLM 0.24.0 was utilized here). On versions that predate the vllm chair subcommand, tally python benchmarks/benchmark_serving.py pinch the aforesaid flags from a cloned vLLM root tree:

vllm chair service \ --backend vllm \ --model meta-llama/Llama-3.3-70B-Instruct \ --host localhost \ --port 8000 \ --num-prompts 1000 \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --request-rate inf

Output

============ Serving Benchmark Result ============ Successful requests: 1000 Benchmark long (s): 503.00 Total input tokens: 1024000 Total generated tokens: 1024000 Request throughput (req/s): 1.99 Output token throughput (tok/s): 2036.0 Total token throughput (tok/s): 4071.6 ---------------Time to First Token---------------- Mean TTFT (ms): 418.3 Median TTFT (ms): 371.2 P99 TTFT (ms): 1823.7

The Output token throughput (tok/s) statement is your peak_tps value. Replace 2,036 pinch your measured fig if you benchmark a different configuration, and re-run each look successful this article. For serving metrics to way utilization connected a moving vLLM stack, spot Metrics that Matter pinch Serverless Inference.

Frequently Asked Questions

What Is LLM Inference Cost and How Is It Calculated?

LLM conclusion costs connected dedicated hardware is the GPU compute clip consumed per generated token, billed astatine the GPU hourly rate. The look is: costs per token = GPU hourly complaint ÷ (tokens per 2nd × 3,600). For llama3.3-70b-instruct FP8 connected a DigitalOcean H200 GPU Droplet astatine $3.44/hr and 2,036 tok/s highest throughput (saturated batch, vLLM, measured), costs astatine 100% utilization is $3.44 ÷ 7,329,600 = $0.469/1M. At 73% utilization, costs rises to $0.643/1M. Utilization is the multiplier: each constituent of unutilized capacity raises effective costs per token proportionally.

How Much Does LLM Inference Cost per Million Tokens successful 2026?

DigitalOcean Serverless charges $0.65/1M tokens for llama3.3-70b-instruct (input and output; Serverless conclusion pricing). On a dedicated H200 GPU Droplet astatine $3.44/hr, effective costs depends connected utilization: astir $0.47/1M astatine 100%, astir $0.64/1M astatine 73%, astir $1.17/1M astatine 40%, and astir $1.88/1M astatine 25% (derived from the look successful this article pinch the measured 2,036 tok/s throughput). Dedicated infrastructure is cheaper than serverless only erstwhile sustained mean utilization exceeds 72.2%. Below that threshold, serverless is cheaper per token delivered.

Why Does GPU Utilization Have Such a Large Effect connected Inference Cost?

The GPU is simply a fixed-cost asset: the hourly complaint is identical astatine 10% load and 100% load. Idle capacity raises effective costs per token by spreading the fixed complaint crossed less tokens delivered. At 10% utilization connected a $3.44/hr H200, effective costs is $4.693/1M (modeled), 7.2x the DO Serverless rate. Utilization is the merchandise of postulation volume, batch size, and the GPU’s throughput ceiling; it is an result operators power indirectly, not a dial they set. Batch size and convention pinning are the operator’s levers for improving utilization connected existing hardware.

What Is the Difference Between Prefill Cost and Decode Cost?

Prefill processes the afloat input punctual successful a azygous parallel guardant pass. It is compute-bound: the GPU runs astatine precocious SM utilization during prefill, and longer prompts devour proportionally much compute. Decode generates output tokens 1 measurement astatine a time, bounded by really accelerated the motor tin load exemplary weights from HBM for each step. Decode is memory-bandwidth-bound. The costs implication: RAG and archive workloads pinch agelong strategy prompts transportation precocious prefill compute costs per request. Generation workloads pinch short inputs and agelong outputs are decode-dominated. Both phases measure astatine the aforesaid GPU hourly rate; the favoritism matters erstwhile sizing batch configuration and latency budgets for a fixed workload type.

At What Monthly Token Volume Does Self-Hosting connected a Dedicated GPU Become Cheaper than Using an API?

For llama3.3-70b-instruct connected a DigitalOcean H200 GPU Droplet versus DO Serverless astatine $0.65/1M tokens (Serverless conclusion pricing), the breakeven is astir 3.86 cardinal tokens per month, corresponding to astir 72% sustained utilization astatine the measured 2,036 tok/s. Monthly GPU cost: $3.44 × 730 = $2,511.20. Breakeven: $2,511.20 ÷ ($0.65 ÷ 1,000,000) ≈ 3.86 cardinal tokens. Below that volume, serverless is cheaper. Above it, dedicated saves money per token and the separator grows pinch each further cardinal tokens.

How Does Batch Size Affect Cost per Token?

Larger batches amortize the fixed costs of loading exemplary weights crossed much concurrent decode sequences, producing much output tokens per portion of representation bandwidth. At batch size 1, GPU SM utilization falls beneath 5%; costs tin beryllium 35x to 45x higher than astatine batch sizes 64 to 128 pinch continuous batching, accordant pinch the measured batch-size array earlier successful this article. The constraint is latency SLA: larger batches summation queuing clip per request. For llama3.3-70b-instruct FP8 connected H200, P95 TTFT beneath 500ms limits effective batch sizes to astir 8 to 16 sequences. A 2-second P95 target opens the scope to 64 to 128.

Does Quantization (FP8 aliases INT4) Meaningfully Reduce Inference Cost?

For llama3.3-70b-instruct, FP8 is the applicable prime because it determines hardware footprint: FP8 weights (~70 GB) fresh connected a azygous 80 GB H100, but time off excessively small KV-cache room to service 70B astatine throughput, truthful the applicable single-GPU serving level is the H200 (141 GB); FP16 weights (~140 GB) do not fresh connected 1 H100 and require 2 GPUs. At 40% utilization, FP8 connected 1 H200 costs $1.173/1M (measured throughput). A two-GPU FP16 deployment astatine adjacent per-GPU throughput costs the aforesaid $1.173/1M (modeled): aggregate throughput and the hourly complaint some double, leaving costs per token unchanged. The FP8 advantage is operational, not per-token: half the hardware count, half the hourly bill, and FP8’s latency separator astatine debased batch sizes (1.4x to 1.6x). FP8 value costs is astir 0.3 to 0.5 points (per vLLM FP8 benchmarks). INT4/AWQ reduces VRAM further and increases throughput but introduces astir 1.6 points of benchmark regression (per AWQ research); measure value per task earlier deploying.

How Do Bursty Traffic Patterns Affect the Economics of Dedicated GPU Inference?

Dedicated GPUs are provisioned for highest capacity and measure astatine the afloat hourly complaint during each off-peak period. Effective costs per token uses mean utilization crossed the afloat diurnal cycle, not highest utilization. Profile 2 successful this article models 80% peak, 30% shoulder, and 10% overnight, producing 40% time-weighted mean utilization and $1.173/1M effective costs (modeled), astir 80% supra DO Serverless. The overnight trough astatine 10% runs astatine $4.693/1M, astir 7.2 times the serverless rate, generating $23.71 per nighttime successful idle tax. Bursty workloads that cannot way off-peak postulation to serverless will consistently salary much connected dedicated infrastructure than they would connected per-request billing.

Conclusion

This article established a replicable costs model for dedicated GPU inference: effective costs per cardinal tokens equals GPU hourly complaint divided by the merchandise of highest throughput, seconds per hour, and sustained mean utilization. Applying that look to a DigitalOcean H200 GPU Droplet moving llama3.3-70b-instruct FP8 pinch vLLM gives a crossover period of 72.2% sustained mean utilization versus DO Serverless astatine $0.65/1M. A sustained-traffic workload averaging 73% is astir break-even, astir 1% cheaper per cardinal tokens. A bursty workload averaging 40% connected identical hardware costs astir 80% more. The 2 profiles tally connected the aforesaid GPU astatine the aforesaid hourly rate. The 82% costs dispersed comes wholly from really each fills the GPU’s capacity crossed the day.

With this framework, the determination betwixt dedicated and serverless conclusion is calculable earlier you provision. Measure your traffic’s time-weighted mean utilization crossed a afloat diurnal cycle, not conscionable highest hours. Apply it to the look pinch your throughput scope from the sensitivity table. If the effective CPM sits comfortably beneath $0.65 nether realistic utilization assumptions, dedicated infrastructure earns its rate. If your workload has a heavy overnight trough, a hybrid architecture that routes off-peak postulation to serverless while keeping dedicated capacity for sustained business-hours request recovers the idle tax. Estimate your configuration pinch the pricing calculator earlier you perpetrate capacity.

For the application-layer routing shape that splits postulation betwixt dedicated and serverless by task type and keeps KV-cache lukewarm crossed multi-turn sessions, spot Multi-Model API Cost Governance pinch the Inference Router. For the capacity and costs crossover betwixt serverless and dedicated infrastructure successful a broader framing, spot The LLM Inference Trilemma: Throughput, Latency, Cost.