The Four Hidden Surcharges in Your GPU Bill

Last period you booked a flight, the fare was $39. Then it was $18 for a spot that isn’t successful the middle, $35 for the bag, $12 for privilege boarding truthful the container really fits, and a “carrier-imposed surcharge” that cipher tin explain. The $39 fare costs you $120. You paid for everything the fare assumed you’d salary for separately.

Your GPU measure useful the aforesaid way. The per-hour complaint is the fare, and the logic your conclusion statement is two-thirds higher than the number connected the pricing page is because of 4 surcharges: egress, idle time, the noisy-neighbor tax, and cold-start latency. In this article, we value each 1 connected a azygous realistic workload, adhd them up, and locomotion done which are optimization problems and which are architecture problems.

The workload we’ll price: a real-time 70B conclusion API connected 4 H100s

We’ll usage 1 workload the full measurement through: an conclusion API serving an unfastened 70B-class exemplary connected 4 H100s, down a customer-facing characteristic wherever a quality is waiting connected the response. It’s a setup that is very communal these days, and is simply a very bully example, because it gets deed by each 4 surcharges astatine once.

H100 on-demand pricing has fallen — VentureBeat, citing Cast AI’s tracking, puts it down from astir $7.57 per GPU-hour successful September 2025 to astir $3.93 today. So 4 H100s moving astir the timepiece is astir 4 × $3.93 × 24 × 30, aliases astir $11,300 a month. That’s the “fare”. Everyone compares this number, and it’s almost ne'er the number that decides your bill.

A statement connected the comparison earlier we start: $3.93 is simply a commodity on-demand rate. Named hyperscalers often complaint much per H100, but they besides people the egress, NAT, and cross-AZ rates we usage below. To make the strongest type of the argument, we’re pairing the cheapest plausible compute fare pinch a realistic hyperscaler data-movement and resilience posture.

Surcharge 1: Egress, positive the NAT and cross-AZ taxes stacked connected top

Uploading information into a unreality is free, getting it backmost retired is not, and that is deliberate. As of mid-2026, first-tier net egress is astir $0.09/GB connected AWS, $0.087 connected Azure, and $0.12 connected GCP Premium: astir 4 to six times what the aforesaid supplier charges to shop a gigabyte for a month.

The header complaint is accurate. Two statement items softly stack connected apical of it:

• The NAT Gateway- If your conclusion pods beryllium successful a backstage subnet and scope the net done a managed NAT, the default unafraid shape connected AWS each gigabyte gets an other $0.045/GB data-processing interest connected apical of egress. That astir doubles your effective complaint earlier postulation leaves the building.
• Cross-AZ traffic- Spread replicas crossed readiness zones for resilience and AWS bills $0.01/GB successful each direction, landing nether “EC2-Other,” wherever cipher goes looking.

Now let’s put numbers connected our workload. Say it handles 30 cardinal requests a period pinch an mean consequence payload astir 250KB erstwhile you count the completion, the echoed context, and the logs you vessel out. That’s 30M × 250KB, aliases astir 7.5 TB of net egress:

• 7,500 GB × $0.09 ≈ $675
• NAT Gateway: 7,500 GB × $0.045 ≈ $338
• Cross-AZ chatter: presume replicas are divided crossed 2 readiness zones down a load balancer, truthful astir half the request/response measurement astir 4 TB, crosses a area boundary. At $0.01/GB successful each guidance that’s $0.02 round-trip, aliases ≈ $80

That’s astir $1,090 a period successful information movement, astir 10% connected apical of compute. For data-heavy workloads, transportation routinely lands successful the 10-15% range, and connected distributed multi-AZ architectures, higher still.

Surcharge 2: Idle clip - you’re paying for GPUs doing nothing

Cast AI’s 2026 State of Kubernetes Optimization Report measured accumulation telemetry crossed astir 23,000 clusters and recovered mean GPU utilization of 5%. Organizations are provisioning astir 20 times the GPU capacity their workloads usage astatine immoderate fixed moment. Co-founder Laurent Gil’s statement is the full problem successful 1 sentence: an idle CPU halfway costs cents per hour; an idle GPU costs dollars.

And the point you’re wasting is getting much expensive, not less. The aforesaid study notes AWS raised H200 Capacity Block prices astir 15% successful January 2026, the first clip since EC2 launched successful 2006 that a hyperscaler meaningfully raised GPU pricing alternatively than cutting it.

That 5% is the fleet average, and it includes a batch of forgotten dev clusters. The report’s best-tuned example: a 136-H200 cluster sustained 49%. So let’s beryllium generous and presume our real-time API runs astatine 30%: good supra the fleet average, still short of a well-managed cluster. That still intends 70% of the 4 GPUs we’re paying for beryllium idle. Of the ~$11,300 successful compute, only astir $3,400 is doing useful work, and astir $7,900 is the idle tax.

And you can’t conscionable “turn disconnected the idle ones.” Scaling a GPU replica isn’t for illustration scaling a web server. A stateless pod comes up successful a second; a GPU replica has to schedule onto a node, propulsion a multi-gigabyte image, initialize the runtime, and load exemplary weights into VRAM. Which, arsenic the adjacent conception shows, takes tens of seconds. You can’t standard reactively astatine the velocity postulation arrives, truthful to protect tail latency you support a buffer of lukewarm capacity up of demand.

Surcharge 3: The noisy-neighbor taxation that ne'er appears connected the bill

Noisy-neighbor taxation shows up arsenic your work needing much hardware.

In a multi-tenant GPU situation you stock beingness silicon, and a GPU has 2 things that are easy to starve: VRAM and representation bandwidth. Token procreation is memory-bandwidth-bound, you watercourse the full exemplary retired of VRAM for each token, truthful erstwhile a co-tenant saturates that bandwidth aliases expands its KV-cache footprint, your throughput drops and your tail latency spikes, and you didn’t touch your ain code.

This is measured, not hand-waved. The iGniter study co-located conclusion workloads connected a shared GPU utilizing NVIDIA’s MPS spatial sharing and watched mean conclusion latency climb from nether 1% to astir 35% arsenic co-tenants grew from 2 to five. The harm is concentrated precisely wherever it hurts: medians and throughput hardly move, while the p99 tail and jitter rustle up, worst for the small, latency-critical service. A dedicated H100, by contrast, gives you the afloat 3.35 TB/s of representation bandwidth pinch nary interference and a predictable p99.

Production SLAs unrecorded and dice connected p99, not the median. When you can’t spot aliases power the neighbour degrading your tail, you do the only point you can: over-provision. Add a GPU of headroom, tally astatine little utilization connected purpose, conscionable to sorb variance you don’t own. On our workload, 1 other H100 of protect headroom is astir $2,840 a month paid to protect against a workload that isn’t yours.

Surcharge 4: Cold starts move scale-to-zero into the idle tax

Scale-to-zero is the evident reply to the idle tax: driblet to zero GPUs erstwhile postulation stops, salary nothing. The drawback is the adjacent request, the acold start.

When a petition hits a acold endpoint, the level creates the container, initializes the ML runtime and CUDA context, fetches weights from entity storage, loads them into VRAM, and warms up CUDA graphs and the KV cache. For a ample exemplary that’s 30 to 90 seconds, dominated by fetching weights. One arXiv breakdown clocks a 130GB Llama-2-70B checkpoint astatine ~26 seconds conscionable to propulsion from retention complete a 5GB/s link, past ~84 seconds to load onto 8 GPUs, while the first token itself generates successful astir 100 milliseconds. It gets worse nether load: erstwhile respective acold containers onshore connected the aforesaid node, they conflict for web bandwidth, fetching weights, and starting to agelong further.

For an async batch job, a infinitesimal of warm-up is irrelevant. For a real-time API a quality is waiting on, a 45-second first consequence is simply a grounded request. The personification is gone.

To beryllium precise: acold starts aren’t what they were 2 years ago. Memory snapshotting captures and restores the full VRAM state, weight caching and prefetching overlap the fetch pinch instrumentality creation. RunPod and Modal person pushed best-case starts toward sub-second Modal cites a vLLM exemplary dropping from ~118s to ~12s, pinch champion cases successful the debased azygous digits. But the costs hides wrong the fix: the reliable measurement to guarantee nary acold commencement is to support a minimum excavation of workers warm. And a lukewarm worker is an idle GPU you’re paying for. Scale-to-zero traded the idle taxation for the cold-start tax, and the modular mitigation trades it correct back. Keeping a one-GPU lukewarm baseline connected our workload is different ~$2,840 a month.

The bill: ~$18,050 shared vs ~$12,800 dedicated

Same workload: 4 H100s, ~7 TB of egress, a real-time SLA.

On a hyperscaler, shared, pinch the protect posture you really request to enactment up:

• ~$11,300 compute (at $3.93/GPU-hr)
• ~$1,070 egress, NAT, and cross-AZ
• ~$2,840 for 1 GPU of noisy-neighbor headroom
• ~$2,840 for a one-GPU cold-start lukewarm baseline

That’s astir $18,050 a month, and only ~30% of the guidelines compute is doing useful work. Strip the 2 security GPUs and you’re still astatine ~$12,390. Now the portion that we thin to miss. Single-tenant dedicated capacity for the aforesaid 4 H100s isn’t cheaper per GPU. On DigitalOcean’s Dedicated Inference it’s $4.41 a GPU-hour against the hyperscaler’s $3.93- truthful 4 is astir $12,700 a period successful compute, astir $1,400 much connected the fare alone.

And the full still comes retired astir $5,000 lower, because dedicated, single-tenant capacity deletes the surcharges the inexpensive fare softly required:

• Egress collapses. Your frontend isn’t co-located, truthful responses still time off DigitalOcean but astatine DigitalOcean’s level $0.01/GB outbound rate, pinch nary NAT-processing taxation and nary cross-AZ tax. That aforesaid ~7.3 TB is astir $73, not $1,070.
• No headroom GPU, because there’s nary neighbour to take sides against.
• No warm-pool GPU, because the capacity is already provisioned and predictable.

Why the accustomed fixes: serverless and reserved commitments fail

When teams yet look astatine this, the first small heart is 1 of 2 reflexes, and some aren’t helpful:

Reflex one: “Go serverless, standard to zero, extremity paying for idle”. You conscionable moved the costs into cold-start latency, and connected a real-time endpoint. Add a lukewarm excavation to hole it and you’re paying for idle GPUs again nether a different name.

Reflex two: “Buy bigger reserved commitments to get the complaint down”. Reserved pricing lowers the fare. It does thing astir egress, thing astir the neighbor, and it makes the idle taxation worse: you’ve now committed, multi-year, to capacity you’ve measured moving astatine 5-30%.

Notice the pattern- Three of the 4 surcharges aren’t optimization problems you tune your measurement retired of, they’re architecture problems. You don’t FinOps your measurement retired of NAT-Gateway egress, you support postulation connected a backstage path. You don’t out-clever a noisy neighbor, you extremity sharing silicon. You don’t hit a acold commencement connected an endpoint that keeps scaling to zero, you tally connected capacity that’s already there.

How to really hole it: backstage traffic, dedicated silicon, capacity that’s already warm

The existent mobility is the aforesaid 1 that governs astir infrastructure decisions: build it yourself, aliases usage a furniture that already solved it.

Building it yourself is existent work. You’re opinionated up dedicated nodes, configuring a serving stack for illustration vLLM, owning the autoscaler and the warm-pool policy, wiring backstage networking truthful egress doesn’t way done a NAT, and re-tuning each of it each clip your postulation distribution shifts.

The managed option is DigitalOcean’s Dedicated Inference. It maps onto the aforesaid list:

• It’s managed LLM hosting connected single-tenant, dedicated GPUs, which takes the noisy-neighbor taxation disconnected the array because location are nary neighbors.
• It runs an opinionated accumulation stack underneath (vLLM pinch a prefix-cache-aware router) down an OpenAI-compatible API, truthful warm-pool and routing complexity is handled alternatively than billed backmost arsenic idle insurance.
• It exposes backstage VPC endpoints, truthful postulation betwixt your app and your exemplary stays disconnected the metered net path.
• It’s reserved per-GPU-hour pricing pinch a bring-your-own-model path, the predictable broadside of the idle-versus-cold-start trade, wherever you support your ain weights.

Two things worthy flagging. First, this is dedicated, single-tenant silicon, surcharge 3 is eliminated by definition: nary co-tenant saturating your representation bandwidth, nary KV-cache contention spiking p99, nary buying a full other GPU to sorb personification else’s burst. Second, the VRAM: 192GB connected a azygous AMD MI300X paper is capable to clasp a exemplary that would different request 3 aliases 4 80GB H100s sharded together, truthful you get the afloat representation bandwidth of 1 spot pinch nary inter-GPU split, its ain quiet triumph connected tail latency. Plans tally from a azygous GPU up to 8-GPU AMD aliases NVIDIA configs.

Under the hood it’s a Kubernetes-native stack DigitalOcean operates for you: the vLLM serving engine, ingress networking, exemplary storage, autoscaling, and prefix-aware routing. The router tracks KV-cache affinity, truthful a petition reusing a punctual prefix is sent to the replica that already holds those tensors alternatively of recomputing from scratch. That stack is precisely what, if you built it yourself, became your warm-pool problem, your cold-start problem, and your on-call pager. Here it’s operator-owned. You tin still group the node count, including scaling replicas to zero erstwhile you want to driblet idle time.

Once it’s active, you telephone it for illustration immoderate OpenAI-compatible endpoint: a POST to the chat-completions URL, a Bearer token, and a exemplary drawstring of the shape dedicated:<your-deployment-name>:<model-slug>. If your app sits successful the aforesaid VPC and you disable the nationalist endpoint, that telephone ne'er leaves the backstage network.

The comparison astatine a glance

There’s a lever beneath moreover that. If your exemplary fits, a azygous 192GB AMD MI300X astatine $2.59 a GPU-hour (about $1,865 a month) holds a exemplary that would different request 3 aliases 4 sharded 80GB H100s, giving you 1 card’s afloat representation bandwidth pinch nary inter-GPU split. That’s not the aforesaid workload arsenic 4 H100s, truthful benchmark your throughput earlier assuming it’s a drop-in, but for a batch of 70B-class serving, it’s the cheapest way connected this list.

When dedicated wins and erstwhile it doesn’t

Dedicated only wins if you support it busy. If your postulation is genuinely spiky pinch agelong dormant zones and nary real-time SLA, serverless pinch per-second billing tin beryllium the cheaper answer.

Measure earlier you migrate. Pull your existent data-transfer-out and NAT statement items. Measure existent GPU utilization: utilized cycles, not provisioned capacity. Check whether you’re azygous aliases multi-tenant today. And clip a acold commencement connected your ain endpoint pinch a stopwatch. You can’t value the 4 surcharges connected your circumstantial workload until you’ve logged them.

Egress and GPU rates move fast. Commodity H100s are still getting cheaper while frontier-tier H200 reserved pricing went up. A number that’s existent this 4th whitethorn not beryllium existent adjacent quarter. Re-run the mathematics astatine renewal.

The takeaway: the cheapest GPU-hour and the cheapest measure are not the aforesaid number

The fare is the 1 number everyone compares, and it’s the 1 that hides the different four. Egress and its NAT and cross-AZ riders, the idle tax, the noisy-neighbor tax, and the cold-start taxation routinely move a sub-$12K compute fare into an $18K measure and 3 of the 4 are architecture problems, not optimization problems. Keep postulation connected a backstage path, extremity sharing silicon, and tally connected capacity that’s already warm, and the surcharges don’t get optimized down. They extremity existing.

Want to spot the numbers connected your ain workload? Start pinch our LLM conclusion benchmarking methodology, past propulsion your transfer-out and utilization statement items and tally the four-surcharge mathematics against your existent setup aliases rotation up a Dedicated Inference deployment and comparison directly.