Serverless vs. Dedicated vs. Self-Hosted LLM Inference: When Self-Hosting Actually Gets Cheaper

Every laminitis hears the aforesaid advice: rent an API to start, past self-host your ain GPU erstwhile you want to prevention money. It’s repeated truthful often it’s practically folklore. So alternatively of taking it connected faith, we tested it connected DigitalOcean. Same model, aforesaid prompts, 3 existent merchandise surfaces (DigitalOcean Serverless Inference, DigitalOcean Dedicated Inference, and a self-hosted GPU Droplet), existent bills.

The short version: a GPU you tally yourself does get cheaper per token than serverless, but only erstwhile you support it genuinely busy. The drawback is that early-stage postulation is spiky and bursty, truthful astir startups support a GPU only 5 to 10 percent busy, and down there, serverless is 2 to 4 times cheaper and zero-ops. So the existent mobility was ne'er “serverless vs. self-host.” It’s: really steadily engaged is your GPU really going to be? This station shows you precisely wherever the statement is.

This article runs the numbers for the 3 deployment paths astir startups really measurement time to day, each tested connected its DigitalOcean product: Serverless Inference, Dedicated Inference, and a self-hosted GPU Droplet. Real numbers, existent dollars, existent gotchas, from really moving the experiment.

TLDR

• Serverless Inference is cheaper than self-hosted GPU Droplet until the GPU is 22-48% busy.
• Dedicated Inference is 30% much costly than a self-hosted GPU Droplet, moving its break-even to ~29% work cycle.
• Default to Serverless Inference, and move to a GPU Droplet erstwhile it will really enactment busy.
• Don’t conjecture your conclusion bill, measurement it carefully.
• Buying aliases reserving a GPU “to prevention money” and moving it near-idle is 2-4x much costly than serverless, positive the ops load connected top.

Methodology

Every number present comes from our ain runs connected DigitalOcean, utilizing Qwen3-32B crossed 2 realistic workload shapes: a chat style (1,000 tokens successful / 500 out) and a RAG style (2,000 successful / 200 out). Latency was measured complete the nationalist net from a azygous client; the max-throughput numbers were measured connected the container itself truthful the web couldn’t headdress them.

Two honorable caveats up front. First: the self-hosted limb ran afloat bf16 precision connected a 192 GB MI300X, while the managed arms service their ain build, astir apt FP8. Precision is truthful not matched, and that’s a existent limitation of this comparison, not a deliberate creation choice. It cuts against self-hosting, not for it, since a matched FP8 self-host would apt beryllium cheaper still. Second: the numbers astatine the highest concurrency tiers (peak throughput, the ~40s tail latency) travel from azygous runs, not averages. Treat those tail figures arsenic approximate alternatively than a coagulated measurement you tin slope on.

All earthy information and scripts are published truthful you tin rerun this yourself, pricing referenced passim is arsenic of the clip this research was run. Check DigitalOcean’s existent rates earlier making a purchasing decision, since they tin change.

Three ways to tally the aforesaid model

Serverless (DigitalOcean Serverless Inference). You telephone an API and salary per token. The supplier owns the GPUs, the scaling, each of it. You negociate nothing.

Dedicated (DigitalOcean Dedicated Inference). You rent a full GPU’s worthy of capacity, kept lukewarm for you, arsenic a managed endpoint. You get isolation; the supplier still runs the box.

Self-hosted (DigitalOcean GPU Droplets). You rent a earthy GPU and tally the exemplary server yourself (or, astatine the extreme, you bargain the hardware). Most control, each of the responsibility.

Judge each 3 connected 4 things: cost, latency, really they scale, and really overmuch operational activity they put connected you.

The experiment

We fixed the exemplary (Qwen3-32B) and 2 realistic workloads: a chat style pinch tons of generated text, and a retrieval style pinch a ample archive stuffed successful for a short grounded answer. Same prompts, aforesaid streaming client, aforesaid metrics. The only point that changed was the deployment model.

• Arm A: DigitalOcean Serverless Inference
• Arm B: DigitalOcean Dedicated Inference (managed dedicated endpoint)
• Arm C: self-hosted connected a DigitalOcean GPU Droplet, renting a earthy GPU and opinionated up vLLM ourselves, some from a fresh image and afloat by hand

Provisioning: Serverless Inference serves successful seconds, Dedicated Inference takes ~25 minutes, a GPU Droplet ~4 minutes

The first quality shows up earlier a azygous personification arrives: really agelong until you tin serve, and erstwhile the metre starts running.

Serverless was unrecorded successful minutes. Grab an API key, paste it into your app, you’re live, and you salary only erstwhile tokens flow.

DigitalOcean Dedicated Inference took astir 25 minutes to proviso earlier it answered a azygous request, and you’re billed for that full warm-up. That’s normal for a ample model; managed endpoints crossed the manufacture typically return 5 to 30 minutes because the level has to propulsion and load tens of gigabytes of weights.

Renting a earthy GPU Droplet ourselves was really faster: astir 4 minutes to a serving endpoint. But past you ain everything: the firewall, the drivers, the container, restarts, teardown. We learned that the difficult measurement twice. A larboard was silently firewalled disconnected from our self-hosted endpoint (the 1-click vLLM image ships ufw pinch only 22/80/443 unfastened by default), and a “deleted” managed endpoint near an orphan GPU that kept billing until we hunted it down manually.

Latency: a self-hosted GPU is 2.5x faster to first token, until concurrency climbs past 32

For a azygous request, a GPU each to yourself is intelligibly faster. Measured the aforesaid measurement complete the nationalist internet, 1 watercourse connected our ain self-hosted MI300X GPU Droplet ran astir 50 tokens/second, pinch first token successful astir 0.6 seconds. Serverless Inference, sharing a pool, was slower per stream, astir 22 tokens/second, pinch first token person to 1.5 seconds. A GPU you power is astir 2.5x snappier to first token.

But that’s only while it’s lightly loaded. Push that aforesaid GPU harder and throughput climbs while latency falls apart.

At 32 parallel requests it still felt fine, astir 15 seconds extremity to end. By 100-plus concurrent requests, end-to-end crossed 30, past 40 seconds, and the slow tail sewage brutal. Strip the web retired wholly and measurement connected the container itself, and first-token latency drops to tens of milliseconds, but that’s the machine’s earthy ceiling, not what users really feel.

One MI300X peaks astatine astir 2,389 tokens/second astatine 192 concurrent requests, but end-to-end latency astatine that constituent blows past 40 seconds. Push to 256 concurrent and it over-saturates: throughput really drops.

Hold that thought, because it changes wherever self-hosting really pays off.

The crossover: self-hosting connected a GPU Droplet thumps Serverless Inference erstwhile the GPU is 22-48% busy

Here’s the number everyone really wants, and it’s easy to get wrong, truthful we’re computing it per answer, not per earthy token. A emblematic chat reply present is 1,000 tokens in, 500 out. On Serverless Inference, that costs astir $0.0005, and it ne'er changes pinch really engaged the GPU is.

A rented GPU is the opposite. You salary for it by the hour, whether it’s moving aliases not, truthful its value per reply is wholly astir utilization. Two things get mixed up present constantly:

• Duty cycle: what stock of the time the GPU is really working. This is what sets your cost.
• Concurrency: really galore requests you tally astatine once. This is the knob that trades latency for throughput.

Run the GPU level retired astatine precocious concurrency, and 1 MI300X handles almost 5 answers a 2nd astatine astir $0.0001 each, astir a 5th of serverless. But level retired intends 40-second responses.

So location are 2 break-even points, not one:

• If you saturate the GPU (accepting ~40s responses), you break moreover astatine only astir 22% work cycle.
• If you support it snappy (~15s end-to-end, the c32 operating point), you break moreover person to 48% work cycle.

Counterintuitively, saturating breaks moreover astatine a lower utilization than staying snappy, because higher concurrency squeezes much tokens retired of each moving hour.

Either way, the connection is the same: erstwhile a azygous GPU is steadily engaged (somewhere betwixt a 4th and half of the day), it thumps serverless connected price. Well supra that, it’s respective times cheaper. And keeping it engaged is not the aforesaid arsenic maxing it out; you tin beryllium astatine the snappy operating constituent and still triumph connected price, arsenic agelong arsenic the GPU isn’t idle.

The trap runs the different direction. At 10% work cycle (where a batch of early-stage GPUs really sit), self-hosting is more than 2x the costs of serverless. At 5%, it’s complete 4x. And that idle GPU still needs personification to babysit it.

These are 2 different operating points, not 1 curve. Pick the statement for the latency you’re really consenting to accept, don’t operation them.

Dedicated Inference costs ~30% much than a GPU Droplet, moving its break-even to ~29% work cycle

Dedicated Inference and a self-hosted GPU Droplet tally the nonstop aforesaid silicon, truthful they execute identically. Dedicated Inference conscionable costs astir 30% much per hour, which moves its break-even from astir 22% up to astir 29% work cycle. Still a debased bar, but that premium is purely for not having to negociate the container yourself.

As for buying hardware outright: modeled honestly (capital dispersed complete 3 years, positive power), an owned accelerator is cheaper still, a mini fraction of serverless erstwhile you support it busy. But 2 large asterisks. First, you can’t really bargain a azygous MI300X; they vessel successful eight-GPU servers, truthful this is an schematic model, not a existent value you tin put connected a card. Second, it only pays disconnected if you prolong that load for years, person the superior up front, and (the portion cipher costs) person personification connected telephone for it. For almost each startup, that personification is excessively expensive.

One pricing statement underlying each of this: the self-host figures supra usage on-demand rates. A reserved GPU Droplet connected a longer committedness runs cheaper still (roughly $1.49 per GPU-hour connected a 12-month MI300X term, versus $1.99 on-demand), which pushes each break-even little again. It only helps if you tin perpetrate to the term, truthful it rewards precisely the dependable base-load lawsuit wherever self-hosting already wins.

Always corroborate the rates pinch DigitalOcean’s existent rates earlier making a purchasing decision, since they tin change.

When a startup should move disconnected serverless: default to Serverless Inference, move for power aliases dependable base-load

Default to Serverless Inference. Pre-product-market-fit, pinch spiky and unpredictable traffic, it’s some the cheapest action and the slightest work. You salary fundamentally thing erstwhile idle, and it bursts for you automatically.

Move to Dedicated Inference aliases a self-hosted GPU Droplet erstwhile you request thing serverless can’t waste you: tight, predictable latency; isolation and guaranteed capacity; aliases information that has to enactment successful your ain boundary. Notice this is simply a power decision, not a savings decision.

Chase the costs crossover erstwhile you person a steady, predictable base-load that keeps 1 GPU astatine slightest a 4th to half engaged astir the clock. That’s erstwhile it genuinely undercuts serverless. Self-host earlier you buy: renting is the aforesaid velocity for astir 30% little and nary superior outlay.

Put a bladed gateway successful beforehand of everything, truthful erstwhile you move a basking way to beryllium dedicated later, it’s a config change, not a rewrite.

Four traps we deed firsthand moving the experiment

Buying aliases reserving a GPU “to prevention money” and moving it near-idle. At the 5-10% utilization astir early GPUs really see, that’s 2-4x much costly than serverless, positive the ops load connected top.

Forgetting the warm-up and idle bills. Dedicated Inference charged america for 25 minutes earlier it served anything. An orphaned GPU kept billing aft we thought we’d unopen it down: the endpoint vanished from the API, but its backing MI300X droplet stayed progressive and kept accruing charges until we deleted it manually.

Measuring throughput wrong. Our first test, tally complete the net from a laptop, under-reported the GPU’s existent throughput by astir 3x. The customer and WAN were the bottleneck, not the GPU. We had to re-run the load generator connected the droplet itself (localhost) to find the existent ceiling.

Comparing costs connected the incorrect basis. This 1 is easy to autumn into. Serverless hands you a value expanse pinch input and output tokens billed astatine abstracted rates, while a rented GPU conscionable hands you an hourly measure and nary token accounting astatine all, truthful you person to build the per-token number yourself. Early connected we built it from output tokens only, past compared that against a serverless value that billed some input and output. The denominators didn’t match, which loaded the GPU’s afloat hourly costs onto a fraction of the tokens it really processed and made self-hosting look much costly than it is. Fixed by pricing 1 full reply extremity to extremity (all input positive output tokens) nether each option, truthful some sides count the aforesaid thing.

The fix, each time, was the same: statement the numbers up honestly. If you benchmark, saturate from adjacent to the box, and comparison costs per answer, not costs per earthy output token.

The takeaway

Optimize your model choice and your utilization earlier you touch the deployment model. Default to DigitalOcean Serverless Inference, and move to a GPU Droplet erstwhile it will really enactment busy. And don’t conjecture your conclusion bill, measurement it carefully.

Further reading

If you want the broader conceptual model for choosing an conclusion mode earlier diving into numbers, DigitalOcean’s inference mode comparison guide covers serverless, dedicated, batch, and conclusion router astatine a precocious level. And for a deeper look astatine the throughput/latency/cost hostility that shapes each of this, The LLM Inference Trilemma is worthy a read.