If you tally LLM inference successful production, you person much hardware options than before. NVIDIA and AMD GPUs are still the default, but they are not the only choice. AWS sells Inferentia2. Google rents TPUs. Groq built a spot conscionable for inference. Tenstorrent is moving connected another. Each specialized spot is amended than a GPU astatine 1 circumstantial thing.
Most articles dainty this arsenic a velocity contest: who generates the astir tokens per second. If you really tally a serving stack, that misses the point. The fastest spot connected a benchmark tin still beryllium the incorrect prime if your p99 latency swings wildly, you vessel a caller exemplary each week, aliases your postulation does not lucifer what the spot was designed for.
This article maps the scenery for teams choosing dedicated conclusion hardware. For each chip, it covers what it does well, what you springiness up to get that, and erstwhile a specialized spot really thumps a GPU. The short reply up front: for astir teams, GPU serving wins because it is flexible. DigitalOcean runs conclusion connected GPU Droplets pinch NVIDIA H100 and H200 and AMD MI300X and MI325X, truthful this position comes from moving these workloads, not from reference spec sheets alone.
TL;DR
- Four spot families, 4 tradeoffs. GPUs waste and acquisition highest ratio for flexibility. Fixed-function accelerators (Inferentia2, TPU) waste and acquisition elasticity for little costs per token connected 1 fixed model. Groq’s LPU trades capacity for predictable latency. Tenstorrent is an earlier-stage dataflow option.
- Most teams request GPU flexibility. Swap models, alteration precision, and set batching without recompiling. Specialized chips make you salary again for each of those changes.
- Groq wins connected predictable latency. Its fixed schedule keeps p50 and p99 adjacent together, which is important for real-time, single-request workloads.
- Inferentia2 and TPU triumph connected costs per token for 1 fixed exemplary astatine precocious volume. The catch: you recompile connected each model, precision, aliases style change.
- Memory, not FLOPS, sets the serving ceiling. Capacity limits exemplary size. Bandwidth limits speed. Two commonly misquoted numbers are corrected below.
Table of Terms utilized successful this article
If you are caller to conclusion hardware, you tin mention to the beneath tables first. It defines the position utilized successful the article.
Core conclusion concepts
| LLM inference | Running a trained connection exemplary to reply unrecorded personification requests. Training updates the model; conclusion only uses it. Example: a chatbot replying to a customer message. |
| Dedicated inference | You rent aliases ain backstage chips (GPUs aliases specialized accelerators) for your app alone. You salary for GPU clip aliases hardware, not per token connected a shared API. |
| Serving stack | Everything betwixt a personification punctual and the streamed answer: load balancer, exemplary server (e.g. vLLM), GPU/chip, networking, and monitoring. |
| Token | A chunk of matter the exemplary sounds aliases writes, often a word, portion of a word, aliases punctuation. Billing and velocity quotes (tokens/sec) are based connected tokens. |
| Throughput | Total output produced per second. If your server generates 500 tokens/sec crossed each users, that is your throughput. Higher throughput intends much answers per 2nd astatine the aforesaid hardware cost. |
| Cost per token | Dollars spent ÷ tokens generated. A spot that costs little per token is cheaper astatine precocious volume, but only if your exemplary and postulation lucifer what that spot was built for. |
| Batch size | How galore requests the server processes together successful 1 GPU pass. Batch size 8 intends 8 prompts tally done the spot astatine once. Larger batches usage the spot much efficiently but tin make individual users hold longer successful queue. |
| Single-stream serving | Serving 1 petition astatine a clip pinch strict velocity targets. Example: a sound adjunct that must respond successful nether 300 ms, pinch nary batching hold allowed. |
| Batch inference | Running galore prompts offline pinch nary real-time personification waiting. Example: summarizing 100,000 support tickets overnight; you optimize for full cost, not per-request speed. |
Latency and reliability
| Latency | Wall-clock clip from sending a punctual to receiving the afloat (or first) answer. If a personification waits 800 ms, that is the latency they feel. |
| p50 (median latency) | Sort each requests by speed; p50 is the mediate value. If p50 = 200 ms, half of requests vanished faster than 200 sclerosis and half slower. It describes the typical request, not the worst one. |
| p99 latency | Sort each requests by speed; p99 is the clip that only the slowest 1% exceed. If p99 = 2 seconds while p50 = 200 ms, astir users are good but 1 successful 100 gets a noticeably slow response, that is what breaks chat UX and SLAs. |
| p999 latency | Same thought arsenic p99, but for the slowest 0.1% (1 successful 1,000). Used erstwhile moreover uncommon slow responses are unacceptable (e.g. trading aliases safety-critical systems). |
| Tail latency | The slow extremity of the distribution, the outlier requests, not the average. A GPU tin person p50 = 150 sclerosis and p99 = 1.5 s; the tail is that gap. |
| SLA (Service Level Agreement) | A statement aliases soul target connected capacity aliases uptime. Example: “p99 latency must enactment nether 500 ms” aliases “99.9% uptime.” Miss the SLA and you breach customer commitments aliases soul alerts fire. |
| Deterministic latency | Same input style → astir the aforesaid hold each time, because the chip’s schedule was fixed earlier runtime. Groq targets this: p50 and p99 beryllium adjacent together. |
| Statistical latency | Delay varies petition to petition because the spot decides scheduling astatine runtime and shares resources crossed concurrent users. Typical connected GPUs nether mixed load: p50 looks fine, p99 tin spike. |
Chip families and products
| GPU (Graphics Processing Unit) | A elastic accelerator that runs galore exemplary types and settings. You tin alteration models, precision (FP16, INT8), and batching without recompiling. In this article: NVIDIA H100/H200 and AMD MI300X/MI325X. Think of it arsenic a general-purpose kitchen, you tin navigator galore dishes without rebuilding the stove. |
| General-purpose GPU | Same arsenic GPU successful this article, the baseline everything other is compared against. |
| Inferentia2 | AWS’s spot built only for inference. Best erstwhile 1 exemplary runs unchanged astatine immense volume: debased costs per token. You must compile the exemplary pinch AWS Neuron earlier serving, and recompile erstwhile the exemplary aliases input sizes change. |
| TPU (Tensor Processing Unit) | Google’s civilization AI chip, rented connected Google Cloud. TPU v5e successful serving mode is akin to Inferentia2: compile erstwhile per model/shape, service cheaply astatine scale, little elastic than a GPU. |
| Groq LPU (Language Processing Unit) | Groq’s inference-only chip. Weights beryllium successful very accelerated on-chip memory; each cognition is scheduled up of time. Result: very predictable latency, but mini representation per spot truthful ample models request galore chips. |
| Tenstorrent | A vendor building dataflow-style AI chips. Covered lightly present because accumulation serving specs and package are little mature than GPU, Inferentia2, TPU, aliases Groq. |
| Fixed-function accelerator | Hardware optimized for 1 constrictive job, usually 1 compiled exemplary astatine circumstantial input sizes. Like a single-purpose appliance: very businesslike for that 1 job, awkward erstwhile requirements change. Inferentia2 and TPU v5e are examples. |
| Dataflow chip | Data moves done a fixed concatenation of specialized units (not a wide programme for illustration connected a GPU). Tenstorrent uses this approach. |
| H100 / H200 | NVIDIA data-center GPUs communal for LLM serving. H200 has the aforesaid compute arsenic H100 but much and faster representation (141 GB vs 80 GB), which unsocial makes it faster astatine conclusion for ample models. |
| MI300X / MI325X | AMD’s data-center GPUs, GPU replacement to NVIDIA successful this comparison. MI325X adds much representation (256 GB HBM3E) than MI300X (192 GB). |
| Hopper | NVIDIA’s architecture procreation for H100 and H200. “Hopper GPU” = H100/H200 class. |
| Blackwell | NVIDIA’s adjacent architecture aft Hopper (newer GPUs successful MLPerf results cited later). |
| Tensor Streaming Processor (TSP) | Groq’s soul sanction for the spot creation wrong the LPU, the hardware described successful Groq’s ISCA papers. |
Compilation, software, and flexibility
| Compile measurement / compilation | Translating your exemplary (PyTorch/ONNX, etc.) into a programme the spot tin tally natively. On Inferentia2 you usage Neuron; connected TPU you usage XLA. Takes minutes to hours depending connected exemplary size. You repetition it erstwhile the model, precision, aliases input dimensions change, not a one-time setup. |
| Neuron / AWS Neuron SDK | AWS’s toolchain to compile and tally models connected Inferentia chips. If Neuron does not support an cognition successful your model, compilation fails aliases runs a slow fallback. |
| XLA | Google’s compiler that turns models into TPU-ready programs (part of the OpenXLA project). Same compile-once-per-shape constraint arsenic Neuron. |
| Hot-swap | Load caller exemplary weights and commencement serving without recompiling. On a GPU: constituent the server astatine a caller checkpoint and restart, aliases switch successful seconds. On Inferentia2/TPU: usually requires a afloat recompile. |
| Continuous batching | As caller requests arrive, the server adds them to the existent GPU batch and removes vanished ones, batch size changes each millisecond. Keeps the GPU engaged nether adaptable traffic. vLLM and akin servers do this connected GPUs. |
| Static schedule | Groq’s compiler decides precisely erstwhile each cognition runs, including postulation betwixt chips, earlier immoderate petition arrives. Predictable timing, but the schedule cannot easy accommodate erstwhile batch size swings from 1 to 64. |
| Operator | One measurement successful the exemplary graph, e.g. “multi-head attention,” “matrix multiply,” “SiLU activation.” The compiler must instrumentality each usability your exemplary uses astatine afloat speed. |
| Operator coverage | The database of operators a compiler/chip supports natively. Missing usability = compile nonaccomplishment aliases a slow CPU fallback that wipes retired the chip’s velocity advantage. Always cheque sum against your model, not the vendor’s demo model. |
Memory, math, and why procreation feels “slow”
| FLOPS / FLOPs | How galore floating-point mathematics operations the spot tin do per 2nd astatine highest (spec-sheet number). High FLOPS sounds impressive, but token procreation often waits connected representation sounds alternatively of math, truthful FLOPS unsocial misleads. |
| Memory bandwidth | How accelerated the spot sounds information from memory, successful GB/s aliases TB/s. Example: H100 ~3.35 TB/s. Token procreation re-reads the afloat exemplary weights for every caller token, truthful bandwidth often caps velocity earlier FLOPS does. |
| Memory capacity | How overmuch information fits connected the spot astatine erstwhile (e.g. H100 80 GB, Groq 220 MiB per chip). If weights do not fit, you divided the exemplary crossed chips aliases quantize to smaller precision. |
| HBM (High Bandwidth Memory) | High-capacity representation stacked connected the GPU package (H100: 80 GB; H200: 141 GB). Much larger than Groq’s on-chip SRAM, but still slower to entree than SRAM. |
| SRAM | Small, highly accelerated representation connected the spot dice itself. Groq stores weights successful SRAM (220 MiB per chip), very accelerated reads, but a 70B exemplary cannot fresh connected 1 chip. |
| KV cache | During generation, the exemplary stores past attraction keys and values truthful it does not recompute the full punctual each token. Grows pinch speech length. More discourse = much KV cache representation sounds = slower and much costly per token. |
| Prefill / punctual processing | Phase 1: the exemplary sounds the full personification punctual successful 1 (or few) parallel passes. Math-heavy, the chip’s compute units enactment busy. Happens erstwhile per petition earlier immoderate output token appears. |
| Decode / token generation | Phase 2: the exemplary outputs 1 token astatine a time. Each measurement re-reads each exemplary weights from memory. Memory-heavy, often why procreation feels slower than “thinking about” the prompt. |
| Autoregressive generation | The exemplary generates token 1, past uses token 1 to make token 2, past 1+2 for token 3, and truthful on. You cannot skip ahead; each token depends connected each erstwhile tokens. |
| Arithmetic intensity | Ratio: mathematics done ÷ bytes publication from memory. Low (token decode): spot waits connected representation → memory-bound. High (prefill): spot stays engaged computing → compute-bound. Same chip, 2 phases, 2 bottlenecks. |
| Roofline model | A elemental floor plan engineers usage to ask: “Is this measurement constricted by mathematics velocity aliases representation speed?” Explains why prefill and decode behave otherwise connected identical hardware. |
| Memory-bound | The spot is idle waiting for information from memory. Token decode connected LLMs is usually memory-bound, adding much FLOPS does not thief until you adhd bandwidth aliases shrink the model. |
| Compute-bound | The chip’s mathematics units are afloat busy; representation keeps up. Prefill connected agelong prompts is often compute-bound. |
| MiB / GB / TB/s | MiB/GB = really overmuch fits (model size). TB/s = really accelerated information moves (serving speed). Example: 70B exemplary astatine FP16 ≈ 140 GB weights; connected 3.35 TB/s bandwidth, reference weights erstwhile takes ~42 ms, that is the theoretical level per token. |
Model size, precision, and quantization
| Parameter / 70B model | Each parameter is 1 learned number (a weight). 70B = 70 cardinal of those numbers. Larger parameter count = larger exemplary = much representation needed and usually smarter answers, but slower and costlier to serve. |
| Weights | All the numbers the exemplary learned during training, the “brain” you load into representation to serve. A 70B model’s weights must fresh connected your chip(s) aliases beryllium divided crossed many. |
| FP16 / BF16 | 16-bit number formats (~2 bytes per weight). Default for galore LLM deployments. A 70B exemplary ≈ 140 GB astatine FP16. |
| FP8 | 8-bit floating constituent (~1 byte per weight). Half the representation of FP16 pinch mini accuracy tradeoff. Supported connected H100/H200 and immoderate accelerators. |
| INT8 / INT4 | Integer formats (1 byte aliases ~0.5 byte per weight). Used aft quantization. A 70B exemplary ≈ 70 GB astatine INT8 aliases ~35 GB astatine INT4, fits connected smaller GPUs. |
| Quantization | Shrinking weights from FP16 to INT8/INT4 (or FP8) truthful the exemplary uses little representation and runs faster. Tradeoff: imaginable value nonaccomplishment connected immoderate tasks, ever measure connected your data. |
| GPTQ / AWQ | Two celebrated open-source methods to compress models to INT4 connected GPUs. Lets you tally a 70B exemplary connected an 80 GB GPU that could not clasp it astatine FP16. |
| TruePoint (Groq) | Groq’s mixed-precision mathematics format connected the LPU, really Groq represents weights and activations connected its chip. |
| cFP8 (Inferentia2) | AWS’s configurable FP8 format connected Inferentia2, a mediate crushed betwixt FP16 size and INT8 compression. |
| TOPS | Trillion integer operations per second, vendor spec for INT8 throughput (like FLOPS but for integer math). Useful for comparing accelerators connected quantized models. |
| Input shape | The nonstop series magnitude and batch size a compiled exemplary expects, e.g. “batch 1, series 2048.” Send a 512-token punctual to a exemplary compiled for 2048 and the compiler pads the remainder pinch zeros. |
| Padding | Filling a short petition up to the nearest compiled size. You salary compute for the padded tokens too, which shows up arsenic wasted activity and p99 latency spikes connected Inferentia2/TPU. |
Scaling crossed aggregate chips
| Tensor parallelism | Splitting 1 model’s layers crossed aggregate GPUs truthful each spot holds a portion and they compute successful parallel. Example: half the layers connected GPU 0, half connected GPU 1, some tally simultaneously connected the aforesaid request. |
| Sharding | Splitting exemplary weights into pieces dispersed crossed galore chips. Required erstwhile 1 spot cannot clasp the afloat model, Groq needs hundreds of chips for a 70B exemplary because each spot only has 220 MiB. |
| NVLink / NVSwitch | NVIDIA’s high-speed cables/chips connecting GPUs successful a server (900 GB/s connected H100). Lets tensor-parallel GPUs speech information quickly without going done slow PCIe. |
| NeuronLink | AWS’s nexus betwixt Inferentia2 chips successful an Inf2 lawsuit (192 GB/s). Used erstwhile a exemplary spans aggregate Inferentia chips. |
| 2D torus | A grid wiring shape successful TPU pods, each spot connects to neighbors successful a mesh. Google uses this to standard TPUs to 256 chips per pod. |
| Interconnect | Any web linking chips together. Slow interconnect = GPUs hold connected each different = you do not get linear speedup erstwhile adding chips. |
| Pod (TPU) | A ample wired cluster of TPUs (up to 256 chips successful v5e). You rent a portion (e.g. 8 chips) aliases the full pod depending connected exemplary size. |
Benchmarks and optimization techniques
| MLPerf Inference | An manufacture benchmark suite pinch fixed rules and third-party auditing. Vendors taxable hardware configs and scores, useful for comparing GPUs/TPUs connected the same benchmark, not crossed different benchmarks. |
| Artificial Analysis | An independent institution that measures unrecorded LLM API velocity and value crossed unreality providers. Used successful this article for Groq throughput numbers connected Llama models. |
| Speculative decoding | A mini “draft” exemplary guesses respective tokens quickly; the ample exemplary verifies them successful 1 pass. If guesses are correct, you output aggregate tokens per large-model step, often 2-3× faster generation. |
| A/B testing | Running exemplary type A for 50% of postulation and type B for 50% to comparison value aliases cost. Easy connected GPUs (hot-swap weights); achy connected compiled chips (recompile some versions). |
Article Scope and Methodology
This article compares spot architectures utilizing published vendor specs and nationalist architecture papers. Every hardware number links to its source. These are not DigitalOcean benchmark results. When a declare is astir really a creation works, not thing we measured successful production, the matter says so.
One caller arena is worthy noting. On December 24, 2025, NVIDIA licensed Groq’s conclusion exertion (reported astatine astir $20 billion), hired Groq’s laminitis and respective elder staff, and kept the woody non-exclusive. Groq still runs GroqCloud connected its ain nether caller leadership. Source: Groq newsroom. The woody shows that predictable, on-chip-memory conclusion matters moreover to the largest GPU maker. It does not alteration the tradeoffs below.
A taxonomy of conclusion silicon
Start pinch 4 spot categories. Each 1 optimizes for thing different.
Four spot families, 4 different tradeoffs. The GPU is the elastic baseline everything other is measured against.
General-purpose GPU. NVIDIA H100 and H200, AMD MI300X and MI325X. Load a exemplary and service it pinch nary compile step. Swap models, alteration precision, and set batching arsenic postulation changes. This is the elastic baseline. Every different action trades immoderate of that elasticity for a circumstantial gain.
Fixed-function accelerator. AWS Inferentia2 and Google Cloud TPU v5e successful serving mode. Built to tally 1 fixed exemplary cheaply astatine precocious volume. Both require a compile measurement that turns your exemplary into chip-specific code. Inferentia2 uses the AWS Neuron SDK. TPU uses XLA. Performance depends heavy connected the input shapes you compiled for.
Deterministic low-latency chip. The Groq LPU (Language Processing Unit). It stores exemplary weights successful accelerated on-chip representation and schedules each cognition up of time, truthful each petition takes the aforesaid number of timepiece cycles. You springiness up capacity and elasticity to get predictable speed. Groq documented the creation successful 2 peer-reviewed papers astatine ISCA: the original Tensor Streaming Processor successful 2020, and multi-chip scaling successful 2022. Source: Think Fast: A Tensor Streaming Processor for Accelerating Deep Learning Workloads, ISCA 2020.
Dataflow chip. Tenstorrent. Its serving package is little mature than the different three, and reliable accumulation specs are harder to find. This article covers it astatine the class level only.
Latency profiles: deterministic versus statistical
Pay adjacent attraction here. p99 latency is simply a day-to-day operations problem, and astir articles skip the hardware reasons down it.
For user-facing apps, median latency tells you little. What breaks an SLA is the slow tail, the p99 and p999. On a GPU, that tail comes from 2 creation choices. First, the GPU decides what to tally astatine runtime, truthful scheduling varies. Second, concurrent requests stock the aforesaid representation channels. These are built into the architecture. They are not signs of a bad deployment.
On a GPU, slow requests beryllium acold from the median. On a Groq LPU, latency stays tight by design.
Groq reduces that variety by deciding everything up of time. Its compiler maps the afloat exemplary to the hardware, down to the timepiece cycle, including connection betwixt chips. The aforesaid petition pinch the aforesaid input style takes the aforesaid number of cycles each time, truthful p50 and p99 beryllium adjacent together. Groq’s ISCA 2020 insubstantial describes this arsenic a halfway creation goal: region reactive hardware for illustration arbiters and caches truthful timing stays predictable. Source: Groq ISCA 2020 paper. That is Groq’s creation declare from a peer-reviewed paper, not a accumulation trial DigitalOcean ran.
The tradeoff is rigidity. GPUs usage continuous batching: the server groups a changing number of unrecorded requests into 1 pass. A fixed schedule cannot accommodate arsenic freely. Groq shines for single-stream, real-time serving. Its advantage shrinks erstwhile batch sizes plaything widely.
Inferentia2 and TPU v5e behave otherwise because of compilation. Both compile for circumstantial input shapes. A petition that does not lucifer gets padded to the nearest compiled shape. That padding shows up arsenic p99 latency spikes moreover erstwhile emblematic requests look fine. Neuron and XLA fto you take which shapes to compile, truthful you tin tune this, but it is simply a existent tail-latency root that GPUs avoid.
Compilation requirements and operational flexibility
GPUs triumph connected operational flexibility. That is simply a existent engineering advantage, not marketing.
Inferentia2 needs the exemplary compiled to AWS Neuron format done the Neuron SDK. TPU v5e needs XLA compilation. Compilation is not a one-time step. You repetition it erstwhile the exemplary changes, erstwhile precision changes, aliases erstwhile input size aliases batch size changes. If you vessel a caller exemplary each week, aliases tally respective versions astatine once, that costs ne'er goes away.
Same exemplary change, 2 paths. A GPU hot-swaps weights. A fixed-function accelerator recompiles each clip thing changes.
We do not person verified compile times for typical exemplary sizes, and we will not invent them. For the published version, propulsion existent figures from the AWS Neuron documentation and Google’s XLA and TPU docs, tally a compile connected a existent model, and study what you measurement pinch the package type noted. Until then, the constituent stands: compilation is simply a recurring costs that grows pinch really often your setup changes.
The 2nd constraint is usability coverage. On fixed-function hardware, you tin only tally what the compiler supports. If your exemplary uses an attraction version aliases activation the compiler does not implement, it either fails to compile aliases runs connected a slow fallback path, and you suffer the velocity you paid for. Check the AWS Neuron supported-operator database and the OpenXLA cognition database against your exemplary earlier you commit. This is the spread betwixt a vendor’s “supported models” database and the models you really service astatine afloat speed.
The 3rd constraint is precision. Precision options matter because quantization is really teams fresh larger models into little memory.
| FP16 | Yes | Yes | Via the BF16 path | High-precision mathematics via TruePoint |
| BF16 | Yes | Yes | Yes, native | TruePoint accumulation |
| FP8 | Yes | Configurable FP8 (cFP8) | Not connected v5e, added connected later TPUs | TruePoint mixed precision |
| INT8 | Yes | Yes | Yes, 393 INT8 TOPS | Yes |
| INT4, GPTQ, AWQ | Yes, software-supported | Limited | Limited | Set by the compiler path |
Sources: AWS Inferentia for the cFP8, FP16, BF16, and INT8 list. SemiAnalysis for the TPU v5e BF16 and INT8 figures. Groq for TruePoint. Verify the INT4 and quantization rows against existent SDK docs earlier publishing, because model support changes betwixt releases.
Choosing a GPU is simply a deliberate tradeoff. You springiness up Groq’s tight p99 latency and Inferentia2’s lowest costs per token connected 1 fixed exemplary astatine afloat utilization. In return, you switch models without recompiling, usage immoderate quantization your serving motor supports, and tally continuous batching to support hardware engaged nether changing load.
Throughput ceilings and exemplary size constraints
When an LLM generates tokens 1 astatine a time, velocity is constricted by really accelerated the spot sounds exemplary weights from memory, not by really accelerated it tin do math. The spot sounds the afloat exemplary for each token. So representation bandwidth, not highest FLOPS, sets the serving ceiling. Each token besides sounds per-request authorities from representation (the KV cache), which adds much traffic. Google researchers formalized this successful a peer-reviewed paper: procreation is dominated by loading weights and cache from memory, while punctual processing is compute-heavy. The 2 phases deed different bottlenecks connected the aforesaid chip. Source: Pope et al., Efficiently Scaling Transformer Inference, MLSys 2023.
Computer architects telephone the ratio of mathematics to representation postulation arithmetic intensity. It comes from the roofline capacity model:
Arithmetic strength = Total FLOPs performed ÷ Total bytes moved from memory
Low arithmetic strength intends memory-bound: the spot waits connected data. High arithmetic strength intends compute-bound: the mathematics units enactment busy. Prompt processing is compute-heavy. Token procreation is memory-heavy. That is why the 2 phases consciousness truthful different connected the aforesaid chip. Source: Williams, Waterman, and Patterson, Roofline: An Insightful Visual Performance Model for Multicore Architectures, Communications of the ACM, 2009.
Processing a punctual keeps the mathematics units busy. Generating each caller token mostly waits connected memory. That is why representation bandwidth, not FLOPS, decides really accelerated a spot serves.
The H200 is the clearest proof. It uses the aforesaid compute die(dies per wafer) arsenic the H100. The only alteration is much and faster memory, and that unsocial makes it overmuch faster astatine inference. Source: NVIDIA H200.
Capacity decides the largest exemplary that fits. Bandwidth decides really accelerated it serves. Groq’s mini barroom shows the SRAM tradeoff.
Two numbers successful that floor plan get misquoted constantly.
Inferentia2 bandwidth. AWS lists 9.8 TB/s for its largest Inf2 server. That is the full crossed each 12 chips, not 1 chip. Per chip, it is astir 820 GB/s. Quoting 9.8 TB/s arsenic a single-chip number overstates bandwidth by much than 10 times. Source: Amazon EC2 Inf2 instances.
Groq on-chip memory. Groq’s ISCA 2022 insubstantial states that each spot contributes 220 MiB to a shared on-chip representation pool. Marketing rounds this to “about 230 MB,” but the insubstantial is the precise source. There is nary HBM connected the chip. That azygous creation prime explains astir of really Groq behaves. Source: A Software-defined Tensor Streaming Multiprocessor for Large-scale Machine Learning, ISCA 2022.
Turning representation bandwidth into an existent millisecond number
The bandwidth declare is much useful arsenic a number you tin cheque against your workload:
Time per token, level = Model weight size successful bytes ÷ Memory bandwidth successful bytes per second
This is simply a theoretical floor, not a measured result. It assumes the spot spends the full measurement reference weights, pinch nary contention, nary KV cache reads, and nary batching overlap, truthful existent accumulation latency runs higher. It still tells you the fastest a spot could perchance go. Use it arsenic a sanity cheque connected vendor latency claims.
Each barroom divides a model’s weight size by that chip’s bandwidth. H200’s only alteration from H100 is memory, and the mathematics shows the effect directly: astir 41.8 sclerosis per token connected H100 versus astir 29.2 sclerosis connected H200, for the aforesaid 70B exemplary astatine FP16.
Worked out: a 70B exemplary astatine FP16 needs 140 GB of weights. On an H100 astatine astir 3.35 TB/s, that is 140 ÷ 3,350 ≈ 0.0418 seconds, astir 41.8 sclerosis per token astatine the floor. On an H200 astatine astir 4.8 TB/s, the aforesaid 140 GB takes 140 ÷ 4,800 ≈ 0.0292 seconds, astir 29.2 ms. The compute dice is identical. Only representation changed, and that unsocial explains the gap. Run this look for immoderate spot and exemplary size, arsenic agelong arsenic the exemplary fits successful that chip’s memory.
How large a exemplary fits connected 1 chip
You tin estimate the largest exemplary a azygous spot tin clasp from its representation size. The method useful for immoderate chip:
Weight size successful bytes = Parameter count × Bytes per parameter
- FP16 aliases BF16: 2 bytes per parameter
- INT8: 1 byte per parameter
- INT4: astir 0.5 bytes per parameter
So a 70B exemplary needs astir 140 GB astatine FP16, astir 70 GB astatine INT8, and astir 35 GB astatine INT4.
Apply that, weights only, earlier the moving representation each petition needs:
- H200, 141 GB: fits a 70B exemplary astatine FP16 astatine the outer edge, pinch almost nary room to spare.
- H100, 80 GB: fits 30B to 40B models astatine FP16, aliases a 70B exemplary erstwhile you quantize it.
- Inferentia2, 32 GB per chip: astir a 16B exemplary astatine FP16.
- TPU v5e, 16 GB per chip: astir an 8B exemplary astatine FP16.
- Groq, 220 MiB per chip: a mini portion of 1 model. A 70B exemplary spreads crossed hundreds of chips.
These are weights-only ceilings. Each petition besides needs moving representation that grows pinch batch size and discourse length, truthful existent limits beryllium lower. Treat these arsenic precocious bounds, not guarantees. No charismatic nationalist root states precisely really galore Groq chips a fixed exemplary needs successful production. Groq describes racks of chips moving together without ever giving a per-model count, dainty immoderate “N chips per model” fig elsewhere arsenic an estimate.
You tin estimate a little bound yourself:
Chips needed, level ≈ (Parameter count × Bytes per parameter) ÷ 220 MiB per chip
A 70B exemplary astatine INT8 needs astir 70 GB of weights. Divide by 220 MiB per spot and you get astir 303 chips, conscionable to clasp the weights. At FP16 (about 140 GB), that rises to astir 607 chips. Both numbers disregard KV cache, activation memory, and networking overhead, truthful dainty them arsenic a floor, not an bid spec.
A GPU holds a ample exemplary connected 1 aliases 2 chips. Groq’s mini per-chip representation intends the aforesaid exemplary spreads crossed a rack of chips, each holding a bladed slice. The spot count shown is simply a capacity-only floor, worked retired from the formula, not a published deployment spec.
The Groq tradeoff successful 1 line: on-chip representation is very fast, but location is very small of it, truthful ample models request galore chips and a batch of rack space.
Scaling crossed chips
When 1 spot cannot clasp a model, you dispersed it crossed several. Then the links betwixt chips go the bottleneck.
Each architecture connects chips differently. GPUs usage a full-speed mesh. TPUs way done a torus. Groq schedules chip-to-chip postulation astatine compile time.
- NVIDIA GPUs usage tensor parallelism complete NVLink. Fourth-generation NVLink connected H100 and H200 provides 900 GB/s full bandwidth per GPU crossed 18 links. NVSwitch lets each GPU successful a server scope each different GPU astatine afloat speed. Source: NVIDIA Hopper Architecture In-Depth.
- Inferentia2 scales complete NeuronLink astatine 192 GB/s betwixt chips wrong an Inf2 instance. Source: Amazon EC2 Inf2 instances.
- TPU v5e scales complete a 2D torus interconnect, up to 256 chips per pod. Source: SemiAnalysis.
- Groq uses a chip-to-chip web scheduled astatine compile time, the aforesaid attack arsenic wrong a azygous chip, truthful cross-chip connection stays deterministic. Groq’s ISCA 2022 insubstantial describes this design, but we could not find a published bandwidth fig successful GB/s. Treat immoderate Groq interconnect bandwidth number you spot elsewhere pinch be aware until Groq publishes one.
The cardinal question: does throughput standard astir linearly arsenic you adhd chips, and really overmuch does connection overhead cost? Use published interconnect bandwidth wherever vendors disclose it, and statement wherever they do not.
What independent benchmarks really show
Everything supra comes from vendor spec sheets. Specs show theoretical limits, not what a spot does nether existent traffic. This conception gathers independently measured numbers we could verify. One caveat: these numbers travel from different tests connected different models, truthful you cannot rank each 4 chips from this information alone. Treat them arsenic abstracted information points, not a leaderboard.
Real, independently measured numbers beryllium for immoderate of these chips, but they were measured connected different models pinch different methods. Read this arsenic respective abstracted information points, not 1 chart.
Groq, connected Meta’s unfastened Llama models, tested by the independent benchmarking patient Artificial Analysis: 241 tokens per 2nd connected Llama 2 70B successful February 2024, 284 tokens per 2nd connected Llama 3 70B a fewer months later, and 276 tokens per 2nd connected Llama 3.3 70B by December 2024, pinch a abstracted speculative-decoding endpoint reaching 1,665 tokens per 2nd connected the aforesaid model. Artificial Analysis’s unrecorded search page later showed Groq tied for the fastest output velocity connected Llama 3.3 70B among 17 providers astatine 296.7 tokens per second. Sources: Groq newsroom, citing Artificial Analysis and Artificial Analysis, Llama 3.3 70B supplier comparison. These are existent independent measurements, though Groq’s ain newsroom is the root for the humanities figures, truthful dainty the inclination arsenic directionally reliable and verify the existent number straight connected Artificial Analysis earlier citing it.
NVIDIA GPUs, connected the MLPerf Inference benchmark, which is tally and audited by MLCommons, a nonprofit manufacture consortium: connected the Llama 2 70B benchmark, H200 delivered up to 1.5 times the throughput of H100, and the newer Blackwell architecture delivered up to 4 times H100’s throughput, successful NVIDIA’s ain MLPerf Inference v4.1 submission. Source: NVIDIA Blackwell Platform Sets New LLM Inference Records successful MLPerf Inference v4.1. In a later round, CoreWeave’s ain H200 submission to MLPerf Inference v5.0 reported complete 33,000 tokens per 2nd connected the Llama 2 70B benchmark, a system-level number that depends heavy connected really galore GPUs were successful that configuration. Source: HPCwire sum of MLPerf v5.0.
Google TPU v5e has 1 straight comparable, audited result: successful its MLPerf Inference 3.1 submission, 4 TPU v5e chips ran the GPT-J 6B benchmark astatine 2.7 times the capacity per dollar of TPU v4, Google’s anterior generation. Source: Google Cloud, cost-efficient AI conclusion pinch Cloud TPU v5e. This is simply a small-model benchmark, not a 70B-class comparison, and I could not find a TPU v5e MLPerf submission connected a ample Llama exemplary to comparison straight against the GPU numbers above.
AWS Inferentia2 is the spread successful this section. We searched for a nationalist MLPerf Inference submission utilizing Inferentia2 and did not find one. AWS publishes its ain claims, specified arsenic 40 percent amended price-performance than comparable EC2 instances, but we could not verify an independently audited LLM throughput number the measurement Artificial Analysis and MLPerf supply for the different 3 chips. Check the existent MLCommons results page directly; submissions alteration each round.
Decision framework: which spot for which workload
Bookmark this section. It is simply a determination guide, not a income pitch.
Start astatine the top. Most paths lead to a GPU, because astir workloads alteration much often than they enactment frozen.
| One fixed model, precocious volume, costs matters most | Inferentia2 aliases TPU v5e | Compile costs pays off. Lowest costs per token astatine scale. |
| Strict p99 SLA, real-time, azygous stream | Groq LPU | Same latency each run, by design. |
| Models alteration often, aliases A/B testing | GPU | No recompile. Swap models freely. |
| Large aliases different models, uncommon operators | GPU | Avoids compiler support gaps. |
| Many models from 1 endpoint | GPU | No per-model compile step. |
In short: 1 stiff exemplary astatine precocious measurement pinch nary strict latency SLA → Inferentia2 aliases TPU whitethorn costs little per token. Strict latency SLA → Groq whitethorn fresh best. Models that alteration often, galore models, aliases different architectures → GPU.
When a specialized spot thumps a GPU
A GPU is the elastic default, but not ever the correct answer. Specialized chips triumph successful these cases:
- A hard, real-time p99 SLA. Voice agents and different interactive devices that request sub-second responses connected azygous requests are wherever Groq’s predictable latency pays off. There is nary cosmopolitan millisecond threshold, trial against what your users really tolerate. A GPU’s latency tail tin break a strict SLA nether load successful a measurement a fixed schedule does not.
- One model, stiff for a agelong time, astatine precocious volume. When the exemplary seldom changes, you salary the compile costs erstwhile and dispersed it crossed billions of tokens. Cost per token tin hit a GPU.
- A exemplary that matches the chip’s compiled shapes. Fixed-function chips activity champion erstwhile your input sizes lucifer what you compiled for, truthful padding stays small.
- Cost-sensitive batch conclusion pinch nary latency pressure. Offline aliases queued jobs attraction astir costs per token, not tail latency, fixed-function chips fresh well.
If immoderate of these picture your workload, a specialized spot is the logical pick. If your setup changes often, GPU elasticity is worthy much than highest ratio connected a constrictive benchmark.
Common Questions connected this topic?
1. Is a GPU ever the champion prime for LLM inference?
No. A GPU is the champion default for teams that alteration models often, trial quantization, aliases service respective models, because it needs nary recompile. For 1 stiff exemplary astatine precocious volume, Inferentia2 aliases TPU tin costs little per token. For a strict real-time latency SLA, Groq tin clasp a tighter tail.
2. Why does representation matter much than FLOPS for inference?
Token procreation sounds the full group of exemplary weights from representation for each token, past does comparatively small math. So serving velocity is constricted by representation bandwidth, and the largest exemplary you tin clasp is constricted by representation capacity. The H200 proves the point. It shares the H100’s compute dice and is faster only because it has much and faster memory.
3. Why does Groq request truthful galore chips?
Each Groq spot holds 220 MiB of on-chip SRAM, per Groq’s ain peer-reviewed architecture paper, and nary HBM. A ample exemplary does not fresh connected 1 chip, truthful it spreads crossed galore chips, each holding a bladed portion of the weights. That is the costs of the on-chip representation that makes Groq accelerated and predictable.
4. What is the existent costs of compiling for Inferentia2 aliases TPU?
The costs is not the first compile. You repetition the compile each clip the model, the precision, aliases the input style changes. A squad iterating play pays it again and again. A squad moving 1 stiff exemplary pays it once.
5. Did NVIDIA bargain Groq?
Not arsenic a afloat acquisition. In December 2025 NVIDIA took a non-exclusive licence of Groq’s exertion and hired its laminitis and respective staff. Groq still operates GroqCloud independently. The woody signals that deterministic conclusion matters, but the architectural tradeoffs successful this article still hold.
Conclusion
Specialized conclusion chips are real, and each 1 wins astatine thing specific. Groq wins connected predictable latency. Inferentia2 and TPU triumph connected costs per token for 1 fixed exemplary astatine scale. GPUs triumph for teams that request to alteration their setup, which is astir teams.
GPU serving stays the default not because nary 1 built a faster chip. It stays the default because specialized silicon comes pinch existent operational costs: recompiling connected each change, compiler gaps for different models, and rigid scheduling that does not accommodate to shifting load. When those constraints do not apply, specialized chips triumph connected a circumstantial dimension, and this article maps which magnitude and when.
If GPU serving fits your workload and you request to exemplary costs next, cheque retired our article connected dedicated GPU conclusion economics. DigitalOcean’s resources connected GPU Inference, What is AI Inference, choosing the correct GPU Droplet, and LLM conclusion benchmarking spell deeper connected moving conclusion astatine scale.
References
Vendor hardware specifications
- NVIDIA H100
- NVIDIA H200
- NVIDIA Hopper Architecture In-Depth
- AMD Instinct MI300X Accelerators
- AMD Instinct MI325X Accelerators
- Amazon EC2 Inf2 instances
- AWS Inferentia
- Google Cloud TPU v5e documentation
- OpenXLA / XLA compiler
- Groq LPU architecture explained
- Tenstorrent
Peer-reviewed papers
- Samuel Williams, Andrew Waterman, and David Patterson, “Roofline: An Insightful Visual Performance Model for Multicore Architectures,” Communications of the ACM, Vol 52, No 4, April 2009.
- Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, et al., “Efficiently Scaling Transformer Inference,” MLSys 2023.
- Dennis Abts et al., “Think Fast: A Tensor Streaming Processor (TSP) for Accelerating Deep Learning Workloads,” ISCA 2020.
- Dennis Abts et al., “A Software-defined Tensor Streaming Multiprocessor for Large-scale Machine Learning,” ISCA 2022.
Independent benchmarks
- Artificial Analysis, Llama 3.3 70B supplier comparison
- Groq newsroom, first independent LLM benchmark via Artificial Analysis
- NVIDIA Blackwell Platform Sets New LLM Inference Records successful MLPerf Inference v4.1
- HPCwire, MLPerf v5.0 results coverage
- Google Cloud, capacity per dollar of GPUs and TPUs for AI inference
- MLCommons, MLPerf Inference results
Other sources
- Groq and NVIDIA licensing agreement
- SemiAnalysis: TPU v5e benchmark
This activity is licensed nether a Creative Commons Attribution-NonCommercial- ShareAlike 4.0 International License.
.png "The Four Hidden Surcharges In Your Gpu Bill")
English (US) · 
Indonesian (ID) ·