Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

Over the past respective months, we’ve seen an detonation successful the capabilities of unfastened root models. With DeepSeek V4-Pro and GLM5.2 reaching near-Opus levels of intelligence, unfastened root has emerged arsenic a real, cost-efficient replacement to the closed root models we’ve been joined to.

But we person yet to spot 1 for illustration Kimi K3. Promising Fable/Sol levels of intelligence, Kimi K3 marks the commencement of a caller era for unfastened source.

But a smarter exemplary intends a bigger exemplary — and these models are expanding successful size conscionable arsenic accelerated arsenic they are successful capabilities. GLM5.2 has 753B parameters, DeepSeek V4-Pro 1.6T, and Kimi K3 weighs successful astatine 2.8T (!!) parameters. That’s complete 1.5TB of VRAM before allocating a KV cache for 1M tokens of context. Not moreover a B200 node (8 GPUs) tin fresh Kimi K3. That leaves you pinch constricted options: service connected a node of B300s, which person 288GB of VRAM per GPU, aliases perpetrate 2 B200 nodes (TP16) to serving Kimi.

But conjecture which different non-NVIDIA GPU has 288GB of VRAM? AMD’s MI355X. Can you show we for illustration these chips yet? At astir ~2.4× cheaper per GPU connected mean versus a B300 and ~1.7× cheaper than a B200, the MI355X is simply a cost-efficient replacement to Blackwells pinch comparable hardware specs. The only problem pinch AMD is package support — slower kernels and little day-0 support connected conclusion frameworks make serving frontier models connected AMD a existent engineering effort. Our declare astatine Wafer is that agents are improving astatine kernel and exemplary optimization, closing this spread arsenic we speak. But pinch AMD shipping day-0 support for Kimi K3, astir of the activity was already done for us.

The results are great: connected a 1,024-token input / 400-token output benchmark, the MI355X reaches 952 tok/s/node and 118 tok/s azygous watercourse — complete 3.8× the aggregate throughput per node and complete 1.3× the single-stream decode of our TP16 B200 deployment (whose 498 tok/s is simply a 16-GPU, 2-node full — ~249/node). B300 nodes still triumph ~1.65× connected aggregate throughput complete the MI355X, but astatine 2.4× the price, the MI355X crushes the B300 connected capacity per dollar.

Perf/dollar astatine $2.50/GPU-hr for the MI355X, $6.00 for the B300, and $4.25 for the B200.

To the B200’s defence, its numbers are somewhat deflated by the truth that it pays a cross-node all-reduce connected the decode captious way (RoCE v2 astatine ~195 Gb/s) — it’s the only config present that spans 2 nodes, because Kimi K3 won’t fresh weights positive a 1M-token KV excavation connected a azygous 8×192GB node. But that’s precisely the point: Kimi K3 astatine its size is 1 of the first models we’ve seen wherever the MI355X’s attraction connected HBM capacity gives it a practical, measurable separator complete the B200.

How we did it

While Kimi K3 served retired of the box, location was still activity to beryllium done to get it to its existent throughput number.

The main lever was speculative decode. K3 ships zero draught tensors — nary MTP, nary EAGLE — truthful the only speculative way is an outer block-diffusion draft: RadixArk’s Kimi-K3-DSpark. On CUDA it conscionable runs. On ROCm our first existent petition breaks the scheduler pinch this error:

sglang’s accept-sampling verifier has 2 ways to build the target distribution: a dense way that calls top_k_renorm_prob, and a sparse accelerated way that routes done torch.topk directly. The CUDA build imports top_k_renorm_prob from sgl_kernel; the ROCm build aliases only a Triton top-p kernel and leaves top_k_renorm_prob undefined — there’s nary top-k renorm kernel for gfx950 to alias. So the infinitesimal a petition lands connected the dense path, the verifier hits that NameError and takes the scheduler down pinch it.

The hole is simply a azygous PyTorch function. Top-k renorm is simply a mini operation: return the model’s probability vector, support the k highest entries, zero the rest, and rescale what’s near to sum to 1. A sort, a masked_fill, a disagreement — dropped consecutive into sglang’s ROCm sampling branch, the aforesaid computation the CUDA build gets from sgl_kernel. No civilization kernel: the reflex connected ROCm is to presume you request one, but present it was a missing definition, not a missing kernel.

With spec dec fixed and hardened, we gained ~2.2× capacity single-stream, ~1.7× per-stream astatine mean load, and +18% highest aggregate. More importantly, our highest aggregate throughput landed connected overmuch higher concurrency (c64 vs c24 no-spec).

Prefill optimizations

Discussion astir exemplary capacity tends to item decode tokens per second. But successful galore cases decode tok/s is fool’s golden — decode is over-glorified, while time-to-first-token, the number users feel the most, gets overlooked.

The MI355X struggles here: an identical 172k-token acold prefill took ~51s connected MI355X versus ~23s connected a B300. On a 1M-context model, a batch of workloads person immense prefills (sometimes cold), and having GPUs rotation connected prefill for minutes tin render full fleets of nodes useless.

The spread was almost wholly 1 kernel. K3 connected ROCm was falling backmost to slow generic Triton attraction because the accelerated AITER MLA prefill kernel wouldn’t load. The problem was a style mismatch, not a missing kernel — K3 astatine TP8 gives 12 attraction heads per rank, and AITER’s MLA way is built for 4, 8, aliases multiples of 16. The hole was trivially simple: zero-pad the caput count 12→16, tally the accelerated kernel, and extract the existent 12 heads from the output.

The result: connected the aforesaid 172k acold prefill, the AITER MLA prefill ASM runs astatine ~13k tok/s steady-state vs the Triton fallback’s ~4–7k, speeding up prefill by ~2–3×. It’s a TTFT lever, not an aggregate-throughput 1 — decode is unchanged, truthful it doesn’t move the numbers above; it moves the number a personification waits connected earlier the first token appears.

Takeaways

Achieving the champion performance-per-dollar ratio connected the MI355X was comparatively retired of the box. There were immoderate expected framework-related bugs — but less than GLM5.2, and this clip it certainly did not require civilization kernels.

SOTA connected AMD is imminent. Is the CUDA moat dead?