DeepSeek V4 Flash on a Single AMD MI300X

Aug 04, 2026 05:00 PM - 3 hours ago 1

This repository contains the configuration and patches I usage to tally deepseek-ai/DeepSeek-V4-Flash-0731 connected one AMD MI300X successful production. It includes the Docker Compose stack, SHA-256-pinned record overlays, reference diffs against upstream, and tuning tables. The checkpoint runs arsenic shipped, without further weight quantization aliases offload.

Results from the pinned stack (vLLM ROCm nightly 0.26.1rc1.dev229+g124154a88.rocm723, AITER 0.1.19):

Metric Result

Single-stream decode (median per-stream, DSpark-7)	168.6 tok/s
Prefill pinch tuned kernels	≈ 7.9–8.5K tok/s (6,988–7,019 tok/s connected caller prompts successful the shipping profile)
8 concurrent streams	542 tok/s aggregate, 90.3 tok/s median per stream
64-stream burst	830 tok/s aggregate, nary OOM, nary motor errors
Context	256K validated (the architecture supports 1M)
Weights successful HBM	156.67 GiB — no further quantization aliases weight offload

The charismatic vLLM look targets NVIDIA and newer AMD hardware. Running the exemplary reliably connected MI300X required fixes for its FP8 format, MoE routing astatine precocious concurrency, causal speculative verification, CPU-KV synchronization, and respective untuned kernel shapes. This repository collects those fixes and pins the versions utilized successful production.

The MI300X has 192 GB of HBM3 and 5.3 TB/s of representation bandwidth, pinch 2.4× the HBM capacity of an H100 SXM5 (AMD). Doubleword's write-up estimates that it costs astir half arsenic overmuch astatine database price. For this 304B-parameter checkpoint, the representation capacity allows a elemental single-GPU deployment:

The full exemplary fits successful HBM without PCIe weight streaming aliases furniture offload.
There is room for a 20 GB GPU KV excavation and a 96 GiB CPU tier for evicted prefix-cache entries.
One paper handles 2–8 emblematic concurrent streams and bursts of up to 64 streams.

MI300X (CDNA3) implements the AMD/Graphcore fnuz version of E4M3, while MI325X and newer usage OCP-standard FP8 (background). A kernel that assumes OCP semantics connected MI300X tin beryllium incorrect by a facet of 2 successful the standard domain. Correctness connected this FP8 implementation was the first priority; capacity tuning came afterward.

Prior art, and what this repo adds

Fergus Finn's MI300X worklog and the accompanying Doubleword repository identified the FP8 incompatibility, missing AITER accelerated paths connected gfx942, HIP-graph hazards successful sparse MLA decode, and MoE routing bugs. The official vLLM recipe covers NVIDIA hardware and newer AMD GPUs (MI325X astatine 4K discourse and MI355X), but not a single-MI300X accumulation configuration for the 0731 checkpoint.

This repository adds:

Correctness overlays for the pinned ROCm nightly, including fixes not yet successful upstream vLLM.
A validated serving configuration pinch probabilistic DSpark drafting, artifact rejection, and fixed K=7. It uses a 2,048-token scheduler fund and a 1,024-token long-prefill headdress to forestall a acold punctual from stalling different streams.
AITER GEMM tuning tables for the recurring gfx942 shapes the packaged tables were missing, positive a gfx942 OGS geometry override for the MXFP4 experts.
A hybrid KV strategy: 20 GB of fp8_ds_mla GPU cache + 96 GiB autochthonal CPU offload, pinch a load-path fencing hole that upstream issue #47282 documents but PR #47291 ne'er merged.

. ├── compose.yaml # The accumulation stack (vLLM ROCm + Caddy), digest-pinned ├── Caddyfile.example # Copy to Caddyfile; group hostname, email, and root CIDR ├── vllm-entrypoint.sh # Removes old CPU-KV mmaps from /dev/shm earlier start ├── SHA256SUMS # SHA-256 pins for each runtime artifact ├── patches/ │ ├── *.py # Byte-for-byte accumulation overlays (mounted read-only) │ ├── diffs/*.patch # Unified diffs vs. the upstream guidelines revision │ └── README.md # Provenance and regeneration instructions └── tuning/ └── *.csv # AITER A8W8 blockscale tuning tables for gfx942

The stack uses a digest-pinned charismatic vLLM ROCm nightly with:

--trust-remote-code and the DeepSeek V4 tokenizer, reasoning, and instrumentality parsers
fp8_ds_mla KV cache (UE8M0 block-scaled FP8, not generic unscaled FP8) pinch 256-token blocks
VLLM_ROCM_USE_AITER=1 and --moe-backend triton; Triton OGS handles the grouped MXFP4 experts, while AITER handles attraction and dense linear layers
DSpark-7 speculative decoding pinch probabilistic drafting and artifact rejection
full/breakable CUDA chart capture, giving 1 chart motorboat per token during dependable decode
Caddy arsenic an IP-allowlisted HTTPS proxy

One MI300X (gfx942, 304 CUs, ~192 GiB HBM), a moving AMD kernel driver, caller Docker Compose, ~235 GiB RAM for the CPU KV tier, and ~500 GB disk (the exemplary cache unsocial is ~156 GB).

2. Pull the pinned runtime and model

VLLM_IMAGE='vllm/vllm-openai-rocm@sha256:e68d18b2ba50298661bfc49baf01158fbf036645c2362cccf3e8a7a79fe6c69a' MODEL='deepseek-ai/DeepSeek-V4-Flash-0731' REVISION='7872f01b1d1fe23eabc4c98b48bffcef5a386062' docker propulsion "$VLLM_IMAGE" docker tally --rm --entrypoint hf \ -v /root/.cache/huggingface:/root/.cache/huggingface \ "$VLLM_IMAGE" download "$MODEL" --revision "$REVISION"

cp Caddyfile.example Caddyfile # past group your hostname, email, and remote_ip CIDR mkdir -p aiter-cache crash-dumps chmod +x vllm-entrypoint.sh sha256sum -c SHA256SUMS # verify the overlays earlier first start

docker constitute config -q docker constitute up -d docker constitute logs -f inference

A patient commencement takes ~5 minutes and must show each of:

Model loading took 156.67 GiB DSpark draught exemplary loaded: 96 params GPU KV cache size: 1,927,444 tokens Maximum concurrency for 262,144 tokens per request: 7.35x Created mmap record /dev/shm/vllm_offload_...mmap (103.08 GB) Capturing CUDA graphs (FULL) Application startup complete

After chart capture, tally rocm-smi --showmeminfo vram. The warmed high-water people is ~204.5 GB of 205.8 GB. If only a fewer 100 MB remain, the server whitethorn commencement but neglect connected the first request.

HOST='your-host.example.com' curl -fsS "https://$HOST/v1/models" curl -sS "https://$HOST/v1/completions" \ -H 'Content-Type: application/json' \ -d "{\"model\": \"deepseek-ai/DeepSeek-V4-Flash-0731\", \"prompt\": \"Calculate 17 * 23. Answer pinch the number only.\", \"temperature\": 0, \"max_tokens\": 32}"

Each patches/*.py record is simply a full-file overlay mounted read-only complete its counterpart successful the container; compose.yaml contains the target paths. The corresponding diffs/*.patch records the alteration from its upstream base. The guidelines image remains digest-pinned, truthful upgrades require changing the image reference and revalidating the stack.

Overlay Mounted over Fixes Needed when

gpt_oss_triton_kernels_moe.pack128-fused-silu-fast-routing.py	vllm/.../fused_moe/experts/gpt_oss_triton_kernels_moe.py	MXFP4 bitmatrix padding lanes + fused-SiLU grouped experts + accelerated DeepSeek routing	Required for the MXFP4 Triton path; the disguise hole is not yet upstream
mxfp4.fused-silu.py	vllm/.../fused_moe/oracle/mxfp4.py	Gate/up interleave layout for the fused-SiLU kernel	Required pinch the fused-SiLU overlay; skip some if you support the modular SiLU path
triton-kernels-matmul-ogs-opt-flags.dsv4-mi300x.py	vllm/third_party/triton_kernels/matmul_ogs_details/opt_flags.py	gfx942 MXFP4 OGS tile geometry (up to 1,536 routed rows)	Performance connected gfx942; the banal geometry slows sharply supra 768 routed rows
fused_compress_quant_cache.fnuz-shuffle.py	vllm/models/deepseek_v4/common/ops/fused_compress_quant_cache.py	FNUZ FP8 + 16×16 preshuffle successful the Lightning Indexer cache writer	Required connected MI300X; MI325X/MI355X usage OCP FP8 and must support the banal bytes
aiter_pa_mqa_logits.i64.py	aiter/ops/triton/gluon/pa_mqa_logits.py	64-bit offsets successful the ChunkK=256 paged-MQA kernels	Required erstwhile KV offsets tin transcend 4 GiB; skip for mini KV pools
rocm_aiter_mla_sparse.prefill-bh64.py	vllm/v1/attention/ops/rocm_aiter_mla_sparse.py	Deterministic torch.topk prefill + BLOCK_H=64 head-512 sparse prefill	Determinism is required for reproducible instrumentality calls; BLOCK_H=64 is performance
rocm_aiter_mla.dspark-causal.py	vllm/v1/attention/backends/mla/rocm_aiter_mla.py	Causal multi-token speculative verification	Required for DSpark connected ROCm small-head MLA — now upstream; the overlay is the upstream record verbatim
dspark-speculator.independent-draft-gumbel.py + spec-decode-utils.independent-draft-gumbel.py	vllm/v1/worker/gpu/spec_decode/dspark/speculator.py + .../spec_decode/utils.py	Draft-proposal Gumbel sound salted distant from rejection/recovery noise	Required only pinch draft_sample_method=probabilistic (the recipe's greedy way does not request it)
kv_offload_cpu_gpu_worker.load-war.py	vllm/v1/kv_offload/cpu/gpu_worker.py	Fence CPU→GPU KV restores down in-flight compute (#47282, PR #47291)	Required only pinch --kv-offloading-backend native

Two important correctness fixes

MXFP4 routing. The MoE bitmatrix kernel pads its artifact columns to a Triton artifact size, but the padding lanes were masked against the world tensor bound alternatively of the logical artifact size. Under load, padded lanes corrupted the routing matrix, causing near-match instrumentality names and forgotten schemas connected agelong prompts. The one-line hole is disguise = (offs_local < BLOCK_SIZE) & (offs_global < nonzero_indx_size), taken from Doubleword perpetrate c32932bb9. The overlay besides includes fused-SiLU and fast-routing changes for grouped MXFP4 experts.

FP8 format. DeepSeek V4's Lightning Indexer cache uses FP8. The banal writer emits OCP E4M3 bytes successful row-major order, while AITER connected MI300X consumes AMD FNUZ E4M3 bytes successful a preshuffled 16×16 tile layout. In the worst case, interpreting 1 format arsenic the different produces a factor-of-two standard error. The overlay selects float8e4b8 pinch FP8_MAX=224.0 and shuffled constitute offsets connected ROCm, while leaving the OCP way unchanged elsewhere.

This stack uses probabilistic drafting pinch artifact rejection. The 2 Gumbel overlays support draft-proposal sound independent of rejection and betterment noise.

Key optimizations successful the accumulation configuration:

Change Effect

Tune 21 recurring A8W8 GEMM shapes for 304-CU gfx942	+42–62% single/double-stream decode; +10–35% astatine 8–64 streams
Fused SiLU, accelerated DeepSeek routing, batch-sensitive master tiles	Native C1 decode 34.5 → 56.6 tok/s (+64%); routing kernel 42.6 → 11.9 µs/layer
BLOCK_H=64 sparse-prefill tile	Prefill reaches 7.9–8.5K tok/s; sparse-attention trace 317 → 142 sclerosis per request
Static K=7, probabilistic + artifact rejection, causal verify	119.5 tok/s single-stream pinch correct output
2,048-token fund + 1,024-token long-prefill cap	Late short-request TTFT down a 52K prefill: 8.2 s → 0.5 s
20 GB GPU KV + 96 GiB CPU tier	1.93M-token length-equivalent capacity; 7 256K requests admitted

Distinct ~400-word prompts, streaming, temperature=1.0, top_p=0.95; C1–C8 astatine 512 output tokens, C64 astatine 256:

Streams Aggregate tok/s Median per-stream decode TTFT p50

1	126.2	168.6 tok/s	1.026 s
2	145.4	152.7	0.939 s
4	316.8	108.6	0.369 s
8	542.3	90.3	1.027 s
64	830.2	16.4	2.190 s

DSpark acceptance is prompt-dependent; dainty these arsenic gates for this nonstop image, not cosmopolitan exemplary benchmarks.

With the tuned kernels, uncached prefill reaches 7.9–8.5K tok/s, depending connected scheduler budget: 7.90–7.99K astatine C1 pinch an 8,192-token fund and 8.46–8.51K astatine C4. The accumulation floor plan uses a 2,048-token fund for latency isolation, giving 6,988–7,019 tok/s connected caller prompts. With the 1,024-token long-prefill cap, an 8.9K-token punctual reaches 5.20–5.29K tok/s astatine C1. In exchange, TTFT for a short petition queued down a 52K acold prefill drops from 8.2 s to 0.5 s. Warm callback of 380K cached tokens takes 0.64–2.65 s aft a 120–125 s acold prefill.

HBM headroom is limited. The warmed high-water people is 204.5 of 205.8 GB. A 30 GB KV excavation loads but fails during chart seizure pinch HSA_STATUS_ERROR_OUT_OF_RESOURCES. Do not raise --kv-cache-memory-bytes; show HBM usage for growth.
The CPU KV tier stores cache entries, not weights. --kv-offloading-size 96 --kv-offloading-backend autochthonal maps ~103 GB successful /dev/shm for evicted prefix-cache entries. The entrypoint removes old mappings aft crashes.
The 1,664-token scheduler informing is expected. DSpark-7 reserves draught slots from the 2,048-token budget. Raising the fund reserves much in-flight sliding-window authorities and reduces usable KV capacity.
Warm the kernels aft restart. The first prefill initializes kernels and takes 5.3 s for 8.9K tokens; consequent runs return 1.7 s. Run 1 uncached prefill earlier admitting traffic.
Test correctness arsenic good arsenic throughput. The validation suite includes two-turn tool-calling fixtures, a BFCL subset (74–76/90 nonstop calls), OpenCode tool-schema checks, and 380K-token needle callback connected some autochthonal and DSpark paths. Cold and cached prefills tin return different floating-point paths, truthful trial both.

The stack, documentation, and vLLM-derived overlays are Apache-2.0 (see LICENSE); the AITER-derived overlay keeps its MIT header. Upstream guidelines revisions for each diff are recorded successful patches/README.md. The exemplary itself is MIT-licensed.

All links verified 2026-08-04.

DeepSeek-V4-Flash-0731 exemplary card — charismatic release; 304B parameters; fused DSpark module; recommended temperature=1.0, top_p=0.95; MIT license
Official vLLM DeepSeek V4 Flash recipe — reference motorboat configuration, DSpark (num_speculative_tokens=7), FP8 KV, artifact size 256, deepseek_v4 parsers; AMD guidance for MI325X/MI355X
Bringing up DeepSeek-V4-Flash connected AMD MI300X (Fergus Finn, Doubleword, June 2026) — the bring-up worklog this repo builds on: FNUZ vs. OCP FP8, AITER gaps connected gfx942, HIP-graph hazards, routing bugs
doublewordai/vllm-amd-blog-doubleword — demo PRs for the above, including commit c32932bb9 ("mask MXFP4 bitmatrix padding lanes by logical artifact size")
vLLM perpetrate 77469c9 — "[ROCm][MLA] Mask the AITER MLA small-head verify flatten causally (#50476)"
vLLM rumor #47282 — CPU-KV load way lacks cross-stream sync pinch compute (WAR gap)
vLLM PR #47291 — projected WAR fix, not merged; carried arsenic an overlay here
AMD Instinct MI300X — 192 GB HBM3, 5.3 TB/s highest bandwidth, 2.61 PFLOPS highest FP8
ROCm/AITER — AMD tuned-kernel room utilized for ROCm attraction and dense linears
vLLM — the serving runtime (ROCm nightlies nether vllm/vllm-openai-rocm)