Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s): 50 tok/s with MTP

I gave Qwen3.8's MTP drafter different 69.2 MiB of precision. Throughput fell from 50.44 to 37.02 tokens per second. That consequence sums up the full experiment: the champion section conclusion setup is seldom made from the individually "best" parts.

I wanted a dense 27B model, its afloat 262,144-token context, multimodal input, maximum useful quality, and speculative decoding connected an NVIDIA RTX PRO 4000 Blackwell SFF pinch 24 GB of VRAM. The server besides had to past existent supplier activity aft printing exemplary loaded. The research followed a hunch I had written astir earlier: observant cognition whitethorn matter arsenic overmuch arsenic moving to a larger model.

The vanished strategy averages 50.44 tok/s successful the existent ten-run accumulation series. On a strict runtime A/B, the civilization llama.cpp build reaches 55.40 tok/s versus 45.42 for cleanable master, a 21.97% gain. Against target-only greedy decoding, embedded MTP moves 21.19 to 59.46 tok/s, aliases 2.81 times the throughput. At the acold extremity of a genuinely occupied 256K cache, it still produces 12.61 tok/s without an out-of-memory failure.

Those numbers came from different gates and should enactment separate. Combining them into 1 heroic speedup would make a amended header and a worse benchmark.

The winning setup came from the fresh betwixt the quant, drafter, CUDA kernels, representation layout, and workload. No constituent won connected its own.

The target was deliberately unreasonable

Qwen3.8 27B is simply a 64-layer dense model. Its repeating shape contains 3 Gated DeltaNet layers followed by 1 full-attention layer, giving 48 recurrent layers and 16 accepted attraction layers. It has a autochthonal 262,144-token context, a one-layer MTP head, and a abstracted 27-layer imagination encoder.

The hardware is lopsided successful a useful way:

• GPU0: RTX PRO 4000 Blackwell SFF, 24 GB GDDR7 pinch ECC, a 192-bit representation interface, 432 GB/s highest representation bandwidth, 24,467 MiB reported capacity, and sm120a. It holds the target, embedded MTP, recurrent state, graphs, and the 256K KV cache.
• GPU1: RTX 2000 Ada, 15,996 MiB, sm89. It holds the F16 multimodal projector and different auxiliary services.
• Runtime: Debian 13, CUDA 12.9.86, GCC 14.2, dual-architecture CUDA build.

Only the 16 full-attention layers turn a accepted KV cache pinch series length, which makes 256K little absurd than it first appears. With Q4 K and V, that cache costs astir 4.25 GiB earlier allocator overhead. DeltaNet adds recurrent authorities and checkpoints instead. Four checkpoints were the useful minimum; the default 32 spent representation I needed elsewhere.

NVIDIA quotes 432 GB/s of highest bandwidth. That is simply a hardware ceiling alternatively than an exertion metric from llama.cpp, but it matters here. Autoregressive decode many times streams quantized weights, and the 16 attraction layers adhd progressively costly KV sounds arsenic discourse fills. This is why the aforesaid floor plan averages astir 50 tok/s connected the accumulation task and 12.61 tok/s astatine the acold extremity of a 261.5K-token cache.

The original scheme was simple: estimate the capacity, prime a quant, past benchmark it. The instrumentality instantly taught maine that capacity estimates are conscionable admittance tickets. The existent trial originates aft loading.

The first victor was Q4_0, and it was the incorrect winner

I began pinch nationalist GGUFs astatine 40K context. Q4_0 was amazingly strong. Target-only decoding reached 22.40 tok/s, and MTP pinch n_max=3 reached 44.95. It hit smaller Q3_K_M and nominally smarter Q4_K_M variants because record size and quant explanation do not picture the CUDA kernel that really runs.

Then value testing spoiled the easy answer. On a short, identical WikiText-2 control, IQ4_XS scored 6.1175 perplexity while Q4_0 scored 6.3798. Q4_0 led the velocity table. Hermes needed a main model, though, and that value waste and acquisition felt excessively costly for a fewer 100 milliseconds. I would person been utilizing a 27B exemplary arsenic oversized autocomplete.

The other utmost grounded too. Q4_1 reached 6.1127 PPL, marginally up of IQ4_XS, but its representation footprint made 256K positive F16 imagination uncomfortable. The useful constituent was location betwixt a accelerated blunt quant and a precise record that near nary room for the remainder of the system.

Loading 256K proves almost nothing

Early capacity tests looked excellent. Q4_0, MTP, Q4 KV, 4 recurrent checkpoints, and the F16 projector each allocated astatine 262,144 context. That still did not reply the mobility I cared about.

I filled the slot pinch 261,500 input tokens, generated different 256, and past reused the basking cache. No truncation. No OOM. The first Q4_0 floor plan decoded astatine 12.06 tok/s adjacent the extremity of the cache, compared pinch 44.95 astir 40K. GPU usage sat astatine 99 to 100%, while the server utilized astir 1 CPU core. The bottleneck was the 16 full-attention layers reference a immense occupied KV cache, not a concealed CPU fallback.

This changed the benchmark method for each tally that followed. "262K loaded" was banned from the results table. A long-context declare had to see existent token fill, post-fill VRAM, basking decode, truncation state, and an output hash.

The ready-made NVFP4 quant grounded the value gate

Blackwell has autochthonal FP4 hardware, truthful a ready-made NVFP4-MEDIUM GGUF looked for illustration the evident route. Its bulk target matrices utilized NVFP4, pinch a Q8 output head, Q6 embeddings, and an IQ4_XS MTP layer. It reached 40.46 tok/s and fitted the complete 256K positive imagination floor plan pinch astir 1,055 MiB free.

Its PPL was 6.4949. Worse than Q4_0.

The conversion look was the problem. Attention and DeltaNet weights from the root FP8 checkpoint had been expanded and requantized into NVFP4 on pinch the large, tolerant matrices. Native arithmetic made the record quick, while indiscriminate debased precision damaged delicate parts of the model. Hardware format support does not show you wherever to walk the bits.

That nonaccomplishment gave america the creation for a civilization quant: usage NVFP4 for the bulk, past protect only the tensors that our ain workload says matter.

I calibrated the exemplary connected really I really usage it

The calibration corpus started pinch 5,472 messages from 296 Hermes sessions. I placed that worldly earlier the generic corpus truthful the 153,600 processed tokens represented coding, Polish and English conversation, infrastructure work, instrumentality calls, and the awkward mixtures my agents really see. A concealed scan ran earlier calibration. No PEM keys, supplier tokens, GitHub tokens, Slack tokens, aliases email addresses were present.

llama-imatrix collected value information for 497 target weights. NVFP4 does not devour an iMatrix straight during artifact quantization, truthful I utilized the matrix arsenic a map: ample tolerant tensors stayed autochthonal NVFP4; selected attention, DeltaNet, and FFN tensors moved to Q5_K aliases Q6_K; embeddings became Q6_K; the output caput stayed Q8_0.

The first 5.14 BPW hybrid was the value victor astatine 6.0967 PPL. It was besides slow astatine 34.19 tok/s and excessively ample to support the desired projector connected GPU alongside 256K. Good experiment. Bad accumulation model.

The 2nd build was tighter:

• Size: 16,321.38 MiB, 5.01 BPW.
• Bulk matrices: autochthonal NVFP4.
• Sensitive target tensors: selected Q5_K and Q6_K utilizing the Hermes iMatrix ranking.
• Embedding and output: Q6_K and Q8_0.
• Embedded MTP: autochthonal NVFP4.
• PPL: 6.1197, versus 6.1127 for Q4_1. The 0.11% spread is acold beneath the correction of this short control.

This became Qwen3.8-27B-Hermes-iMatrix-NVFP4-Balanced.gguf. It preserved the measured value of the Q4_1 reference, ran faster, and near capable room for the existent serving stack.

MTP had a trapdoor astatine n=8

The early expanse suggested n_max=3. Values 4 done 7 sewage slower arsenic rejected draught activity accumulated. Then n=8 jumped to 49.31 tok/s.

The curve is jagged. Eight candidates deed a favorable batch and kernel shape. Nine was nary faster, and each other campaigner costs astir 150 MiB. With the afloat 256K allocation, n=9 astatine ubatch=256 grounded connected 1 much 162 MiB CUDA chart buffer. Reducing ubatch to 128 made it load, but throughput fell to 52.70 tok/s and 32K prefill suffered. N=10 grounded connected different 81 MiB. Even an experimental scheduler excavation mislaid to a 31 MiB allocation.

I kept n=8 because it was the past accelerated constituent earlier the allocator started biting.

More meticulous MTP made the strategy worse

I wanted a controlled rival for the NVFP4 MTP choice. A patched MTP-aware iMatrix tally processed 300 Hermes-history chunks and added each 8 MTP matrices. For the comparison, each 851 non-MTP tensors were verified byte-for-byte identical to production. Only the 8 MTP weight tensors changed.

The higher-bit drafter whitethorn beryllium person to the BF16 root model. In accumulation it had 1 job: foretell this quantized target. The NVFP4 errors successful the MTP caput happened to align amended pinch the NVFP4-heavy target, truthful its nonstop proposals survived much often. Standalone precision mislaid to quant-drafter alignment.

This changed really I dainty drafter quality. The drafter and target shape 1 quantized system, and their relationship decides acceptance and throughput. A standalone value people for either half misses it.

DSpark had a full 2nd GPU and still lost

I besides tested Qwen3.8-27B-DSpark, a 1.36B diffusion drafter pinch a Markov caput and assurance head. A llama.cpp spot allowed the sidecar to tally connected GPU1 while the target stayed connected GPU0.

The champion type was Q8_0 astatine 26.49 tok/s, 34.5% supra its 19.70 tok/s target-only reference. Embedded MTP reached 49.61 tok/s connected the aforesaid comparison. DSpark was 46.6% slower and utilized much aggregate VRAM. The sidecar had been trained against an FP8 target, while ours was a mixed NVFP4/Q5/Q6 target. PCIe postulation and the weaker Ada paper did the rest.

DSpark's diffusion system worked, but this sidecar was trained for a different target and mislaid connected this box.

The llama.cpp build mattered almost arsenic overmuch arsenic the model

Once the exemplary stabilized, I benchmarked runtime changes 1 branch astatine a time. N-gram speculation was abnormal for each A/B because repeating a punctual taught the cache and pushed evident throughput from 46.77 to astir 180 tok/s. Useful successful production, poison successful a kernel comparison.

The last build pins #26001, #26048, #26705, #27173, #24891, and #25635 to audited perpetrate hashes. The patches screen Gated DeltaNet, CUDA dispatch, faster Q4/Q5 speculative verification, chained MTP, recurrent-checkpoint correctness, and Flash Attention swizzling.

The Flash Attention spot deserves its ain number. At 32K it moved prefill from 759.38 to 815.64 tok/s, up 7.41%, and basking decode from 37.26 to 38.23, up 2.61%. It helped wherever the attraction workload was ample capable to matter.

The builder refuses to proceed if immoderate PR caput moves. It builds sm89 and sm120a, past runs sampling, quantization, Gated DeltaNet, Flash Attention, and NVFP4 matrix tests. Without those checks, the adjacent upstream update could move a accelerated backstage binary into an outage.

The last accumulation profile

The progressive exemplary is Qwen3.8-27B-iMatrix-NVFP4-256K-MTP. The important runtime settings are:

CUDA_VISIBLE_DEVICES=0,1 MTMD_BACKEND_DEVICE=CUDA1 LLAMA_SPEC_CHAIN=1 GGML_CUDA_GRAPH_OPT=1 llama-server \ --model Qwen3.8-27B-Hermes-iMatrix-NVFP4-Balanced.gguf \ --device CUDA0 --n-gpu-layers 999 --fit disconnected \ --ctx-size 262144 --parallel 1 --ctx-checkpoints 4 \ --flash-attn connected \ --cache-type-k q4_0 --cache-type-v q4_0 \ --batch-size 512 --ubatch-size 256 \ --temp 0.6 \ --spec-type draft-mtp --spec-default \ --spec-draft-n-max 8 --spec-draft-n-min 0 --spec-draft-p-min 0 \ --spec-draft-type-k f16 --spec-draft-type-v f16 \ --spec-draft-backend-sampling \ --mmproj mmproj-F16.gguf \ --reasoning-preserve --jinja --metrics

--spec-default adds n-gram speculation successful this build. It stays enabled because supplier sessions repetition code, instrumentality schemas, and punctual prefixes. It ne'er appears successful comparative benchmark numbers. Temperature is 0.6 successful the stack, while reasoning effort remains request-controlled by Hermes. I do not unit xhigh globally.

--agent is deliberately absent. Hermes owns devices and MCP. Enabling llama-server's supplier furniture would copy the instrumentality loop and widen the code-execution surface. --fit disconnected is deliberate too: automatic fitting would silently alteration discourse aliases offload to reserve its default VRAM margin, invalidating the floor plan we measured.

The F16 projector uses 982 MiB connected GPU1. That is amended than squeezing it onto GPU0, and overmuch amended than moving a abstracted 4B imagination model. The progressive 17 GiB exemplary and binaries enactment connected section NVMe for accelerated restarts; inactive experiments unrecorded connected slower web storage.

The numbers I kept

The full-context tally generated 256 tokens without truncation aliases OOM. Cold and some basking outputs had the aforesaid hash. The ten-run accumulation bid was besides deterministic wrong its configuration.

One correctness caveat remains. Target-only greedy and MTP n=8 nutrient different continuations connected a quantized target, matching the unfastened llama.cpp batch-invariance rumor #25618. Both modes are internally deterministic and the outputs are coherent, but this max-TPS floor plan is not bitwise distribution-preserving comparative to target-only decoding. N=1 is the safer lossless mounting erstwhile that spot matters.

The mistakes worthy keeping

I would commencement pinch the accumulation task distribution and 3 difficult gates: quality, existent discourse fill, and deterministic A/B. Quant labels would travel later.

I besides learned to time off the past fewer 100 MiB alone. We intentionally ran adjacent 76 MiB free during 1 tuning phase. It worked until a 31 MiB scheduler allocation and chart fragmentation showed why arithmetic free representation is not operational headroom. The last astir 500 MiB separator keeps the server live done chart creation, imagination requests, and allocator variation.

Next clip I will trial the drafter against the nonstop quantized target from time one. The 69 MiB "upgrade" settled the question: the higher-precision MTP weights reduced agreement, acceptance, and throughput.

Five xhigh artifacts from the vanished model

Synthetic benchmarks only screen portion of the system, truthful I gave the last Qwen setup 10 one-shot browser tasks done Cursor Agent Local. "One shot" intends the supplier received the punctual erstwhile successful an quiet directory. There was nary quality feedback, 2nd attempt, aliases repair walk aft the answer. It could plan, constitute files, inspect them, and trial its ain activity during that azygous run. The consequence truthful measures the exemplary and the harness together.

Below are 5 untouched artifacts, ordered arsenic I would show them to different engineer.