August 18, 2026
Inference is the bottleneck of the supplier era. Agents read, plan, and call tools, often for hours aliases days. They devour tokens astatine a complaint chat never approached. Every 1 of those tokens takes a afloat guardant walk complete the model. At Inco AI, we are building the conclusion stack scaled to the token economics of tomorrow. This station is simply a sneak peek.
Our squad released DFlash successful January; it now runs successful SGLang, vLLM, TensorRT-LLM, and llama.cpp. NVIDIA measured up to 15× throughput with it connected Blackwell GPUs; Google reported 3× much tokens per second on TPUs; CoreWeave's accumulation Kimi K2.7 Code endpoint, the fastest for that exemplary connected Artificial Analysis, runs DFlash by default. The ecosystem now builds connected it: NVIDIA, Red Hat, and Modal person each published DFlash drafters; Meta (Muse Glimmer), Poolside (Laguna), Xiaomi (MiMo-V2.5-Pro), and NVIDIA (Nemotron 3.5 Lightning) ship charismatic drafters pinch their ain models. On Hugging Face, DFlash models have been downloaded more than 3.5 cardinal times (as of August 2026).
Speculative decoding is simply a halfway portion of the modern inference stack.1 A mini draught exemplary guesses a artifact of tokens, and the target exemplary verifies the full artifact successful 1 guardant pass. Good guesses turn 1 walk into respective tokens; bad ones conscionable get thrown away. For years, though, the draught itself stayed autoregressive: one token astatine a time. DFlash made it one-pass too: the full block, every position, predicted in parallel.
DFlash 2 pushes parallel drafting 1 measurement further: over 20% much output from every verification pass, for astir 1% added rhythm latency, pinch the output provably unchanged. Across benchmarks the summation runs 16–25%. With the Qwen3.8-27B drafter released today, SGLang serves astatine 2.7–3.4× the throughput of autoregressive decoding astatine batch size 1. Predicting every position independently leaves headroom successful 2 places: choosing the right tokens and holding accuracy to the extremity of the block. DFlash 2 recovers both without giving up the one-pass design.
Run It Now
DFlash 2 already runs successful the mainstream conclusion engines:
The Right Tokens Are Already There
DFlash predicts each position independently, successful parallel. Each prime is plausible connected its own. Yet thing makes them fresh together, and an incoherent artifact is trim short astatine verification. Recent methods specified arsenic Domino and DSpark bargain coherence pinch sequential heads that rewrite each position's full-vocabulary distribution. But is that costly autoregressive correction really necessary?
No. The grounds is already successful DFlash's ain campaigner lists. Take the first position: DFlash's apical prime is correct 85.4% of the time, but the right token is successful its apical 16 candidates 99.5% of the time. Even erstwhile the apical pick is wrong, the correct token is usually connected the list.
| Recall@1 | 85.4% | 80.3% | 79.4% | 78.3% | 77.5% | 75.9% | 72.9% | 4.27 |
| Recall@16 | 99.5% | 97.3% | 94.8% | 92.6% | 90.8% | 89.4% | 87.8% | 6.79 |
An oracle that ever picks the correct campaigner from the apical 16 would lift the acceptance magnitude from 4.27 to 6.79. That spread is axenic selection headroom. We conscionable request to prime the correct way done the candidates.
Diffusionisgood⟨mask⟩⟨mask⟩⟨mask⟩Independent Top-1 Picksfor✗same word, twiceposition 1decodingspeculativeslow⋮position 2decodingthinkingmodels⋮position 3⟨eos⟩again…⋮all adjacent pairs scored astatine erstwhile → 1 way keptaccepted outputfor
target-decoded tokenmask tokenaccepted draftselected path
A Lightweight Path Selector
Coherence is mostly local: a candidate's fresh depends chiefly connected the token just earlier it, truthful scoring neighboring pairs should beryllium enough. DFlash 2 keeps the apical 16 candidates astatine each position and scores each adjacent pair: for predecessor aa and existent campaigner bb,
St(a,b)=Ut(b)+⟨A(a)⊙H(ht),B(b)⟩.S_t(a,b)=U_t(b)+\langle A(a)\odot H(h_t),B(b)\rangle.The people has 2 parts. The first, Ut(b)U_t(b), is DFlash's ain logit: how much the drafter already liked bb connected its own. The 2nd asks really well bb follows aa: AA and BB springiness each token a compact 256-dimensional embedding, and the 2 embeddings are matched nether a context gross H(ht)H(h_t) that decides which parts of the lucifer count. In essence, this is simply a low-rank bilinear attraction complete adjacent candidates.
Scoring stays afloat parallel. Every adjacent brace astatine each position is scored successful 1 shot, pinch nary other backbone aliases LM-head pass. The only sequential activity is the last locomotion complete precomputed scores: starting from the past verified token, greedy follows the champion successor astatine each step, sampling draws from the aforesaid scores, and rejection sampling restores the exact target distribution.
| DFlash | — | — | 4.27 | 3.78 |
| + DSpark correction | +77.8M | +9.6% | 4.49 | 4.08 |
| + way action (ours) | +2.0M | +0.6% | 4.61 | 4.25 |
The selector improves DFlash by 0.34 tokens astatine T=0T=0 and 0.47 at T=1T=1. It thumps the DSpark correction successful some settings pinch astir 40× fewer parameters and 16× little latency overhead. Choosing is cheaper than predicting. And location is still room: the oracle reaches 6.79. Pairwise scoring is the simplest selector we could deliberation of, and we judge there is plentifulness to explore.
Suffix Decay Is a Local Problem
We besides noticed both callback rows above diminution toward the extremity of the block. Even the oracle decays: pinch cleanable selection, accuracy still falls from 99.5% astatine the first position to 87.8% by the last. No selector tin fix that, because the candidates themselves are moving out. We telephone this suffix decay, and it is simply a backbone problem.
One fishy is capacity: a five-layer backbone whitethorn beryllium excessively mini to preserve limitations crossed the block. If that is right, extent should help astir astatine later positions. And it does! 3-, 5-, and 15-layer DFlash models are almost identical astatine the first position, and instrumentality isolated down the block. But extent is indiscriminate: ten extra attraction blocks adhd capacity everywhere, moreover astatine the early positions that had small near to gain, and erase overmuch of the efficiency that makes DFlash attractive.
| 85.21% | 79.26% | 77.18% | 75.75% | 73.96% | 70.4% | 64.97% |
| 85.39% | 80.31% | 79.39% | 78.27% | 77.39% | 76.03% | 72.86% |
| 86.42% | 81.61% | 80.68% | 80.34% | 80.59% | 79.66% | 78.73% |
| 85.83% | 80.94% | 79.98% | 79.68% | 79.73% | 79.43% | 77.61% |
We want a targeted fix, and DFlash's attraction shows where. It has two jobs: publication the discourse earlier the block, and exemplary the dependencies inside. But it spends little and little connected the second: the block's share of attraction falls from 30% successful Layer 1 to 8% successful Layer 5, and what remains concentrates successful a shrinking fistful of heads. So we divided the jobs: a dedicated module takes the within-block work, and attraction keeps reading the context.
1
4
8
12
16
20
24
28
32
Layer 1
Layer 2
Layer 3
Layer 4
Layer 5
Attention head
0%90% within-block mass
| 17.6% | 2.9% | 41.4% | 50.8% | 29.2% | 50.5% | 44.6% | 5.9% | 44.5% | 11.3% | 17.9% | 36.7% | 0.0% | 14.2% | 0.1% | 0.0% | 13.3% | 1.5% | 18.7% | 7.6% | 45.0% | 33.5% | 53.1% | 42.2% | 64.3% | 60.1% | 32.8% | 47.7% | 49.6% | 57.0% | 26.0% | 52.9% |
| 20.8% | 26.4% | 39.6% | 18.9% | 8.9% | 22.6% | 13.1% | 32.1% | 22.9% | 25.1% | 24.2% | 28.6% | 36.6% | 26.1% | 41.0% | 36.1% | 17.8% | 25.5% | 25.7% | 25.6% | 4.3% | 21.8% | 23.3% | 22.1% | 15.6% | 70.9% | 58.0% | 2.7% | 28.3% | 38.5% | 20.3% | 33.5% |
| 1.8% | 11.0% | 9.5% | 5.2% | 34.8% | 8.4% | 12.1% | 14.4% | 11.8% | 22.0% | 8.8% | 3.7% | 4.9% | 10.6% | 17.7% | 52.0% | 4.4% | 19.0% | 13.1% | 9.9% | 61.3% | 76.1% | 47.0% | 60.3% | 1.4% | 8.9% | 6.0% | 64.1% | 9.4% | 3.3% | 8.3% | 8.3% |
| 0.4% | 37.7% | 28.3% | 85.5% | 0.3% | 1.5% | 0.4% | 0.5% | 1.2% | 12.5% | 36.6% | 1.2% | 1.7% | 0.6% | 2.5% | 1.3% | 7.2% | 3.1% | 48.9% | 3.8% | 3.2% | 1.0% | 23.8% | 1.0% | 0.1% | 0.1% | 0.2% | 0.3% | 2.8% | 6.7% | 12.9% | 12.3% |
| 1.5% | 0.2% | 0.6% | 0.1% | 60.2% | 76.0% | 0.9% | 0.0% | 0.2% | 12.3% | 32.3% | 0.1% | 15.8% | 0.5% | 0.5% | 0.5% | 0.2% | 0.1% | 0.6% | 0.2% | 0.3% | 28.1% | 0.2% | 1.3% | 0.1% | 0.1% | 0.2% | 29.9% | 0.1% | 0.1% | 0.1% | 1.2% |
A Lightweight Local Convolution
The within-block activity is short-range to statesman with: a artifact spans only 4 to 16 tokens, and the tightest limitations beryllium betwixt neighbors. The natural usability is simply a short convolution: 2 taps, 1 connected the current position and 1 reaching 1 position back, pinch weights that accommodate to the content. Following Canon Layers, Dynamic Short Convolutions, and Convolution for Large Language Models, we insert this two-tap move depthwise convolution earlier and aft each attention and feed-forward sublayer:
Convk(x)t=kt,0⊙xt+kt,1⊙xt−1.\operatorname{Conv}_{k}(x)_t =k_{t,0}\odot x_t+k_{t,1}\odot x_{t-1}.Each coefficient combines a learned guidelines kernel pinch a mini correction computed from the existent hidden state; each 16 channels stock one correction. The first position sounds the past verified token's representation, and each later position sounds its predecessor's. Information crosses the artifact while each positions still compute in parallel.
two-tap convlast verified tokendraft positions
The convolution is block-local and stateless, truthful it drops into DFlash without changing attention, the LM head, aliases verification.
With only 16.5M added parameters (3%), five-layer DFlash with convolution comes adjacent to 15-layer DFlash, substantially reducing suffix decay. The convolutions adhd 0.7% to draft–verify cycle latency; 10 much Transformer layers adhd 15.2%. Average within-block attention crossed Layers 4 and 5 besides falls from 9.4% to 0.5%, consistent pinch the convolution absorbing the section activity while attention goes backmost to reference the context. A kernel reaching 1 position back recovers astir of what 10 other layers buy: suffix decay is mostly a local problem.
Putting It Together
So far, the selector and the convolution person been measured separately; the afloat comparison below puts them together. We trained the DFlash and DSpark drafters ourselves nether matched setups, while MTP ships with the model.
Qwen3.5-4B
| GSM8K | 4.78 | 4.99 | 5.69 | 6.20 |
| MATH-500 | 5.04 | 5.42 | 6.20 | 6.76 |
| HumanEval | 4.84 | 5.43 | 5.80 | 6.28 |
| MBPP | 4.16 | 4.49 | 4.96 | 5.41 |
| MT-Bench | 3.90 | 4.26 | 4.77 | 5.20 |
| Mean | 4.54 | 4.92 | 5.49 | 5.97 |
DFlash 2 leads connected each benchmark. Averaged crossed them, it gains 1.05 tokens complete DFlash (21%) and 0.48 complete DSpark. The upgrade stays cheap: the selector and the convolution together adhd only 1.3% to the five-layer DFlash draft–verify rhythm latency.
On MATH-500, the summation is visible position by position: DFlash 2 holds dependable adjacent 86% to the past position, and each baseline ends the artifact 6 to 9 points beneath it.
| 84.57% | 80.23% | 79% | 78.42% | 78.63% | 78.17% | 77.36% | 77.74% | 77.91% | 76.96% | 78.06% | 77.4% | 77.49% | 77.48% | 77.85% |
| 88.35% | 77.7% | 77.8% | 79.45% | 80.3% | 81.12% | 81.22% | 81.07% | 81.29% | 80.28% | 80.64% | 80.29% | 79.56% | 78.77% | 77.48% |
| 87.24% | 84.59% | 83.79% | 83.63% | 83.6% | 83.27% | 82.97% | 82.54% | 82.21% | 82.39% | 81.58% | 80.7% | 81.35% | 80.57% | 79.86% |
| 88.3% | 85.3% | 84.98% | 84.88% | 85.41% | 85.3% | 85.36% | 85.13% | 85.95% | 85.99% | 86.41% | 86.46% | 86.43% | 86.02% | 86.48% |
Two Drafters, Out Today
We are releasing 2 DFlash 2 drafters today: one for Qwen3.8-27B and one for Meta's Muse Glimmer. For Qwen3.8-27B, we comparison against the model's autochthonal MTP way and a community DSpark drafter.
Qwen3.8-27B
| GSM8K | 5.02 | 4.36 | 5.46 |
| MATH-500 | 4.72 | 3.92 | 5.28 |
| HumanEval | 3.91 | 3.30 | 4.39 |
| MBPP | 3.99 | 3.51 | 4.79 |
| MT-Bench | 3.74 | 3.01 | 4.10 |
| Mean | 4.28 | 3.62 | 4.80 |
For Meta's Muse Glimmer, we comparison against the charismatic DFlash drafter shipped pinch the exemplary and a community DSpark drafter.
Muse Glimmer
| GSM8K | 5.43 | 5.45 | 6.57 |
| MATH-500 | 5.39 | 5.01 | 6.56 |
| HumanEval | 4.11 | 4.33 | 5.66 |
| MBPP | 3.74 | 4.02 | 5.30 |
| MT-Bench | 3.52 | 3.59 | 4.42 |
| Mean | 4.44 | 4.48 | 5.70 |
The margins are wide: connected some models, DFlash 2 averages much than a afloat token up of DSpark. It besides thumps each model's official drafter, MTP connected Qwen3.8-27B and DFlash connected Muse Glimmer. That translates into 2.7–3.4× the throughput of autoregressive decoding on Qwen3.8-27B, and 3.1–4.6× connected Muse Glimmer. The model cards break the speedups down by task and concurrency.
The Bottom Line
An supplier writes successful an day what a chatbot writes successful a month, and decoding sits nether each 1 of those tokens. DFlash 2 decodes at close to 3× the velocity of autoregressive decoding, astir a 3rd of the compute per token, pinch the aforesaid output.
In 7 months, DFlash went from our insubstantial to an manufacture standard, with much than 3.5 cardinal downloads. Inside the aforesaid design, DFlash 2 decodes 1 much afloat token per pass, for free. That is only 1 constituent of the serving stack. Inference is obscurity adjacent its floor.
At Inco AI, we are building an end-to-end serving stack to support pushing that floor lower. DFlash 2 is the first piece. Two drafters are retired today on Hugging Face.
If you service agents astatine standard and want to measure DFlash 2 successful your stack, or want a drafter for a exemplary you run, including your ain fine-tunes, write to us: [email protected].
We are besides hiring. If you want to thief build this stack, scope retired to us.
Connect the candidates. Keep drafting parallel.
Get updates
One email erstwhile we vessel thing new.
We will ne'er stock your email address.
Citation
Please mention this station as:
English (US) · 
Indonesian (ID) · 
