Neutrino-1 8B

A dense decoder-only transformer. Grouped-query attraction holds the KV cache astatine a 4th of the query width, 144 KiB per token astatine fp16, truthful a 4k-token convention costs 0.60 GB of cache beside the 3.88 GB of weights.

Where the bytes live

Only the transformer linears transportation the coded format. The 2 embedding tensors enactment int8 because their rows are publication 1 token astatine a time, not multiplied against the afloat activation stream, and the normalization weights are excessively mini to beryllium worthy coding. A 3rd of the record is vocabulary.

Inside the coded lane

Across the 6.95B coded weights, 62.63% beryllium astatine zero and the remainder splits 18.68% positive to 18.69% minus: sign-balanced to a hundredth of a constituent pinch nary constraint asking for it. The equilibrium is not azygous successful depth. The gross and down feed-forward projections spike to 70 to 72% zeros successful layers 1 done 3 while each 4 attraction projections clasp wrong astir 1 constituent of 62% astatine each depth: the early feed-forward blocks shed weights the web does not need, and attraction keeps a changeless codification density from furniture 0 to furniture 35.

One container, 3 doors. The download is simply a coded carrier of the container, not a compressed transcript of an fp16 model: it expands bit-exactly to the record each runtime executes, and that 1 record is what runs connected a datacenter GPU and connected a laptop alike.

Distribution surfaces

The merchandise artillery runs connected the shipped instrumentality pinch reasoning disabled, truthful each people beneath is the artifact you download and not a investigation checkpoint. Protocol rides each row: changeable count, grading mode, and point count.

Single-stream decode, the complaint that governs 1 punctual and 1 reply. Same artifact connected each row; the aboveground changes, the weights do not.

Neutrino-1 0.6B drafts a tally of tokens, the 8B scores the full tally successful 1 guardant pass, and the agreeing prefix is kept. A draught token is accepted only erstwhile it equals the 8B's ain argmax, truthful the output watercourse is the plain greedy stream: connected the shipping configuration, 27,648 consecutive tokens matched pinch zero divergences.

The speedup is draft-acceptance physics, truthful it is stated per punctual people complete the 396 tok/s plain rate. On counting prompts the 8B accepts the afloat six-token draught connected each pass, astir 7 tokens emitted per 8B forward; connected actual prompts acceptance holds astatine 96.5%. A move controller sizes each draught to the people it is decoding, which is why each people clears the plain rate.

What the pairing costs

Both containers are the aforesaid format and tally connected the aforesaid binaries, truthful the draught loads into the verifier's ain process pinch nary 2nd deployment and nary conversion step. Its 328 MB beryllium beside the 8B's 3.88 GB, and 4k tokens of shared discourse costs precisely 1 gibibyte of cache crossed the pair.

Drafting connected a laptop

The pairing is not a datacenter feature. On a 16 GB Apple M5 some models load into 1 MLX process nether a 6 GiB headdress and highest astatine 4.3 GiB together, pinch the draught accounting for 0.53 GiB of it. The exactness gross returns 6 of 6 prompts token-identical pinch drafting connected and off, and connected actual prompts the drafted complaint is 25.71 tok/s against 22.00 plain astatine an acceptance of 0.744.