Breaking the 1.58-bit Barrier for Ternary LLMs

[Submitted connected 14 Sep 2026]

View PDF HTML (experimental)

Abstract:Ternary Large Language Models (LLM) shop each weight arsenic 1 of 3 symbols $\{-1,0,+1\}$, truthful the costs of a ternary exemplary is conventionally referenced to the information-theoretic $\log_2 3 \approx 1.585$ bits per weight. The prevailing deployment format packs 5 ternary weights into 1 byte (five-trit packing), and owed to the power-of-two group sizes utilized successful believe this rounds up to $1.625$ bits per weight. This effective retention bit-width treats the 3 symbols $\{-1,0,+1\}$ arsenic equiprobable. We measurement the existent awesome distribution of 29 ternary LLM models and find that zeros relationship for up to $51.5\%$ of each weights. Motivated by this finding, we present BITCOS, a elemental distribution-adaptive layout comprised of a dense beingness bitmap positive a compacted motion vector, and costs $2 - z$ bits per weight constituent fixed a zero density $z$ successful the model's weights. BITCOS stores weights much compactly than the five-trit packing successful 26 of the 29 tested models, and reaches $1.485$ bits per weight connected the sparsest of them. BITCOS is amenable to businesslike unpacking connected modern processors and GPUs, and we coming optimized unpacking sequences for AVX-512, AVX2 and Intel Xe2 GPUs. Measured against accumulation state-of-the-art ternary matrix-vector multiplication kernels, astatine the zero densities real-world ternary models exhibit, the realized summation pinch our projected layout is up to $1.28\times$. Finally, we exemplify end-to-end LLM conclusion results connected 5 different platforms (client and server CPUs, integrated and discrete Xe2 GPUs) wherever decode throughput improves by up to $1.18\times$ connected CPUs and $1.27\times$ connected GPUs.