The Kimi K3 architecture fig for yesterday’s large open-weight exemplary release, on pinch immoderate observations and thoughts.
-
Yes, it looks comparatively complicated, but it’s fundamentally a scaled-up accumulation type of their Kimi Linear model they released past twelvemonth (scaled up from 48B -> 2.8T; K3 is by acold the biggest open-weight exemplary correct now)
-
The 1 caller constituent compared to Kimi Linear is the LatentMoE. I omitted it successful the fig beneath since it’s already very crowded, but that’s fundamentally the aforesaid LatentMoE arsenic successful Nemotron 3 Ultra (you tin find it successful my LLM Architecture Gallery if you are curious). The thought present is to compress (down-project) ample linear layers akin to multi-head latent attention.
-
Kimi K3’s wide inclination (similar to Nemotron 3, DeepSeek V4, and others) is besides towards amended conclusion efficiency. That is, location are galore components that switch existing components pinch efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attraction -> multi-head latent attraction and Kimi Delta Attention. (I besides person short tutorials and write-ups successful my gallery if you are funny astir further details).
-
The 1 constituent alteration that is not an ratio tweak is attention residuals. Like DeepSeek V4 improved the residual way pinch mHC (manifold-constrained Hyper-Connections), attraction residuals are a measurement to amended the residual path, but it useful a spot differently. I.e., mHC made the residual way wider. Attention residuals (also already portion of Kimi Linear) link the residuals crossed layers; the relationship itself uses an attraction people for an important/contribution weight. According to the report, it improves the validation nonaccomplishment and downstream capacity (a bit) consistently and adds astir 4% successful training costs and 2% successful conclusion cost.
-
Interestingly, Kimi K3 sewage free of each RoPE layers and uses NoPE (No Positional Embeddings) everyplace instead. (Again, this is inherited from Kimi Linear). In different architectures, the caller inclination was towards RoPE successful section attraction layers (like sliding model attention) and NoPE successful the world layers. There were a fewer architectures that only utilized NoPE everywhere, but this is the first frontier-level 1 arsenic acold arsenic I know.
-
Kimi K3 now besides has autochthonal multimodal support, which is great!
There are respective different absorbing training tidbits successful the method report, but that’s it from the architecture beforehand truthful far. A really awesome merchandise overall.
Source: website type of my Substack note.
Read Next
A Few Notable Open-Weight Models This Week Short statement connected the architectures of six caller open-weight models, including Nanbeige 4.2, Laguna S 2.1, Motif-3-Beta, Solar Open 2, Antares 1B, and BTL-3. 
Correction for Listing 6.5 successful Build a Reasoning Model From Scratch Short correction statement for the random seed successful Listing 6.5 connected page 198 of Build a Reasoning Model From Scratch. 
Inkling: A New Open-Weight 975B MoE pinch a Few Surprises Short statement connected Thinking Machines Lab's 975B Inkling model, including benchmarks, sparse MoE design, short convolutions, RMSNorm, and position bias.
English (US) · 
Indonesian (ID) ·