Shivam Duggal1, Zongze Wu2, William T. Freeman1, Antonio Torralba1, Richard Zhang2, Eli Shechtman2
1MIT2Adobe
Paper Code (coming soon)
Abstract
Language models typically addition per-token computation through deeper or looped layers and latent reasoning steps, during richer residual connections enhance data stream throughout layers. We alternatively addition computational capacity: the figure of energetic computational states a example maintains. We current the Lattice of Latents, a set of per-layer horizontal latent streams that run recurrently alongside time, alongside the vertical residual stream that flows through depth. Like hyper-connections, mHC, and notice or value residuals, the Lattice enhances the residual stream, but fairly than adding or rerouting residual pathways through depth, it provides companion computational states that persist throughout tokens. Unlike hybrid Transformer–SSM architectures, whose recurrent states chiefly assist as inner recollection for sequence mixing, these latents immediately circumstance the vertical stream’s attention, KV-cache writes, MLP computation, and decoding. On Transformers trained on 20B FineWeb-Edu tokens, a 406M-parameter Lattice example lowers validation perplexity from 15.41 to 14.40 complete a parameter-matched Transformer. These gains broaden throughout architectural variants. The Lattice composes alongside previous residual-stream enhancements: alongside hyper-connections and input residuals, it lowers validation perplexity from 15.41 (base Transformer) and 14.22 (Transformer alongside identical enhancements) to 13.61. On Gated DeltaNet, Mamba-2, Triadic GDN, and GDN–Mamba-2 hybrids, it improves perplexity at all tested native province size, alongside Lattice-GDN & Lattice-Triadic-GDN variants outperforming clean baselines alongside 4× larger native states. These results acknowledge persistent per-layer province as a complementary axis for scaling language-model computation.
The Lattice of Latents
Vertical residual streams stream through depth; horizontal latents persist throughout tokens. Switch between training and inference, and difference models.
Click the fig to pause. Input injection can be switched on or off at any moment; the animation continues from the identical point.
Results
Interactive versions of the chief results. Hover complete any bar or item for details.
Language modeling following 20B-token pretraining
LLLM (dmodel, dr) has a vertical residual-stream width dmodel and horizontal-latent width dr.
Transformer LLLM (single pass) LLLM + Jacobi refiner (J3)
GFLOP/token: floating-point operations per token, excluding notice scores (which add lone ≈ 0.1 GFLOP/token) and the embedding lookup.
J3 runs three sweeps; its sequential rollout uses a sole continue at conclusion (click Show complete table for results).
Scaling pretraining tokens: 10B → 20B → 30B
FineWeb-Edu validation perplexity as training continues. Circle area is proportional to indicator count.
Both LLLMs use lone 22 layers vs. 40 for the Transformer 40L, so alongside the identical notice width (1024) they keep a 45% smaller KV cache.
Lattice of Latents vs. previous residual-stream enhancements
400M models, 20B FineWeb-Edu tokens. Lattice models narrow the vertical width from 1024 to 932 to remain iso-parameter. The Transformer and its sole enhancements appear first, afterward the Lattice and its sole enhancements, afterward two-enhancement combinations.
Transformer variants Lattice variants (ours)
HC: hyper-connections; mHC: manifold-constrained HC; value: value residual (ResFormer; learned weights for the sole +value runs, fixed 0.5 in x0 + value); x0: nanochat-style input residual.
Deconstructing the Lattice
A difficult regime: the vertical residual stream is squeezed to 256 dimensions. Press perform to add the horizontal-update enhancements one at a period and observe them regain most of the gap to a full-width Transformer. 10B-token pretraining.
+ adds a component; ↻ replaces one. Parameter counts are shown under all bar.
Recurrent backbones
Pure vs. Lattice recurrent models (∼405M parameters, 10B FineWeb-Edu tokens) throughout native-state sizes per mixer. Each Lattice example is iso-parameter alongside its clean counterpart. Bars appear from highest to lowest perplexity, all at its state-size position.
Pure backbone + Lattice (ours)
Native-state size counts recurrent-memory entries per mixer (1K = 1024 scalars), excluding the horizontal latents and convolution states. Lattice models use input injection + vstream.