DeepSeek-v4.1 Flash: Pushing the Limits of KV Cache Compression

TL;DR

When DeepSeek-V4.1 Flash was released, I thought it mightiness conscionable beryllium a post-training loop version... but aft utilizing it for a while, I recovered it reached astir 420 Tokens/s successful speed, and past Cui said each DeepSeek-V4 Pro models would beryllium taken offline... abruptly I felt this was nary mini matter... until the Technical Report was afloat released, only past did I recognize it should beryllium called DeepSeek-V5 Flash...

As the insubstantial title states, the intent of DeepSeek-V4.1 Flash is to push KVCache compression to the extreme. The main logic is that Long-horizon Agent Workflows origin the Context to turn longer and longer, while various instrumentality calls bring dense prefill computation pressure. The retention unit of KVCache successful HBM and outer SSD is very high, each of which are reasons that make Scaling impossible. Therefore, a bid of optimizations were made connected the exemplary architecture, particularly successful the compression of KVCache and the computation optimization of Prefill.

• Prefill computation optimization: Drawing connected YOCO, the full exemplary has 40 layers, and only 20 layers are needed during Prefill. Therefore, the Prefill activated parameters are only 8B, and the Decode activated parameters are 16B
• KVCache compression: Engineering-wise, KVCache compression is divided into respective dimensions: caput count compression akin to GQA, past block-based compression for illustration CSA, and the cross-layer compression of CSA2 successful this paper. At the aforesaid time, the indexer computation of Sparse Attention is besides optimized. Finally, location are immoderate numerical precision optimizations, for illustration DS41F adopts FP4 KVCache.

Finally, nether the premise of maintaining high-quality task completion by the model, KVCache is further compressed by 4x:

In addition, the original penning of the insubstantial is somewhat complex, particularly the explanation of CED. In fact, if we redraw a sketch centered connected KVCache and mixed pinch the position of machine architecture, it seems to go clear each astatine once. It tin beryllium seen arsenic a benignant of Recursive Transformer architecture, a measurement of modifying Q and reusing KV during the recursive process.

Regarding the Recursive Transformer architecture, you tin mention to 《On the Future Transformer: Loops Are Not What You Need》. Next, we will behaviour a elaborate mentation and study according to the section building of the method report. This article is the first successful this series, analyzing the exemplary architecture successful detail, and the much captious contented is successful Chapter 3.

1.1 Why KVCache compression is needed

First, the study states that successful caller years Long-horizon Agents person made ultra-long-context processing an progressively important exemplary workload. Supporting this type of workload not only requires businesslike processing of agelong sequences, but besides requires persistent storage, reuse, and transportation of ample KVCache. Therefore, KVCache guidance has go a basal capacity of exemplary deployment, while besides bringing important challenges successful computation, storage, and communication.

Then it goes connected to present the DeepSeek-V4 architecture, which processes by combining a Sparse Attention that afloat covers the discourse pinch a Sliding Window Attention (SWA) that covers the section window. Although advances related to Sparse Attention person importantly reduced the computational costs of agelong series processing, persistent retention and information activity person gradually go much salient bottlenecks. In agelong contexts, the usage of the Global KV Cache will dominate, and being persisted for prefix reuse, it will heavy inhabit Host representation capacity and SSD capacity, and will besides spot precocious demands connected the interconnect bandwidth for KVCache movement. These limit work throughput, summation deployment cost, and yet inhibit the deployment and promotion of agents toward longer task spans and broader exertion scenarios.

Therefore, further reducing the key-value cache footprint is important for alleviating retention and connection bottlenecks and reducing long-context serving costs. DeepSeek-V4.1-Flash is simply a multimodal mixture-of-experts exemplary designed for much fierce KVCache compression. DeepSeek-V4.1-Flash has a parameter standard of 552B, natively supports multimodal input, and supports contexts of up to 1 cardinal tokens. It adopts a Causal Encoder-Decoder (CED) architecture, successful which the Decoder's Global KVCache is obtained by projecting the Encoder's last hidden states. This creation makes the exemplary activate 8B parameters per token during the Prefill shape and 16B parameters during the Decoding stage, which is particularly cost-effective for input-dominated Agent scenarios. Although DeepSeek-V4.1-Flash is importantly larger than DeepSeek-V4-Flash, astatine the aforesaid series length, its required runtime KVCache retention is only astir 1/4 of the latter, and its persistent KVCache retention is only astir 1/8 of the latter. In addition, the wide capacity of DeepSeek-V4.1-Flash is superior to DeepSeek-V4-Flash.

These compressions for KVCache chiefly travel from the associated optimization of exemplary architecture, cache precision, and deployment strategy. For DSv4, it is simply a exemplary pinch SWA arsenic the backbone and enhanced by world compressed attraction (CSA/HCA). Based connected this perspective, the DeepSeek squad carried retired a bid of optimizations. First, it is worthy noting that they abandoned the block-based high-compression-ratio building for illustration HCA, and alternatively carried retired much optimizations connected CSA, forming CSA2. The main optimizations compress the KV Cache from 3 dimensions:

• In the transmission dimension, a 512-dimensional latent vector is utilized to stock the practice of the keys and values required by each attraction head;
• In the series dimension, the Encoder merges 2 adjacent positions into 1 cache introduction done channel-wise learned weights, while the Decoder retains per-position entries;
• In the furniture dimension, aggregate layers stock the aforesaid world KV, and the full web retains only 3 copies of Encoder cache and 1 transcript of Decoder cache.

Combined pinch FP4 quantization, the retention maturation of the world main KV and the Indexer is astir 890 bytes per token.

1.2 Overview of exemplary architecture

The wide exemplary architecture is arsenic follows:

The insubstantial reports that the backbone parameters are astir , the Engram parameters are astir , and the activated parameters per token for prefill and decode are astir and respectively. The halfway is to usage CED to trim long-context Prefill computation, usage CSA2 to trim attraction and KV cache overhead, and past harvester Engram conditional representation pinch DSpark speculative decoding.

The exemplary has layers successful total, hidden magnitude , and vocabulary size . Each furniture contains attraction and MoE, organized done mHC residual connections. The full attraction system is divided into 2 modules, Encoder and Decoder, forming a Causal Encoder-Decoder (CED) architecture. The cardinal of CED is that the Decoder's world KV comes from the Encoder's extremity representation, and consequent Decoder layers stock these KVs. Therefore, astir positions of a agelong punctual only request to walk done the first 20 layers,

The cardinal CSA2 among them adopts a system of local sliding model + world sparse retrieval + cross-layer KVCache reuse. The sliding model size is 128, attraction uses Q heads, sharing a -dimensional KV latent, of which RoPE is -dimensional and NoPE is -dimensional. Q uses a low-rank projection of rank , and the output projection is divided into groups, each of rank . The applicable parameters of the full exemplary are arsenic follows:

Among them:

• mHC: Each token maintains residual streams of dimensions. The residual mixing matrix is constrained to an astir doubly stochastic matrix done Sinkhorn iterations. The cardinal of Single-Pass is to usage the input mixing coefficients produced by the erstwhile sub-layer, releasing the dependency of the existent coefficient computation, facilitating kernel fusion and reducing representation read/write.
• DSpark: An further SWA-128 draught blocks, each furniture adopts a small-scale MoE pinch Top-3 retired of routed experts. It sounds the mean of the 4 residual streams astatine the introduction of backbone web layers , computes draught positions successful parallel astatine once, cooperates pinch a Markov caput to exemplary dependencies, and a assurance caput assists successful deciding the verification length.
• Vision branch: A ViT pinch layers and hidden magnitude , spot size . Features spell done pixel-unshuffle, reducing the number of tokens to of the original, past mapped to dimensions by an MLP and inserted into the matter sequence, pinch a maximum of ocular tokens per azygous image.

Regarding CED and CSA2, we will present them successful item successful Chapter 2. Finally, arsenic the Context grows, the required computation of DeepSeek V4.1 Flash grows almost linearly wrong the 1M range, and the computation overhead is acold little than that of erstwhile generations of models

2.1 Multimodal architecture

The ocular way of DeepSeek-V4.1-Flash tin beryllium summarized as: complete ocular encoding connected a finer image spot grid, rearrange adjacent features into less wide vectors, and past task them into the input abstraction of the connection backbone. Among them:

• vision_patch_size=14: is the spot size of the original image
• vision_downsample_ratio=3 : is the merging scope of the ViT output characteristic grid

The full processing travel is shown successful the fig below. The ViT first completes intra-image relationship connected high-resolution features, and past feeds them into the LLM done spatial rearrangement and compression projection.

The supra fig takes a 1008 x 1008 pixel quadrate image aft preprocessing and padding arsenic an example:

Image preprocessing: It should beryllium noted that it does not execute matter nickname for illustration OCR. After the record is loaded, it is straight decoded via load_image and converted to RGB. Then it checks the minimum pixel area. If the original image area is beneath , it scales up the target size proportionally. Subsequently, it aligns the 2 edges upward to a aggregate of vision_patch_size=14, and fills the aligned blank areas pinch RGB gray. And statement that it checks whether the expanded image span exceeds the vision_max_n_token=1024 budget. When exceeding the budget, it re-computes a smaller target canvas based connected the facet ratio.

Let the pixel size aft preprocessing beryllium , the spot broadside magnitude beryllium , and the spatial merge facet beryllium . The codification first makes the pixel broadside magnitude an integer aggregate of :

The Aligner allows the spot grid to not beryllium divisible by 3, because it pads zeros connected the correct and bottommost of the characteristic grid:

The merged ocular grid and its contented token count are:

And what the section preprocessing usability really budgets is:

Among them, 1 IMAGE_NEW_LINE per row, positive IMAGE_START and IMAGE_END.

For the paper's "supporting input resolutions up to astir 1344 ×1344 pixels", it is fundamentally constrained by vision_max_n_token=1024. For example, according to the paper's , , and past the token count aft consequent downsampling is 1024, but considering that adding tags wrong the span will transcend the vision_max_n_token=1024 budget, this image will beryllium scaled to , i.e., tokens, positive 31 NL tags and 2 START/END tags, for a full of 994 tokens.

And communal surface resolutions specified arsenic will beryllium scaled to , for a full of 968 tokens.

Then the image is normalized according to the RGB channels as:

Continuing pinch the image arsenic an example, the codification past splits the image into non-overlapping blocks. If the row-column coordinates of an image artifact are , and the intra-block coordinates are , the worth it takes retired is . It first traverses the columns wrong a row, past enters the adjacent row, obtaining image blocks pinch style . Splitting into blocks rewrites the spatial coordinates arsenic artifact numbers and intra-block coordinates.

Patch Embedding: All numbers of each block, past flatten each block, and each blocks stock the aforesaid linear furniture pinch bias. Finally, a matrix is obtained. Note that the insubstantial explains why the convolution needs to beryllium replaced by linear projection, the main logic being to guarantee compatibility pinch the Muon optimizer.

DeepSeek-ViT: Next, 32 layers of ViT processing are performed to springiness them intra-image context. Both the input and last output are . Each furniture first performs RMSNorm connected the existent features, past computes attraction and adds backmost the residual; subsequently normalizes again, executes the SwiGLU feed-forward network, and adds backmost the residual. After the 32 layers, location is 1 much RMSNorm astatine the extremity of the imagination tower.

Taking the attraction of 1 of the layers arsenic an example, a linear furniture first generates Q, K, V from the normalized features, pinch a full output width of . The 3 are respectively organized into , that is, 16 heads, each caput 64-dimensional. Before computing scores, 2D RoPE rotates Q and K according to the original row-column coordinates, and does not rotate V. Attention solves cross-position communication, while the consequent SwiGLU chiefly does transmission translator wrong each position. It first projects to dimensions and splits into 2 branches, applies SiLU to the gating branch, multiplies it element-wise pinch the different branch, and past projects backmost to 1024 dimensions. In this way, 1 furniture simultaneously contains 2 kinds of processing: "fetching accusation from different positions" and "reorganizing features astatine the section position".

Spatial rearrangement: It loads 9 adjacent features into the aforesaid wide vector, and adopts downsampling. The circumstantial attack is arsenic follows:

2-Layer MLP: Used to make tokens for the LLM, aligning hidden_dim = 5120.

Finally, wrong the token span generated by the image, immoderate markers still request to beryllium supplemented, arsenic shown successful the fig below:

Then these tokens will beryllium sent to the backbone network. Here location is different optimization, multimodal auxiliary-loss-free load balancing for MoE. Image and matter tokens grounds different practice distributions, and whitethorn shape different master routing preferences successful MoE. Therefore, balancing their aggregated load whitethorn disguise the imbalance wrong each modality. To lick this problem, the DeepSeek squad maintains a group of per-expert correction biases for matter and image tokens respectively. During routing, each token uses the correction bias corresponding to its modality for master selection, while retaining the original routing people to weight the output of the selected experts. After each training measurement ends, these 2 sets of biases are independently updated according to their respective master loads. This creation balances master usage wrong each modality, helping unchangeable and businesslike multimodal training.

2.2 Causal Encoder-Decoder(CED)

2.2.1 Why is CED needed?

The substantive problem is that successful Agent workflows, predominant instrumentality calls will make a ample number of prefill requests, which causes dense computation overhead erstwhile the KV Cache misses.

To alleviate this prefill bottleneck, the authors propose a Causal Encoder-Decoder (CED) architecture inspired by YoCo . YoCo reduces prefill computation by letting the upper-half layers straight stock the KV Cache produced by the lower-half layers.

In position of actual implementation, YoCo separates the accumulation and depletion of humanities memory: the little half establishes memory, and the precocious half many times sounds memory, but nary longer generates the per-layer humanities KV that must beryllium saved for consequent inference.

Here fto america concisely grow connected the full architecture improvement process of YoCo. First, the attraction system of each furniture of a Decoder-Only exemplary causes each furniture to person KV computation during Prefill, which is the guidelines origin of debased efficiency.

The first intermediate solution is to usage the SWA algorithm, which done a fixed sliding window, each furniture produces KV, that is, the Efficient Self-Attn (ESA) mentioned successful YoCo's original paper. But if each layers usage SWA, the world attraction system will beryllium mislaid (note: successful the original paper, ESA tin optionally beryllium SWA aliases gRet...). Another solution is to divided the full exemplary successful depth, pinch the first half utilizing ESA to nutrient world KV, and the 2nd half utilizing modular Attention to publication the first half's KV, truthful that the world attraction system tin beryllium restored. But we request to find which furniture the 2nd half's KV comes from?

The last wished solution is that the 2nd half's KV is projected from the -th layer's hidden authorities via , which constitutes the YoCo architecture.

In fact, YoCo's solution has been tried by immoderate guidelines exemplary teams, usually described pinch a different sanction called KV-Mirror. Some teams person not made it public. The publically searchable 1 is Tencent's WeLM .

2.2.2 Concrete implementation of CED

On apical of the YoCo concept, CED introduces a bid of structural improvements, simultaneously expanding the wide capacity of the KV Cache and the computation extent of KV generation, yet successfully reducing prefill computation by astir half while maintaining capacity comparable to the baseline.

For world attention, CED regards the bottommost layers of the Transformer arsenic a Causal Encoder. For the upper-half layers (i.e., the Decoder, ), the KV entries are nary longer derived from their respective hidden states , but are straight projected from the -th layer's hidden authorities via layer-dependent projection weights ( and ):

Where and correspond the KV entries and the corresponding compression weights respectively. This creation allows CED to get the world KV Cache of the precocious layers astatine an highly debased computational cost, only needing to compute the first half of the layers during the prefill stage.

Specifically, CED divides the exemplary into 2 parts, Causal-Encoder and Decoder, pinch 20 layers each. By comparison, YoCo names the 2 parts Self-Decoder and Cross-Decoder, chiefly distinguishing the 2 parts by the accusation root of attention. The first half processes the series done businesslike self-attention, and the 2nd half uses the queries produced by its ain layers to publication the shared KV generated by the first half. DeepSeek changed to a different study angle: since the cardinal work of the first half is to make reusable discourse representations, it tin beryllium seen arsenic an Encoder; the precocious half uses these representations to proceed computing predictions, truthful it is called a Decoder. In YoCo's insubstantial viewpoint, it chiefly emphasizes that the Self-Decoder is astir "how to efficiently process sequences", and the Causal Encoder emphasizes "what it provides for consequent networks". The favoritism betwixt the Encoder/Decoder names is only a quality successful perspective.

In YoCo, the first half adopts ESA (gRet aliases SWA), while successful CED, the first 2 layers of the Causal-Encoder are besides SWA, and the consequent 18 layers adopt CSA2, which is simply a Sparse Attention pinch compression mixed pinch SWA. It calls the KV built by Sparse Attention Global KV, and calls the KV built by SWA Local KV, which are concatenated and past utilized pinch Q to compute the Attn-Score. We will grow connected this successful item successful a later conception and elaborate connected the CED architecture successful operation pinch CSA2.

For SWA, CED maintains regular per-layer computation successful each layers: the section KV of immoderate furniture is straight derived from the existent layer's hidden authorities , which really increases the computation extent of section KV generation. But maintaining per-layer computation requires an SWA replay process: computing the SWA KV Cache for the Decoder during the prefill shape requires further processing of tokens ( is the model size). For multi-turn interactions wherever each turn's punctual is comparatively short, this portion of the Decoder overhead cannot beryllium ignored. Fortunately, anterior activity (Chen et al., 2025) shows that the existent effective receptive section of SWA is acold smaller than the theoretical worth . Inspired by this observation, the authors present Decoder SWA Bounded Replay: only compute the SWA of the past tokens of the prefill punctual for the Decoder, thereby importantly reducing the computation cost.

1. Why does SWA really summation the computation extent of section KV generation?

First denote the Encoder extent arsenic , the Decoder extent arsenic , and the model size arsenic . CED's world KV comes from the Encoder bound representation:

Although the first half has galore layers, the world representation publication by the Decoder still comes from the projection of . Subsequent layers tin nutrient different queries, but this will not move the root of the humanities world KV into a deeper-layer representation. And each consequent furniture adds SWA-based section KV, which retains the way of "each furniture generates KV from its ain input representation". "Depth increase" refers to: the KV of these caller positions tin incorporate deeper-layer computation results, not conscionable different projections of the aforesaid Encoder bound representation.

2. Why is Bounded Replay needed?

But these Local KVs besides bring immoderate problems. For mean complete prefill, location is nary specified problem: each punctual words walk done each layers, and the section KV of each furniture is people generated pinch the guardant computation.

But CED wants astir humanities positions to extremity computing aft the Encoder ends. At this point, though the world KV tin already beryllium prepared, these humanities positions person not passed done the Decoder, truthful the Local KV of the heavy Decoder layers has not yet been generated. Decoder SWA Bounded Replay is to dress up for the Decoder Local KV missing aft CED ends prefill early, while avoiding the costs of restoring these KVs offsetting the use of early exit.

Since SWA only saves the section KV of the astir caller positions, it seems that replaying the past positions is enough. But the problem is: the representations of these positions successful the heavy layers still dangle connected earlier positions extracurricular the window. For example, each furniture model is 4. To compute the second-layer practice of position 100, the representations of first-layer positions 97 to 100 are needed. And first-layer position 97 needs to publication input positions 94 to 97. Therefore, the deeper the recovery, the much it needs to trace backmost forward. For a -layer Decoder, the humanities span for nonstop betterment is approximately:

If each instrumentality telephone only adds a fewer twelve words, but to reconstruct the cache, thousands () of humanities positions must walk done the Decoder, the computation saved by early exit of prefill whitethorn beryllium consumed by a ample magnitude of replay. The authors referred to the activity of . This is simply a insubstantial studying the receptive section of sparse attraction and cross-layer accusation propagation. In its Section 4.3 location is an research that evaluated the exemplary connected a passkey retrieval task. In its SWA research pinch astir a 2K window, the authors estimated that accusation has decayed rather weakly aft propagating done astir 6 layers.

Therefore DeepSeek adopts the Bounded Replay approach, and the domiciled of Bounded is to limit this overhead: only replay the past positions, nary longer continuously tracing backmost guardant for nonstop recovery. The costs is that the reconstructed Local KV is successful an approximate state. The opening of the replay conception lacks earlier section dependencies, and consequent deep-layer representations whitethorn alteration accordingly.

However, we statement that successful CSA2 what is truncated is the reconstruction scope of the Local SWA state, not the world memory. When replaying the past positions, the Decoder tin still publication the Global KV of longer history according to the causal and sparse attraction rules. Therefore, DeepSeek accepts this approximation and controls the effect done value information and post-training adaptation.

3. Why is this problem important?

Under agelong prompts, tail replay is conscionable a mini portion of activity extracurricular the Encoder's large-scale computation. But successful Agent multi-turn interactions, astir of the history whitethorn person already deed the world cache, and the genuinely recently added contented successful this move is very short. Let the recently added magnitude of this move beryllium . The Encoder main assemblage activity of the caller contented astir grows pinch , but the costs of restoring the Decoder section authorities does not automatically shrink pinch . For example, erstwhile , , the humanities span of nonstop betterment is astir positions. Even if only a fewer twelve positions are recently added successful this turn, it whitethorn still look backmost astatine a very agelong tail to reconstruct the section state, and past execute multi-layer computation.

Bounded replay limits this restoration activity to the astir caller positions passing done the Decoder, limiting the authorities restoration of each move to a fixed overhead.

Overall, for series magnitude , CED reduces the prefill complexity from to , really halving the full computation.

Why is the computation halved?

For the cold-start prefill of a agelong prompt, the Encoder processes each positions: , and the Decoder only processes tail positions:

So the main assemblage workload is:

Substituting , and comparing pinch the afloat computation :

When , the 2nd word is very small, and the ratio approaches . Taking DeepSeek V4.1 Flash's , arsenic an example, presume the punctual magnitude is . The afloat computation is astir token-layers; CED's Encoder positive bounded replay is astir , which is of the former.

2.3 CSA2

2.3.1 Why is CSA2 needed?

Serving agelong contexts requires simultaneously controlling KV cache retention and attraction computation. These costs tin beryllium reduced on 3 dimensions pinch a multiplicative relationship:

• Entry size dimension: For example, GQA reduces the number of KV heads, and MLA shares a mini latent practice crossed different heads
• Sequence dimension: wherever each tokens are compressed into 1 entry, specified arsenic CSA and HCA successful DeepSeek-V4;
• Layer dimension: wherever definite layers reuse the cache and action results of different layers, alternatively of retaining their ain cache and action results, aliases are wholly replaced by much businesslike layers.

In the activity connected the furniture dimension:

• proposes cross-layer attention, letting a information of attraction layers straight publication the KV produced by earlier layers, thereby avoiding redeeming an independent KV cache per layer.
• reuses Top-K indices crossed layers to trim Indexer computation
• computes sparse routing only erstwhile and shares it pinch each layers
• lets sparse layers reuse the KV cache of dense layers.

However, simply reusing indices does not prevention the main KV storage, sharing routing crossed the full web limits performance, and hybrid designs still clasp afloat attraction layers; much importantly, nary of these methods screen each 3 dimensions pinch a multiplicative relationship.

Analyzing carefully, though Cross-Layer Attention (CLA) tin stock KV crossed layers and trim independent KV copies, each furniture still computes its ain attention; KV retention is saved but computation is not; IndexCache shares Top-K indices betwixt immoderate layers, which tin trim the number of Indexer runs, but the main KV is still saved per layer, truthful though immoderate Indexer computation is saved, retention is not. And YOIO reuses the Decoder-Decoder building of YOCO, redeeming half of the KVCache, but aggregate layers of Sparse Attention successful the cross-decoder request to stock 1 group of selected TopK campaigner set, which has an effect connected the model's performance. HySparse uses Full Attention to nutrient KV, and though sparse layers tin reuse these KVs, the ratio of the afloat Attention computation will still impact performance.

Before introducing CSA2, we tin reappraisal successful item DeepSeek's optimizations of Attention computation complete the past fewer years. In MLA, associated low-rank compression is done connected the K/V representations of each heads, compressing connected the entry size dimension. Then Sparse Attention (DSA) was introduced successful DeepSeek V3.2, reducing the computation request of Attention. Then successful DeepSeek-V4, compression connected the sequence dimension was added done CSA (compression ratio 4:1) and HCA (compression ratio 128:1). And CSA2 further pushes compression toward the layer dimension.

CSA2 jointly utilizes these 3 dimensions: it shares the main KV and Indexer K crossed layers, and allows layers to reuse Top-K indices, while decoupling cache sharing and scale reuse. It combines these reuse strategies pinch a simplified compressor and a hierarchical sparse Indexer, the second of which narrows the hunt scope of consequent scale layers successful the Decoder.

Some subtle computational differences from CSA: Similar to CSA, CSA2 includes a lightweight Indexer that uses Indexer Q and Indexer K to people the main KV entries, selecting Top-K entries for each query, and each Q simultaneously attends to the selected entries and the intra-layer section sliding model KV (SWA KV).

The Compressor of CSA2 besides has immoderate differences. First, it includes the typical lawsuit of compression ratio 1, that is, the expertise not to compress the Main KV, utilized for Decoder Layers.

In the actual implementation, the Causal-Encoder uses a compression ratio of which tin trim the number of entries and scale candidates of each transcript of the Main KV; the Decoder uses to clasp per-token addressability. After combining pinch CED and cross-layer cache sharing, the Decoder does not request to prevention an independent long-sequence main KV for each layer, truthful it tin allocate a information of the abstraction fund to a finer series granularity.

At the aforesaid time, CSA2 simplifies the Compressor and Indexer. In CSA, a compression ratio intends that each main KV introduction is produced by original KV cache entries, and location is overlap betwixt the root entries utilized by adjacent compressed entries. It besides includes absolute position embeddings to encode the positions of these entries during compression. CSA2 removes this overlap and the absolute position embeddings.

In addition, CSA2 obtains the Indexer K by projecting the main KV entries, replacing CSA's independent compression way starting from the hidden state. Both of these designs simplify the implementation and amended training efficiency.

2.3.2 Cross-layer KV and Index reuse

Specifically, CSA2 chiefly adds information reuse successful the layer dimension connected the ground of CSA. From the position of machine architecture, we tin respect the Attention artifact arsenic a compute component, MoE/FFN arsenic a retention component, and the KV-related portion arsenic the Cache of computation. From this perspective, we tin respect the cross-layer reuse of KV and Index arsenic a benignant of Data Locality processing, arsenic shown successful the fig below:

Cross-layer reuse is chiefly divided into 3 modes, arsenic shown successful the fig below:

The quality betwixt these modes lies successful really the Main KV, Indexer K, and Top-K Indices are obtained. Green squares bespeak quantities computed astatine the existent layer; yellowish squares bespeak the main KV and Indexer K reused from the astir caller Full mode layer; reddish squares bespeak the Top-K indices reused from the astir caller furniture that generated indices. In addition, each 3 modes compute Main Q and SWA KV astatine the existent layer.

Full mode This furniture computes its ain Main KV and Indexer Q, obtains the Indexer K by projecting from the Main KV, and runs the Indexer, producing caller Top-K indices. Therefore, it executes the complete CSA2 computation path, and the responsibilities borne by each constituent are the aforesaid arsenic those of a complete CSA furniture successful DeepSeek-V4.

Reindex mode This furniture reuses the astir precocious disposable Main KV from a erstwhile layer, arsenic good arsenic the corresponding Indexer K. And it computes its ain Indexer Q, re-scores the reused Keys, and generates caller Top-K indices. This allows the sparse action to alteration crossed layers, while the Main KV and Indexer K stay shared.

Reuse mode. This furniture reuses the astir precocious disposable Main KV, arsenic good arsenic the latest Top-K indices computed for that Main KV by a erstwhile Full mode furniture aliases Reindex mode layer. It uses this action to execute the attraction computation, does not compute Indexer Q, and does not measure scale scores.

Sharing the Main KV and Indexer K reduces the retention footprint of the KVCache, while reusing the Top-K Indices tin debar further Indexer computation. The Reindex mode allows the selected entries to alteration crossed layers while retaining cache sharing. We will analyse the circumstantial mode usage successful operation pinch CED.

2.3.3 The operation of CED and CSA2

In the Causal-Encoder, we tin respect it arsenic a building composed of the pursuing macro blocks. First is simply a 2-layer modular SWA block, past followed by a building composed of 3 macro Encoder blocks. Engram is injected earlier the 2nd SWA artifact and the past Encoder block, arsenic shown successful the fig below:

An Encoder artifact is simply a 6-layer structure, pinch the first furniture being a Full mode CSA2, and the consequent 5 layers being Reuse mode CSA2. The supra attack tin amended picture the full cross-layer reuse mechanism.

1. Why respect 1 furniture Full + 5 layers Reuse arsenic an integral Encoder block?

The first-layer Full mode CSA2 will compute and constitute the full Main KV and TopK indices. The consequent 5 layers of Reuse mode CSA2 will each reuse the Main KV and TopK indices produced by the first layer. In short, some KV and TopK action are reused successful the Reuse mode block, and what each furniture really modifies is Q.

Therefore, for the full Encoder artifact building containing 6 layers, we tin respect it arsenic a recursive Transformer building that performs recursion by modifying Q astatine each layer.

2. What is the domiciled of the first 2 layers SWA + Engram?

Described successful 1 sentence: the first furniture provides section context, Engram injects representation addressed by short patterns and controlled by context, and the 2nd furniture integrates the 2 into a practice usable for consequent compression.

The first-layer SWA model size is , and the group of positions that position tin publication is:

For example, the aforesaid connection successful different sentences will delegate different weights to different adjacent positions. The output of the first furniture is transformed by SWA into a practice processed by section context. Then comes the injection of Engram. Regarding Engram, location is simply a elaborate study successful the erstwhile article 《On DeepSeek Engram: Conditional Memory》.

Engram sounds the 2-gram, 3-gram, and 4-gram ending astatine the existent position, past concatenates aft hash array lookup. Therefore, Engram has the expertise to encode many times occurring building patterns, section operation regularities, etc. into the parameter table. And the output of the first-layer SWA provides discourse gating for Engram. Note that, compared to straight injecting Engram astatine the first layer, the output aft first-layer SWA processing is already a practice that combines section context.

Engram's injection straight does a residual update of the existent position; it does not straight constitute the existent position's representation into different positions. But the second-layer SWA tin publication the Engram-enhanced practice of each position successful the window. Therefore, some the Q and section KV of the second-layer SWA tin beryllium influenced by Engram to heighten applicable building patterns, section operation regularities, and different information. After the second-layer SWA processing, the integration of section accusation is completed. Therefore, the first 2 layers shape the pursuing order:

On the different hand, the consequence of the 2 layers of SWA expands the receptive section magnitude to: . Therefore, successful the CSA2 processing starting from the 3rd layer, the input contains: token embedding / the consequence of the 2 layers of section discourse computation / the Engram representation injected aft discourse gating and integrated—these processed section representations. This is besides 1 logic why the Compressor successful CSA2 does not request to execute Overlap processing.

Then fto america look astatine the building of the Decoder. We tin likewise respect it arsenic a building composed of 5 Decoder Blocks:

Among them, the first CSA2 successful the first Decoder Block is Full Mode, truthful we besides telephone it the Full Mode Decoder Block. Similarly, the consequent 4 Decoder Blocks are besides called Reindex Mode Decoder Blocks. Note that each CSA2 successful the Decoder person a compression ratio of 1, and only Reindex Mode CSA2 is utilized successful the Decoder.

The first furniture of the Full Mode Decoder Block is simply a Full Mode CSA2. Like the Full Mode CSA2 successful the Causal-Encoder, it computes its ain Main KV arsenic good arsenic Indexer K and TopK indices. The Main KV and TopK indices will beryllium reused by the consequent 3 layers of Reuse Mode CSA2. But it introduces a Hierarchical Sparse Indexer (HSI) processing approach, constructing a block-level action arsenic the Candidate Pool for the consequent Reindex Mode CSA2 done the scores erstwhile computing the Top-K Indices. We will grow connected this successful item successful a later section.

Reindex Mode Decoder Block: In the Decoder, location are 4 Reindex Mode Decoder Blocks aft the Full Mode Decoder Block. Its soul first furniture is simply a Reindex Mode CSA2. The Reindex Mode CSA2 will recompute its ain Indexer Q for scoring, and prime its ain Top-K entries wrong the Candidate Pool.

2.3.4. Hierarchical Sparse Indexer (HSI)

Cross-layer scale reuse reduces the number of Indexer evaluations, but the remaining Indexers still request to people each causally visible context. For ultra-long contexts, this costs is still a awesome computational bottleneck.

In , it studies really to usage "block-level coarse filtering + token-level good filtering" to trim the overhead of the sparse attraction indexer.

As shown successful the figure, it divides the prefix into blocks of size astir by consecutive positions, pinch the -th artifact being . The number of blocks is . Each artifact maintains the Indexer's mean:

The mean serves arsenic an auxiliary usability for searching. The original token Indexer and Main KV are still retained. During decoding, the vector sum and count tin beryllium maintained for the existent block, a caller token only updates 1 block, and the completed humanities blocks reuse the summary.

The first shape reuses the aforesaid group of to people the artifact means:

Only the high-scoring blocks are retained, past expanded into a campaigner position set:

The 2nd shape computes the original for , and selects the last positions from the candidates:

The DeepSeek squad recovered that successful the Decoder, the accusation of the shallow-layer Indexer tin people beryllium utilized to limit the candidates considered by the deep-layer Indexer, without needing to adhd immoderate other state. Therefore the authors present the Hierarchical Sparse Indexer (HSI), utilized only successful the CED decoder, to trim repeated scoring during decode. For each query, the first Full Mode furniture constructs a Candidate Pool arsenic the hunt domain for the consequent Reindex Mode layers. When the Candidate Pool size is fixed, the per-query costs of the deep-layer Indexer changes from increasing linearly pinch the discourse to a constant. This system is training-aware, introduced successful the post-training stage: training and conclusion enforce precisely the aforesaid campaigner restriction, truthful that the deep-layer Indexer is optimized nether the hunt domain utilized for inference.

The moving rule of HSI is arsenic follows:

The first Full Mode CSA2 scores each causally visible main KV positions, producing Top-K indices for its ain attention; astatine the aforesaid clip it performs block-level campaigner selection: each artifact takes the maximum scale people of its soul positions, selects the respective blocks pinch the highest scores, and collects the positions covered by these blocks into a campaigner excavation larger than the last Top-K set. For example, prime 2048 blocks, each artifact pinch 8 positions, obtaining 16384 campaigner positions. The campaigner excavation decides "where to search" for the consequent Indexer, and the last Top-K decides "which entries to read" for each layer.

The consequent Reindex mode layers only people the campaigner positions of the corresponding query, and prime their ain Top-K entries wrong the pool; the Reuse mode layers do nary caller indexing and straight usage the latest Top-K indices computed for the reused main KV. Thus the campaigner excavation is shared crossed scale layers, while the last action tin differ.

When the campaigner excavation size is fixed, the number of positions scored per query by each consequent Indexer is independent of the discourse magnitude and bounded; the first Full mode furniture still needs to scan the full causally visible range. Therefore hierarchical indexing reduces the costs of consequent Indexer information while retaining the first full-range scan.

The quality from HISA is that HISA needs to first execute artifact pooling scoring astatine each layer, past execute sparse TopK selection. And HSI takes the artifact maximum worth from the complete position scores already obtained astatine the shallow layer, consequent layers stock the campaigner range, and accommodate to the regularisation successful post-training. Both do coarse filtering first past good selection, but the ground for coarse filtering is different.

In addition, HSI taking the artifact maximum worth avoids the highest dilution that mean pooling whitethorn cause.

1. How does the Full Mode furniture make the Candidate Pool? The campaigner artifact width candidate_block_size is , the precocious limit of the number of blocks candidate_topk_blocks is , and the last selected TopK is . Use to denote the -th causally visible block,

The first furniture first computes the complete Indexer Score . For itself to nutrient TopK indices

The aforesaid group of scores is besides utilized for artifact maximum reduction,

The corresponding implementation is scores = F.pad(logits, (0, -width % block_size), value=-torch.inf) padding the past artifact pinch , past unflatten splits the history axis into the number of blocks and 8 positions per block, and amax(dim=-1) precisely implements . Finally, astatine astir blocks are selected by artifact score

In addition, a bound strategy is added. It temporarily sets the artifact people containing the latest visible position to , forcing this artifact to inhabit a selected-block quota. That is

For the selected blocks, a bitmap is produced, past done repeat_interleave(block_size, dim=-1)[..., :width] the artifact flags are expanded into a bitmap of the aforesaid width arsenic the original scores, constituting the campaigner excavation utilized for the consequent reindex mode computation.

In addition, we besides request to statement that the insubstantial states "This system is training-aware, introduced successful the post-training stage: training and conclusion enforce precisely the aforesaid campaigner restriction, truthful that the deep-layer Indexer is optimized nether the hunt domain utilized for inference." But the circumstantial post-training method is not disclosed here.

Assuming HSI is not used, each furniture of the Decoder that needs to re-index tin hunt the complete history for the astir applicable 512 positions. After utilizing HSI, the first retriever first shrinks the history into a database containing astatine astir 16384 positions, and the consequent retrievers tin only each prime 512 positions from the list. If during training the consequent layers are still allowed to hunt the full history, but erstwhile deployed online they are abruptly only fixed a list, location will beryllium an inconsistency betwixt the training-inference distribution and selection, affecting performance.

The insubstantial states that DeepSeek-V4.1-Flash already uses sparse attraction from scratch successful the pre-training stage, first training pinch 64K sequences, past extending to 1M, and intelligibly states that HSI is introduced successful the post-training stage. That is to say, the pre-training shape should person nary HSI connected the exemplary structure. The Reindex Mode layers still usage complete computation. In the post-training stage, to make the exclusive parameters of the deep-layer Indexer genuinely learn, location must besides beryllium an Indexer nonaccomplishment aliases different gradient estimation mechanism, but the study does not disclose the nonaccomplishment form. We estimate that it uses a hybrid nonaccomplishment function:

Among them, makes the backbone accommodate connected the restricted campaigner set; whitethorn usage the main attraction distribution to distill the Indexer's continuous scores. Combined pinch the implementation of DSA, we estimate it to be:

Where denotes extremity gradient, is the group of queries pinch valid supervision astatine this layer, and is the explicitly specified supervision support set. is the auxiliary nonsubjective coefficient.

2.4 Efficient architecture extensions

2.4.1 Single-Pass mHC

Regarding mHC, location is simply a elaborate study successful the erstwhile article 《On DeepSeek mHC》. It maintains residual streams betwixt adjacent Transformer blocks, arsenic shown successful the fig below

For each token, usage to denote these streams, wherever is the artifact scale and is the hidden dimension. These streams are updated arsenic follows:

Where , and are per-token coefficients predicted from . The coefficient predictor includes normalization and projection. But location is simply a information dependency during computation. Single-Pass mHC shifts the input mixing coefficients by 1 block, that is, each artifact uses the mixing coefficients produced by the erstwhile block, thereby eliminating the dependency:

The comparison is shown successful the figure:

2.4.2 Engram

Regarding Engram, location is simply a elaborate study successful the erstwhile article 《On DeepSeek Engram: Conditional Memory》. The differences betwixt DeepSeek-V4.1-Flash and the original paper's exemplary building are arsenic follows: the n-gram is extended to {2,3,4} and the short causal convolution is omitted, because of its trade-off betwixt the complexity of the conclusion package stack and the capacity gain. In addition, momentum-based updates are used, followed by Sinkhorn Balance to optimize the Engram Embedding.

It evenly distributes the 196B Engram parameters to 2 modules. Each module adopts N-gram orders , each bid contains 8 hash heads, and the full embedding magnitude is 2048. Each caput indexes a array pinch astir 16M entries, and each array size is chosen arsenic a chopped premier number. Both the embedding tables and the key/value projections usage FP8 precision. The modules are placed astatine furniture 1 and furniture 14, that is, earlier the Full mode CSA2 successful the 2nd SWA and the past Encoder Block, to equilibrium representation usage betwixt training pipeline stages.

During inference, deterministic addressing allows prefetching embeddings from big representation via inheritance RDMA transfers, and the prefetching of the first module overlaps pinch the computation of the first Transformer block.

2.4.3 DSpark

Regarding Dspark, location is already a elaborate study successful the erstwhile article 《A Detailed Discussion connected the Principle of DSpark Speculative Decoding》.

In DeepSeek-V4.1-Flash, the Draft Model consists of 3 Transformer blocks, whose sliding attraction model is 128 tokens. One guardant computation will compute successful parallel the basal unnormalized scores of 5 draught positions, while a lightweight Markov caput models the dependency betwixt Draft tokens.

DSpark is introduced successful a dedicated shape aft pre-training. In this shape only DSpark is trained, while keeping the backbone frozen. During post-training, DSpark continues to beryllium trained together pinch the backbone, but the gradient of the DSpark nonsubjective is not propagated to the backbone. This keeps DSpark aligned pinch the continuously evolving policy, thereby some accelerating online conclusion serving and accelerating the trajectory procreation of reinforcement learning RL and on-policy distillation OPD.

2.4.4 FP4 Main KV Cache

Further reducing the retention footprint of the KVCache from the numerical precision aspect. In DeepSeek-V4, quantization-aware training QAT was already utilized for the FP4 Indexer Q and K to accelerate scale computation and shrink the indexer cache. Here it is chiefly astir FP4 processing for the Main KV. To support arsenic galore hardware platforms arsenic possible, the OCP modular MXFP4 format is still adopted. Here the domiciled of FP4 is to trim storage, not to accelerate matrix multiplication. Before the attraction computation, the cache values are dequantized to a much meticulous format, without needing autochthonal support for matrix multiplication successful this format, thereby maintaining compatibility crossed hardware platforms.

The fig beneath shows NVFP4(E2M1) arsenic a reference:

DeepSeek chooses E2M1, pinch each 16 channels sharing 1 E4M3 standard factor, pursuing NVFP4, but omitting its second-level world standard factor, to equilibrium precision and simplicity. As shown successful the fig below:

After omitting this standard factor, the main KV cache still has ample move range: this format supports a maximum magnitude of , acold higher than the precocious bound of the cache magnitude. In DeepSeek-V4.1-Flash, the maximum RMSNorm weight magnitude obtained by training is astir 1. After RMS normalization, the L2 norm of the 512-channel KV latent adaptable is astatine astir about . RoPE preserves this norm, truthful the maximum absolute worth of each transmission aft rotation is besides bounded by astir . In addition, the maximum magnitude observed during training is astir 10. Therefore, omitting the world standard facet causes nary measurable precision degradation and simplifies the cache layout.

To support FP4 main KV cache retention successful DeepSeek-V4.1-Flash, QAT is introduced during post-training. The non-RoPE constituent and the RoPE constituent usage the aforesaid quantization format. In addition, the cache is quantized aft RoPE: successful experiments, quantizing earlier RoPE only brings a flimsy precision improvement, but introduces further overhead during decoding. Since the KV cache of sliding model attraction SWA is delicate to quantization, FP8 precision is retained. Compared pinch the FP8 Main KV cache of DeepSeek-V4, this format makes the retention footprint successful HBM and erstwhile offloaded to SSD some astir halved.

First, we will count the sources of KV Cache savings successful the first section. Overall, the optimization of KVCache by DeepSeek-V4.1 Flash is divided into respective parts. First, FP4 Main KV saves half of the overhead, CED reduces a ample magnitude of computation depletion during Prefill, and the astir captious is still the cross-layer sharing mode built successful CSA2. The reuse mode and reindex mode CSA2 wholly reuse the Main KV produced by the Full mode. The substantive problem is arsenic described below:

In modular Full Attention, the Q, K, V of each furniture are changing. And CSA2 shares the Main KV / Indexer K crossed layers, and tin complete training pinch precocious quality. Essentially, 1 mobility we request to reply is: under the premise of fixed KV, really does a recursive Transformer architecture composed of aggregate layers of reuse mode CSA2 execute expressiveness akin to Full Attention by only rewriting Q? This is the attraction of our study successful the 2nd section.

3.1 Why KVCache is 890B

First, fto america cipher why KVCache is 890B. DeepSeek-V4.1-Flash maintains 2 types of KV authorities per layer

1. Global KV - the compressed full-context branch, containing 2 parts:
   • main KV: the MLA latent vector produced by the compressor (compress_kv_cache).
   • indexer K: the lightweight cardinal utilized by the sparse indexer to people positions (k_cache).
2. Local KV (SWA KV): the sliding model cache of the astir caller tokens (window_kv_cache).

What resides successful HBM is the Global KV, that is:

The parameter array utilized for the derivation is arsenic follows:

Both world caches are stored pinch FP4 quantization-aware training. The retention costs of a azygous introduction = (FP4 payload) + (scale facet metadata). For a -channel vector, pinch 1 byte of standard per channels, the wide look for the number of bytes per introduction is:

Main KV entry (, ):

Index K entry (, ):

A furniture pinch compression ratio stores 1 introduction per tokens, truthful its retention per token = (bytes per entry) . Summing complete the Full mode root layers :

The root layers are (at ) and (at ), so

Main KV subtotal

Indexer K subtotal

Total

Summarized arsenic follows:

By comparison pinch DeepSeek-V4-Flash, the V4-Flash backbone has 43 layers: 2 axenic SWA layers, 21 CSA layers (sequence compression ), and 20 HCA layers (). Unlike V4.1, it is simply a CSA-HCA hybrid, and each furniture independently holds its ain world cache (no cross-layer reuse). The format per entry:

• Main KV entry: 448-channel non-RoPE FP8 (448 B) + 64-channel RoPE BF16 (128 B) + FP8 Scale (8 B) = 584 B
• CSA Indexer K: 128-dimensional MXFP4 = B. HCA has nary Indexer.

Relative to DeepSeek-V4-Flash, the sources of the KV Cache savings summation are arsenic follows:

3.2 From the position of machine architecture

In the article 《On the Evolution Path of Large Model Architectures, The Art of memory.》 from early past year, a viewpoint was mentioned, regarding the full Transformer artifact arsenic a computer:

Then from the position of architecture, we want to summation the Cache deed complaint arsenic overmuch arsenic possible, which fundamentally requires cross-layer reuse of KVCache. And if we respect an integral Encoder/Decoder ( 1x Full + B x Reuse ) artifact arsenic a recursive Transformer architecture, past the MoE of different layers precisely represent different page tables. Next, based connected this viewpoint, respect 1 furniture of CSA2 arsenic a mini computer, and the 3 modes are the cognition of the aforesaid group of information paths nether 3 cache deed states.

By information lifecycle, 1 furniture of CSA2 is divided into 3 architectural levels:

• Compute (execution unit): the Main Q and SWA KV recently computed astatine each furniture are section operands (like the registry operands recently fetched by each instruction), and the main attraction sparse_attn is the execution portion itself.
• Cache (on-chip shared cache): Main KV, Indexer K, TopK indices, and the HSI campaigner excavation are cross-layer shared reusable state, written by a definite root furniture and publication by aggregate consequent layers.
• Memory (large-capacity backend): the MoE / FFN of each furniture is large-capacity retention and transformation, receiving the attraction output.

Key correspondence: Main Q and SWA are ever "newly fetched operands", ne'er entering the shared cache; while Main KV / Indexer K / TopK are "cacheable state", whether to recompute depends connected deed aliases miss.

Listing the read/write of the 3 types of shared authorities into an entree shape table:

Thus the 3 modes are precisely 3 cache states:

• Full = write-back capable aft cache miss (write-allocate): the execution portion computes each cacheable authorities and writes it, pinch the highest cost.
• Reindex = information hit, reside recompute: the information (Main KV, Indexer K) hits, only the reside procreation is re-run to get caller TopK, for illustration a cache statement resident but redoing 1 reside translation.
• Reuse = afloat hit: some information and reside hit, the execution portion only uses caller operands (Main Q, SWA) to do 1 readout, which is the way closest to a axenic load.

The Indexer of CSA2 corresponds to the CPU's reside procreation portion (AGU) and TLB: it does not move data, only produces the reside group TopK of "which entries to read". The main attraction is the information path, which fetches the Main KV from the shared cache by reside and past computes.

• Full: the AGU runs astatine afloat speed, scanning each causally visible positions to nutrient addresses, while filling data.
• Reindex: the information cache is resident, only restarting the AGU to re-translate addresses wrong a restricted range.
• Reuse: moreover the AGU is skipped, straight reusing the past reside vector, which is balanced to doing communal subexpression elimination (CSE) and consequence memoization connected the costly reside computation.

Decoupling of reside procreation and information path The HSI campaigner excavation is simply a TLB aliases working-set constraint. Layer 20 selects 2048 blocks totaling 16384 positions, which is balanced to establishing a page array pinch a constricted addressable scope for the consequent Reindex layers: the reside recomputation of Reindex tin only autumn wrong the covered page (), and positions extracurricular the excavation are simply not addressable. The two-level TopK (block first, past position) is precisely a two-level page table: first usage the block-level maximum people to prime pages, past prime circumstantial entries wrong the page.

Three timepiece domains: usually we tin respect Q arsenic a query, the sparse action specified arsenic TopK arsenic an address, and the Main KV arsenic contented (a much precise meaning refers to the adjacent section). In fact, the 3 modes of CSA2 represent 3 refresh timescales (content ×4, reside ×8, query ×40) for illustration 3 timepiece domains aliases the different refresh rates of 3 levels of storage: the query is register-level updated each cycle, the reside is L1-level medium-frequency refill, and the contented is L2/L3-level low-frequency refill. The person to the execution unit, the faster and cheaper the refresh; the person to the shared backend, the slower and much costly the refresh. The mode scheduling of CSA2 is precisely placing each type of authorities astatine a refresh complaint matching its recomputation cost.

3.3 The mathematical rule of cross-layer KV sharing

3.3.1 The 3 types of degrees of state of attraction (content, address, query)

As introduced successful 《On the Future Transformer: Loops Are Not What You Need》, for a transformer block, the injectable surfaces are: residual , the summation and bias of normalization, the metric , the bias , the summation scope , the caput group , the output gate, and the consequent FFN. A fig summarizing them is arsenic follows:

For Sparse Attention, immoderate azygous sparse attraction readout is fundamentally wished only by 3 adaptable inputs. Write the readout astatine query position as

The 3 adaptable inputs play non-overlapping roles.

• Content is simply a group of contented vectors obtained aft compressing the history. In the reference implementation, the aforesaid simultaneously serves arsenic K and V (key = worth = ): erstwhile computing weights it acts arsenic the Key, appearing successful the soul merchandise of the exponent; erstwhile producing the consequence it acts arsenic the Value, appearing successful the weighted sum . So the contented decides 2 things astatine once, really akin an introduction is to the query, and what is publication retired aft a deed (the readout payload); this is different from modular attention, which splits K and V into 2 sets of projections.
• The reside decides which contented vectors this summation is normalized over, i.e., which positions to publication from;
• The query decides the weight guidance among these vectors. The output is simply a weighted mean (convex combination, besides called the barycenter) of the selected contented vectors, falling wrong the convex hull they span. Content, address, and query, these 3 are the "three types of degrees of state of attention".

The definition, mathematical type, semantic role, and procreation costs of the 3 types of degrees of state are each different.

Content : what to publication (readable dictionary)

• Definition: , generated by the gated pooling compressor astatine the kv-source furniture complete each published main entries. In the reference implementation cardinal = worth = .
• Type: continuous tensor , soft and differentiable; it is the vertex group of the convex hull wherever the output lies.
• Role: "what to read". Content provides the retrievable semantic carrier; without it, some reside and query suffer their pointing target.
• Cost: astir expensive. The compressor must scan each visible history, which is -level world computation.

Address : wherever to publication from (sparse addressing)

• Definition: , selected aft the Indexer scores the candidates.
• Type: discrete combinatorial entity , ; non-differentiable (TopK has zero gradient almost everywhere).
• Role: "where to publication from". The reside restricts the readout to normalize only complete the selected contented vectors, i.e., which vertices of the convex hull are chosen.
• Cost: medium. The Indexer needs to people the campaigner set: the Full furniture scans each causally visible positions, the Reindex furniture only scans the campaigner excavation (at astir ).

Query : really to publication (readout direction)

• Definition: , computed astatine each furniture by the layer's ain parameters from the existent hidden state
• Type: continuous vector , soft and differentiable.
• Role: "how to read". After is fixed, the query determines the weight distribution , i.e., the readout guidance among the selected contented vectors.
• Cost: cheapest. It is conscionable the layer's ain projection of the existent hidden state, per-token , without scanning the history.

The threefold asymmetry of these 3 is the fulcrum of the full CSA2 design:

• Continuous vs discrete: are continuous and differentiable, is discrete and non-differentiable. Training gradients tin only travel on ; tin only beryllium learned by an independent distillation way pinch the main attraction arsenic the coach (Part four).
• Local vs global: is the layer's ain section projection, must compress the world history, and must people each candidates. In procreation cost, .
• Fast-changing vs slow-changing: the semantic contented of the history changes slowest crossed layers, the reside worthy attending to drifts astatine mean speed, and the readout guidance of each furniture changes fastest.

The 3 besides person a one-way dependency chain. The reside is obtained by the Indexer scoring the content, truthful depends connected ; the query is injected astatine the readout extremity only aft is fixed to find the weights. Denoted as

Therefore reuse tin only proceed top-down on the dependency chain: 1 tin reuse the contented and jointly reuse the reside (Reuse), 1 tin reuse the contented but reselect the reside connected the aforesaid contented (Reindex), but 1 cannot alteration the contented while reusing the aged address. So the 3 modes of CSA2 are precisely 3 choices of "which degrees of state to refresh":

3.3.2 The Reuse mode is based connected query perturbation

Fix a furniture , denote its input hidden authorities arsenic , and a azygous token arsenic . The main attraction of 1 furniture of CSA2 requires 4 tensors: Main Q, Main KV content, Top-K action set, and section SWA KV. Below we first springiness the meaning of each 1 by one, past beryllium that the reuse mode freezes 3 of them, leaving only the query variable.

Main Q is computed by the layer's ain parameters from the existent hidden state, accordant crossed the 3 modes:

Where is the RoPE of position , acting only connected the statement tail channels, and each furniture has its ain independent .

Main KV contented is generated only astatine the kv-source furniture by the compressor ; is gated pooling, degenerating into a per-token projection erstwhile :

Top-K action group is generated only astatine the index-source furniture by the Indexer , wherever is the Indexer query, and is the per-head weight:

Local SWA KV is recently computed astatine each layer:

Readout concatenates the SWA and the selected Main KV into a azygous associated sparse attention:

Where .

Introduce 2 root mappings: is the Main KV root of furniture , and is its scale source; erstwhile the furniture generates them itself, return aliases . Using the indicators and , constitute the contented and action uniformly arsenic piecewise functions:

The 3 modes are precisely the worth combinations of this brace of indicators:

That is, Full simultaneously recently generates contented and selection, Reindex reuses contented but regenerates selection, and Reuse reuses both; for the reuse furniture , some contented and action are taken from an earlier root layer, independent of the layer's ain hidden authorities , truthful their partial derivatives pinch respect to are zero:

Substituting these 2 zero partial derivatives backmost into the readout mapping, the dependency of connected the layer's ain input decomposes into 2 caller channels positive 2 stiff constants:

For the stiff world representation , the only transmission carrying the dependency is the query ; SWA is different independent caller section channel, reconstructed furniture by furniture according to the fixed 128 window, without rubbing the world memory. Therefore, the inter-layer adjustment of the reuse furniture to the world representation mathematically contracts precisely into 1 query rewrite

Where is the baseline query utilized by the scale root furniture erstwhile publishing the selection, and is the learnable perturbation of this layer. This is precisely the entity of the consequent description analysis.

3.3.2.1 The main attraction tin beryllium written arsenic a readout of a fixed dictionary

Let the campaigner group of the reuse furniture beryllium , and stack the reused cache vectors by rows into . Because cardinal = worth = , the attraction output is

Since is simply a family of non-negative weights summing to , the output is simply a convex operation of the dictionary vectors . Therefore

The output is locked successful a convex polytope pinch fixed vertices, and rewriting tin only move its position wrong that convex hull.

The adjacent mobility to reply is: how ample a scope of this convex hull tin moving really cover?

In the DeepSeek-V4.1-Flash configuration, (head_dim), (sliding_window), (index_topk), truthful the campaigner group size .

3.3.2.2 Rewriting Q is balanced to exponential tilting of the baseline distribution

Denote the root layer's query arsenic , and the baseline distribution . Any query of the reuse furniture tin beryllium written successful perturbation shape . Substituting into (D1), the logit only gains 1 other word , so

This shows that the query perturbation does 1 exponential tilting of the baseline distribution on the guidance and past renormalizes. The achievable group of tiltings is

• If (requires ), past immoderate reweighting is reachable.
• In the DeepSeek-V4.1-Flash configuration, , truthful the tilting is restricted to a subspace of astatine astir dimensions.

3.3.2.3 How ample a scope tin the reuse output cover

Let the dictionary vectors beryllium (i.e., the rows of , ), the guidelines weights (i.e., the baseline distribution , satisfying ), and the scaling changeless . Denote the query arsenic a full by (absorbing the merger of and ), define

is precisely the attraction weight successful (D1), and is the log partition usability needed for its normalization. The attraction output is denoted arsenic the mean map

Thus the mobility of "how ample a scope the output tin cover" tin beryllium seen arsenic "what is the image group of the mean representation ". When the query tin freely traverse , aliases its effective linear subspace , and , the image of the mean representation is

However, this is an capability precocious bound, it describes "what tin beryllium reached astatine astir if the query is wholly free". But the existent model's normalization, shared query bottleneck, and finite parameters do not guarantee entree to each .

3.3.2.4 The expressiveness of only rewriting Q

So why does this query perturbation still person beardown expressiveness erstwhile the Main KV is frozen?

1. Preserving the full attraction simplex. Within the scope allowed by the rank, the reuse furniture tin still ore the wide onto a azygous selected introduction (approaching a definite vertex), flatten it, aliases do arbitrary soft interpolation. Sharing the reside is not adjacent to sharing the output.
2. Fully differentiable, zero re-retrieval cost. The tilting is soft pinch respect to , the gradient flows backmost normally, and each furniture tin specialize its measurement of reference the aforesaid memory; the skipped Indexer scoring and the non-differentiable TopK are not repeated.
3. Geometric alignment. cardinal = worth = , expanding simultaneously raises the weight of and pulls the output toward , and the perturbation is simply a straight interpretable power of the output position.
4. The global-vs-local ratio is besides managed by . SWA and Main KV are successful the aforesaid softmax, and besides allocates the wide betwixt the stiff world representation and the caller section window.

Another imaginable speculation is that we can, done immoderate benignant of learnable Q-aware parameters, representation the KV abstraction of the adjacent furniture backmost to the existent furniture to execute reuse.

Specifically, successful a modular Transformer, the KV of each furniture changes pinch the residual, frankincense constituting a manifold of a high-dimensional abstraction successful the furniture dimension. Then is location a business where, separating retired , utilizing this accusation arsenic immoderate benignant of spatial mapping, particularly erstwhile mHC tin support a comparatively unchangeable residual, truthful that the abstraction constituted by the KV of the later furniture tin beryllium mapped backmost to the abstraction constituted by the KV of the earlier furniture done immoderate Q-aware parameter weights, and fto the exemplary sorb this spatial mapping accusation into MoE/FFN and Engram during the training stage.

In this way, the KV tin beryllium fixed, while the Q-aware parameter mapping maps it to an due position? Even if location are immoderate defects that cannot beryllium remedied, is it imaginable to person these defects from the furniture magnitude of the exemplary into a longer series dimension, for illustration a caller fixed KV abstraction constituted by immoderate typical CoT to correspond it?

This contented is recorded successful immoderate soul documents, and related experimental study is being carried out.

3.3.2.5 The quality betwixt Reuse and Full modes

The Full furniture has 3 degrees of state that alteration independently per layer:

For fixed representation location is strict inclusion , and Full besides takes the national complete each . The expressiveness spread is precisely the 3 things reuse cannot do:

• Recall precocious bound: contented pinch is not successful , nary tin scope it, and an upstream TopK action correction cannot beryllium corrected downstream.
• Vertices immovable: the output is nailed wrong the fixed convex hull , and Full changing tin spot the output extracurricular the convex hull.
• Tilting geometry unswappable: the reachable distribution of reuse is simply a fixed exponential family wished by , and Full reselecting is balanced to replacing the full family.

In addition, RoPE only applies a position-dependent orthogonal rotation to the statement tail of and , which is absorbed into the soul merchandise , and does not alteration the supra convex hull and tilting argument.

3.3.3 The synergy of Full / Reindex / Reuse

In the magnitude of the exemplary layers, the 3 modes represent the pursuing structure:

The 3 modes represent a coarse-to-fine refresh schedule, corresponding to 3 timescales:

• Full rebuilds the KV cache, runs the complete Indexer positive TopK, and is the anchor.
• Reindex retains the content, re-scores and reselects Top-K wrong the HSI campaigner pool, and does not rebuild the KV.
• Reuse only does Q projection, SWA, and 1 sparse_attn, without scoring, without TopK, and without penning KV.

In the Decoder, HSI is besides introduced, truthful the section of labour of the 3 modes successful the decoder is:

• Full, denoted (layer 20): defines the contented , defines the campaigner excavation (16384 positions), and gives its ain Top-512.
• Reindex, denoted (24, 28, 32, 36): the contented is unchanged, reselects each of their ain Top-512 wrong , i.e., .
• Reuse, denoted : follows the published by the astir caller scale layer, only rewriting the query.