Baidu dropped ERNIE 5.5 Flash Base

1. Introduction

A basis example is precious the two for the capabilities that pre-training builds and for the starting item it offers for additional training. Different domains, reasoning tasks and agentic applications call for distinct data, training objectives and behaviors. Releasing the weights at the end of pre-training lets developers build on the broad capabilities already in location and choose for themselves anywhere to obtain the example next.

Today we are releasing ERNIE-5.5-Flash-Base, a sparse Mixture-of-Experts (MoE) tongue example alongside 280B total parameters, 14B of which are activated per token. It was pre-trained from scratch on 33.5T tokens alongside a training environment of up to 128K, and supports a 1M-token environment for fine-tuning and inference. The weights are the checkpoint at the end of pre-training, before any mid- or post-training, and can assist as a average starting item for domain adaptation, education tuning and reinforcement-learning research.

To be value construction on, a basis example has to remain affordable to train and run, so effectiveness was a scheme goal from the outset. Further training samples from the example repeatedly, and agentic applications bring multi-turn interactions and lengthy contexts. Both add to compute and recollection costs. ERNIE 5.5 Flash hence revisits the trade-off between capability and effectiveness in all component, from notice and residual connections to the MoE layers and the output head, and weighs activated compute, KV cache and long-context disbursal together fairly than one at a time.

The identical thinking shaped the data and the training system. Every afterward phase of training builds on what pre-training provides, so the corpus emphasizes capabilities that are difficult to add afterwards, specified as repository-scale code and long-context understanding. And since an productive architecture pays off lone if it can additionally be trained efficiently at scale, we designed the example and the training scheme together.

We anticipation this publish spares additional developers the disbursal of large-scale pre-training, so they can expend their own resources on the capabilities, tasks and training methods they attention about.

Key highlights:

• An open starting item for additional training. Base weights alongside no mid- or post-training, leaving the community liberated to prosecute its own approaches to domain adaptation, reasoning and delegate training.
• Built for productive additional training and inference. Sparse MoE and hybrid sparse notice balance example capacity, activated compute and long-context disbursal in the architecture itself.
• Broad data, productive training. Diverse, high-quality pre-training data gives afterward training a well-rounded foundation, and a training scheme co-designed alongside the architecture keeps large-scale pre-training efficient.
• A spotless investigation baseline. A sole checkpoint taken at the end of pre-training makes it uncomplicated to difference training methods under controlled conditions and to measure what all one adds.

2. Model Architecture

ERNIE 5.5 Flash has 43 Transformer layers, a hidden size of 4096 and 64 notice heads.

2.1 Latent-Gated Sparse Attention

Hybrid MLA + HCA attention: Most layers need lone a coarse perspective of the distant past, so lone one tier in seven keeps token-level admission to the complete context, using Multi-head Latent Attention (MLA). The remaining layers use Heavily Compressed Attention (HCA): all compresses all 128 tokens into a sole cached entry, afterward attends to all of these entries affirmative a abbreviated opening of recent, uncompressed tokens. An HCA tier hence motionless sees the entire sequence, during caching lone a small fraction of what a full-context tier needs.

Layer-specific caput dimensions: Instead of using the identical caput size in all layer, we set it according to how costly all layer's notice is: cheap layers get broad heads, costly layers get narrower ones. HCA layers act complete a sequence compressed 128×, so their notice is cheap and they can oversee broad 512-dim heads at small additional compute. MLA layers act at token flat and disbursal more, so they use narrower 256-dim heads for the two query/key and value.

Concentrating the KV-cache prosperity on MLA layers: With HCA in most layers, the chief inquiry is anywhere to expend the remainder of the KV-cache budget. One choice is many Compressed Sparse Attention (CSA) layers, which compress the former 4× and act lone to a selected part of it. We alternatively use a few MLA layers that keep all token. In our pre-training comparisons at a akin KV-cache budget, HCA blended alongside MLA outperforms HCA blended alongside CSA.

Sparse notice in the full-context layers: The MLA layers are the lone layers that see all distant token individually. With compact attention, they would document for most of the notice compute at lengthy environment lengths, so they use sparse notice (DSA) instead. A lightweight indexer scores former positions for all query, and center notice is computed lone complete the top-k highest-scoring positions. The per-query disbursal of center notice hence stays changeless as the environment grows. With DSA, these layers equivalent compact notice in pre-training defeat and train substantially faster.

Two additional changes use inner the notice layers:

• Low-Rank Gated Attention: Gated notice multiplies the notice output by a learned gate, which has been shown to enhance the two norm and training stability. Normally the entrance is computed from the layer's hidden state. We alternatively compute it from the low-rank query latent, a compressed form of the query that all notice tier already computes. Because this latent is 4× narrower than the hidden state, the entrance projection has 75% small weights, and our ablations additionally display improved performance. Every notice tier in ERNIE 5.5 Flash, MLA and HCA alike, uses Low-Rank Gated Attention.
• Virtual Head Attention (VHA): After attention, VHA adds a lightweight low-rank mixing stage throughout heads, alongside an character skip connection. This lets heads portion information, which improves expressiveness and training quality, and at conclusion it is fused into the notice kernel alongside negligible overhead.

2.2 Multi-Router Latent MoE

A accepted MoE tier picks experts alongside a sole router and a difficult top-k. We discover that a sole router's scores correlate lone weakly alongside how helpful all expert really is, particularly in the center and deeper layers. And since experts remaining out of the top-k obtain no gradient from that token, the router gets no indication that it chose wrongly, so incorrect selections lean to fortify themselves.

Multi-Router replaces the sole router alongside multiple independently initialized sigmoid routers that mark the experts jointly. The routers create plainly distinct preferences: an expert can be chosen since one router powerfully favors it or since multiple favor it together, so a incorrect choice is easier to correct. The errors of any one router no longer decide the selection, and routing becomes small delicate to noisy features and allocation shift. The figure of activated experts, expert FLOPs and communication quantity remain the same. Compared alongside a sole router, routing scores correlate noticeably improved alongside how helpful all expert is, alongside the largest gains in the center and deeper layers.

All layers apart from the archetypal use an MoE feed-forward block:

• 512 routed experts (top-10) affirmative 1 shared expert, alongside auxiliary-loss-free, bias-corrected burden balancing.
• Latent experts. Each MoE tier archetypal projects tokens from 4096 to a 2048-d latent space; the experts (intermediate size 2048) run in that space, and a sole shared up-projection maps the outcome rear to 4096. This approximately halves dispatch/combine traffic and expert FLOPs related to full-width experts.
• Stable LatentMoE scheme alongside learnable scalars. The MoE blocks prosecute the Stable LatentMoE scheme (an RMSNorm before the up-projection and SiTU-GLU expert activations), which keeps expert outputs fine scaled throughout all 43 layers and stabilizes optimization. In addition, all routed expert's routing importance is multiplied by its own learnable scalar.

2.3 Manifold-Constrained Hyper-Connections (mHC)

ERNIE 5.5 Flash replaces the norm residual association alongside mHC: all tier says from and writes to 4 parallel residual streams through a constrained mixing matrix, which gives layers a wider conduit for passing data to one another at negligible additional compute.

2.4 MultiMax Output Head

A sole softmax heat forces a trade-off: lowering it makes the allocation sparse but additionally suppresses another plausible tokens; raising it keeps them but fattens the lengthy rear of implausible ones. MultiMax replaces the sole heat alongside a learnable piecewise temperature, so the output caput no longer has to choose: it suppresses the lengthy rear of implausible tokens during keeping multiple plausible next tokens. The logits are rescaled element-wise inner the chunked, Triton-fused cross-entropy kernel fair before the softmax. This adds negligible training overhead and no additional memory, and considerably lowers pre-training loss.

2.5 ECHO: Replacing the MTP Module

Conventional multi-token prediction (MTP) attaches additional modules to the backbone to foretell forthcoming tokens, so the defeat on forthcoming tokens mostly trains those modules, and the backbone is motionless trained chiefly to foretell the next token. ERNIE 5.5 Flash replaces the MTP component alongside ECHO, which reuses the backbone's own final layer to foretell forthcoming tokens. Because there is no distinct MTP layer, the defeat on forthcoming tokens immediately trains the identical parameters that create the next token. Learning to appearance former the next token encourages the backbone to anticipate anywhere the sequence is heading, alongside apparent gains on reasoning and code. ERNIE 5.5 Flash is pre-trained alongside single-step ECHO. The identical tier additionally serves as the outline example for speculative decoding, and it generalizes beyond its training: although trained to foretell lone one additional token, it drafts multiple tokens ahead, alongside a higher acceptance charge than an auxiliary MTP component trained the identical way.

2.6 Tokenizer

ERNIE 5.5 Flash uses a new BPE tokenizer alongside a vocabulary of concerning 200K entries. Numbers are tokenized right-to-left (R2L) in groups of up to three digits, so that digits alongside the identical location value row up throughout numbers of distinct lengths (e.g. 1,234,567 → 1 234 567). This helps alongside arithmetic and number-heavy text.

3. Pre-training

ERNIE-5.5-Flash-Base was pre-trained from scratch on 33.5T tokens alongside the Muon optimizer and FP8 blended precision. The training environment was gradually extended to 128K tokens complete the way of pre-training, and fine-tuning and conclusion assistance a 1M-token context. The released checkpoint is taken at the end of pre-training and has not gone through any mid-training or post-training.

3.1 Data

The pre-training corpus covers a broad range of domains and languages and is curated for the two breadth of cognition and quality. Beyond this broad base, we focused on three areas:

• Web data. Multilingual web data is crawled at scale, extracted alongside elevated fidelity and filtered by models that mark norm alongside multiple dimensions. The outcome covers a broad range of cognition and is compact in helpful training signal.
• Code data. Code collected from repositories and the web is filtered to eliminate non-code content, aggressively deduplicated and graded for norm at multiple granularities. Beyond fixed code, the corpus includes data on how application evolves and executes: commits, issues, drag requests, code reviews, requirement-to-PR and multi-commit trajectories, generated tests and CI-verified fixes. This data targets repository-level reasoning and agentic application engineering.
• Long-context data. Three complementary sources: natural lengthy documents, selected for long-range requirements and cleaned alongside embedding-based deduplication, LLM norm scoring and rules; lengthy sequences built by concatenating natural texts alongside retrieved, semantically akin distractors; and synthetic data for MRCR-style retrieval, citation-grounded QA and multi-hop QA.

3.2 Training Infrastructure

We built two infrastructure components for this run: a distributed Muon optimizer that stays productive throughout thousands of GPUs, and FlashMask-4, an notice kernel for ample caput dimensions.

Distributed Muon. Unlike element-wise optimizers specified as Adam, Muon orthogonalizes the update of all indicator matrix as a whole. This changes how distributed training communicates and how recollection is dispersed throughout GPUs. Two optimizations create it applicable at the measure of thousands of GPUs:

• Communication-efficient updates. Conventionally, compact parameters are synchronized complete a sole circle that passes through all GPU in the cluster. Data travels throughout a lengthy sequence of inter-node links in many small messages, which leaves the links underused, and latency and jitter develop as the collection gets larger. MoE parameters are synchronized differently: all GPU exchanges data lone alongside the GPUs that clasp the identical position on another nodes. We move dense-parameter synchronization onto this identical way and merge the compact and MoE transfers into one company operation. Messages rotate into larger, so bandwidth is used additional fully; the communication form becomes additional regular and fits expert-parallel communication; and inter-node latency and jitter autumn significantly. The optimizer's updates remain exactly the same.
• Memory-balanced sharding. Muon needs ample messages to communicate efficiently, so naive sharding groups many indicator tensors together and places them on the archetypal few GPUs. A fistful of nodes afterward clasp far additional optimizer province than the rest. They are prone to OOM failures and dilatory to save, upload and download checkpoints, which lengthens nonaccomplishment recovery, and the issue grows alongside GPU count. Our inter-node stable sharding spreads these optimizer states complete additional GPUs, including those afterward in the order. Memory is afterward stable nearly evenly throughout nodes, during messages remain ample adequate for productive communication.

FlashMask-4 for ample caput dimensions. At ample caput dimensions, FlashAttention-style kernels run up against the limits of on-chip retention specified as shared recollection and tensor memory: compute use drops, and several configurations are difficult to execute at all. FlashMask-4 is built on CuTe-DSL for the two SM90 and SM100 GPUs. It plant about these limits alongside 2-CTA tiled parallel computation and, on SM100, double-buffer rotation in tensor memory. It keeps FlashMask's compact O(N) column-wise disguise depiction and motionless skips completely masked blocks. It supports caput dimensions up to qk_head_dim = 576 and v_head_dim = 512, beyond what FlashAttention-4 supports. On SM100 GPUs, it is up to 175% faster in the onward continue and up to 74% faster in the backward continue than FlashAttention-4 MaskMod.

4. Evaluation

4.1 Setup

Alongside ERNIE-5.5-Flash-Base, the fig and array study three basis models of akin size: DeepSeek-V4-Flash-Base (284B-A13B), Hy3-preview-Base (295B-A21B) and MiMo-V2.5-Base (310B-A15B). All four models are evaluated from their publically released checkpoints alongside the identical evaluation harness, prompts and few-shot examples, and alongside the identical LLM fairness for free-form QA.

4.2 Overall Results

ERNIE-5.5-Flash-Base performs powerfully throughout all benchmark categories, alongside reasoning, code and exam benchmarks among its finest results. Although 37 of the 43 layers act complete a sequence compressed 128×, the six MLA-DSA layers are adequate to keep achievement powerful on the six long-context benchmarks.

5. Quick Start

ERNIE-5.5-Flash-Base is accessible on Hugging Face in FP8 and BF16 safetensors (the BF16 type is ERNIE-5.5-Flash-Base-BF16), under the ERNIE 5.5 Flash Base Model License Agreement v1.0. The example expects a BOS token at the commencement of all input, so keep add_special_tokens=True whenever tokenizing.

from transformers import AutoModelForCausalLM, AutoTokenizer import torch model_name = "baidu/ERNIE-5.5-Flash-Base-BF16" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto") model_inputs = tokenizer(["The hybrid notice architecture of ERNIE 5.5 Flash"], add_special_tokens=True, return_tensors="pt").to(model.device) print(tokenizer.decode(model.generate(**model_inputs, max_new_tokens=256)[0], skip_special_tokens=True))

SGLang and vLLM the two assist the example through the completions API; see the example cardstock for deployment instructions. For fine-tuning, delight mention to the ERNIE 5.5 example.

6. Conclusion and Outlook

ERNIE-5.5-Flash-Base starts from one premise: a basis example meant for additional training have to be productive by design. We stable capability and effectiveness in all component, from notice and residual connections to the MoE layers and the output head. The architecture is backed by broad, high-quality pre-training data and by a training scheme designed together alongside the model. With 14B parameters activated per token and a compact KV cache, the example performs powerfully throughout a broad range of benchmarks, including long-context ones.

This basis example is lone a starting point. The post-trained ERNIE 5.5 Flash volition follow, and we appearance onward to seeing what the community builds on top of it, whether for domain adaptation, reasoning or agents.

Citation

If you discover ERNIE 5.5 helpful in your investigation or projects, delight regard citing this blog post:

@misc{ernie5.5flash, heading = {ERNIE 5.5 Flash Base}, author = {{Baidu ERNIE Team}}, twelvemonth = {2026}, howpublished = {ERNIE Blog}, url = {https://ernie.baidu.com/release/ernie-5-5-flash-base/} }