Baidu open-sources ERNIE 5.5 Flash Base, an MoE model with 1M-token context for fine-tuning and inference

Baidu open-sourced ERNIE 5.5 Flash Base (文心5.5 Flash Base) on November 5, releasing a Mixture-of-Experts base model with support for up to 1 million tokens of context during fine-tuning and inference. The announcement was published in Chinese by QbitAI (量子位) as content supplied by Baidu and republished with authorization. The weights are on Hugging Face at huggingface.co/baidu/ERNIE-5.5-Flash-Base.

Specifications

• Architecture: Mixture-of-Experts (MoE)
• Total parameters: 2800亿 (2,800 hundred-million)
• Active parameters per token: 140亿 (140 hundred-million)
• Pretraining data: 33.5 trillion tokens
• Pretraining context length: up to 128K tokens
• Fine-tuning and inference context: up to 1 million tokens

Benchmark claims

According to Baidu, the model reaches state-of-the-art results among models of the same size across knowledge, reasoning, math, code, multilingual and long-context benchmarks. Baidu says it outperforms DeepSeek-V4-Flash-Base, Hy3-preview-Base and MiMo-V2.5-Base. Specific benchmark names and scores were not given in the announcement, and these results come from Baidu rather than independent evaluation.

Design: efficiency across attention, routing and training

Baidu says the model was optimized around efficiency, as longer documents, multi-turn interaction and agent tasks raise compute and memory costs.

Attention. The model uses Latent-Gated Sparse Attention, which combines three techniques:

• Hybrid sparse attention, built by combining HCA and MLA, so different layers either compress historical information or access specific historical positions in fine detail.
• Low-Rank Gated Attention, which computes information gates from existing low-dimensional representations. Baidu says this cuts gating projection weight parameters by 75%.
• Virtual Head Attention (VHA), which Baidu says improves expressiveness and training quality. At inference it is fused into the attention operator with minimal overhead.

Expert routing. Multi-Router Latent MoE uses several independent routers to select experts, instead of a single router. Baidu says the aim is to improve the correlation between routing scores and actual expert utility and to reduce the effect of any single router's scoring bias. The number of activated experts, expert compute and communication volume stay unchanged.

Training objective. The model introduces ECHO, which avoids adding a separate Multi-Token Prediction (MTP) module. Instead it reuses the backbone's last layer so the backbone learns to predict subsequent tokens. Baidu says this improves reasoning and code ability. ERNIE 5.5 Flash uses single-step ECHO pretraining only. The reused final layer can also serve as a draft model for speculative decoding. According to Baidu, although it was trained for only one step, its multi-step drafting quality is substantially better than that of an auxiliary MTP module.

Infrastructure. Baidu optimized the distributed Muon optimizer and introduced FlashMask-4, an attention operator for large head dimensions. The company says both improve communication, memory use and compute efficiency in large-scale training.

What this means

This is a base model, not an instruction-tuned application model. Baidu positions it as a foundation that developers and enterprises can fine-tune into specialized models for their own use cases, and says open base models lower training cost and technical barriers.

The 128K pretraining context against 1M tokens at fine-tuning and inference matters for teams working on long-document and agent workloads. The attention changes, including the 75% reduction in gating projection weights, target the memory and compute cost of those workloads. The benchmark superiority claims are Baidu's own and name three comparison base models. Teams considering the model should check results on their own tasks, since no scores were published alongside the announcement.