Yandex Releases AliceAI-Foundation-80B-A3B-Base, an 80B-Parameter MoE Model with 262K Context

Yandex Releases AliceAI-Foundation-80B-A3B-Base

Yandex has published AliceAI-Foundation-80B-A3B-Base, an 80-billion-parameter base language model built on a hybrid architecture combining Mixture-of-Experts (MoE) layers with a mix of KDA (linear attention) and gated attention blocks. The model activates only 3 billion parameters per token and supports a context window of 262,144 tokens. Weights are available on Hugging Face under the yandex namespace.

Architecture

According to Yandex, the model was trained entirely from scratch, including a newly assembled training corpus, architecture and hyperparameter choices, and data curated for reasoning and tool-use tasks. Key design decisions were validated across a series of separate 2-trillion-token training runs before scaling to the final model.

Specifications disclosed by Yandex:

• Total parameters: 80B (3B active per token)
• Hidden size: 2048
• Vocabulary size: 129,024
• Layers: 48, arranged as 12 × (3 × (KDA → MoE) → 1 × (Gated Attention → MoE))
• KDA block: 32 query heads, 32 KV heads, 128-dim query/KV heads, convolution kernel size 4
• Gated Attention block: 16 query heads, 2 KV heads, 256-dim query heads
• MoE: 512 experts, top-10 routing plus 1 shared expert, expert intermediate dimension 512
• Multi-token prediction: 1 layer
• Context length: 262,144 tokens

Benchmarks

Yandex ran internal evaluations (vLLM inference, temperature 0) comparing the model against Qwen3.5-35B-A3B-Base, GLM-4.5-Air-Base (106B-A12B), Nemotron-3-Super-120B-A12B-Base, and DeepSeek-V4-Flash-Base (284B-A13B). Reported scores include:

• WikiWebFacts (5-shot, Russian factual knowledge): 86.5 vs. 62.4 (Qwen3.5), 70.2 (GLM-4.5-Air), 72.8 (Nemotron-3), 83.2 (DeepSeek-V4-Flash)
• HardMultiQA (5-shot, Russian factual knowledge): 67.9 vs. 47.2 / 48.6 / 54.5 / 65.4
• CultCat (4-shot, cultural facts): 86.5 vs. 59.2 / 59.1 / 66.3 / 80.7
• MATH-500 (5-shot): 91.1 vs. 81.9 / 60.2 / 84.8 / 80.7
• AIME 2026 (pass@32): 96.7, matching Qwen3.5-35B-A3B-Base and ahead of Nemotron-3's 90.0
• MMLU-Pro CoT (5-shot): 66.8 vs. 63.2 / 58.4 / 69.9 / 66.5, where Nemotron-3 leads

Yandex claims the model performs on par with larger open-source models in math, coding, and reasoning tasks, and is particularly strong on Russian-language factual knowledge. The company also released two new Russian-focused factual benchmarks, WikiWebFacts and HardMultiQA, along with evaluation protocols. All benchmark figures come from Yandex's internal measurement infrastructure and have not been independently verified.

Deployment

The model can be run via Hugging Face Transformers (reference version 5.16.1, requiring flash-linear-attention 0.5.0 for KDA layers on GPU) or via vLLM using a Yandex-provided Docker image with tensor parallelism and speculative decoding via multi-token prediction. The tokenizer is a SentencePiece BPE implementation loaded as LlamaTokenizer, with chain-of-thought and tool-use markers implemented as ordinary vocabulary tokens rather than special HF tokens.

What this means

AliceAI-Foundation-80B-A3B-Base is a base (pretrained, non-instruction-tuned) model, so it is not directly comparable to chat-optimized releases without further fine-tuning. Its low active-parameter count (3B of 80B) makes it comparatively cheap to run relative to its total size, positioning it as a Russian-market-focused alternative to Qwen, GLM, Nemotron, and DeepSeek open MoE models. The strongest claimed gains are concentrated in Russian-language factual and educational benchmarks — an area where general-purpose multilingual models typically underperform — suggesting Yandex optimized training data specifically for Russian-language knowledge coverage rather than pursuing broad multilingual parity.