Zyphra Releases ZAYA1-8B: 8.4B Parameter MoE Model with 760M Active Parameters Matches 80B+ Models on Math Benchmarks
Zyphra has released ZAYA1-8B, a mixture-of-experts language model with 760M active parameters and 8.4B total parameters. The model scores 89.1% on AIME 2026, competitive with models exceeding 100B parameters, while maintaining efficiency for on-device deployment.
Zyphra ZAYA1-8B Achieves Frontier-Level Math Performance with 760M Active Parameters
Zyphra has released ZAYA1-8B, a mixture-of-experts (MoE) language model with 760M active parameters and 8.4B total parameters that achieves competitive performance with models over 10x its size on mathematical reasoning benchmarks.
Benchmark Performance
ZYPHA1-8B scores 89.1% on AIME 2026, outperforming Qwen3-4B-Thinking-2507 (77.5%) and Gemma-4-E4B-it (50.3%). According to Zyphra, the model matches or exceeds performance of significantly larger reasoning models:
- AIME 2026: 89.1% (vs. 90.2% for Qwen3-Next-80B-A3B-Think with 80B total parameters)
- HMMT February 2026: 71.6% (vs. 79.3% for Qwen3-Next-80B)
- LiveCodeBench v6: 63.8% (comparable to larger models)
- GPQA-Diamond: 71.0%
- MMLU-Pro: 74.2%
- IFEval: 85.8%
The model also scores 59.3% on IMO-AnswerBench and 32.2% on APEX-shortlist, significantly ahead of same-class models.
Architecture and Efficiency
ZYPHA1-8B uses a mixture-of-experts architecture with only 760M parameters active during inference while maintaining 8.4B total parameters. This design enables on-device deployment despite its competitive performance with frontier models like Mistral-Small-4-119B (6B active, 119B total) and Intellect-3 (12B active, 106B total).
The model requires specific installation from Zyphra's forked versions of vLLM and Transformers libraries. Deployment requires the --mamba-cache-dtype float32 --dtype bfloat16 flags and uses a custom reasoning parser.
Technical Specifications
- Active parameters: 760M
- Total parameters: 8.4B
- Model type: Mixture of experts with reasoning capabilities
- Inference format: Requires vLLM server with custom flags
- Recommended dtype: bfloat16 with float32 mamba cache
Availability
The post-trained reasoning version is available on Hugging Face. Zyphra has also released the pretraining base model separately. Pricing information has not been disclosed.
What This Means
ZYPHA1-8B demonstrates that mixture-of-experts architectures can achieve frontier-level mathematical reasoning with a fraction of the active parameters typically required. The 760M active parameter count makes it viable for edge deployment scenarios where models like Qwen3-Next-80B (3B active, 80B total) would be impractical. However, the model's relative weakness on creative writing tasks (62.97% on Creative Writing v3 vs. 83.75% for Gemma-4-E4B) and agentic benchmarks (39.22% on BFCL-v4) suggests the efficiency gains come with tradeoffs in general capability. The requirement for custom library forks may limit immediate adoption.
Related Articles
China Telecom Releases Xing4.0-29B-A4B, a 29B MoE Model Trained Entirely on Ascend NPUs
China Telecom Artificial Intelligence Technology has released Xing4.0-29B-A4B, a 29-billion-parameter mixture-of-experts model with only 4B parameters active per token and native 256K context. The company claims it is the first model of this scale trained entirely on Huawei's Ascend NPU platform using the MindSpore framework.
DeepSeek Ships V4.1-Flash With Novel Encoder-Decoder Architecture, Cuts KV Cache to 1/8 of Predecessor
DeepSeek released V4.1-Flash, a 763B-parameter model built on a new causal encoder-decoder architecture that splits 8B active parameters for prefill and 16B for decode. The model adds native vision support, a 1M-token context window, and shrinks KV cache footprint to roughly 1/8 of DeepSeek V4 Flash, while retiring V4 Pro.
PrismML Releases Ternary Bonsai 2 27B, a Compressed Reasoning Model with 262K Context
PrismML has released Ternary Bonsai 2 27B, a 27B-parameter reasoning model derived from Qwen3.8-27B that uses ternary weight compression to shrink to roughly 8.5 GB. The model supports a 262K-token context window, image understanding, tool calling, and thinks by default at 'xhigh' reasoning effort.
OpenAI RLHF Co-Inventor Launches Jev, a Non-LLM Model That Outputs Probabilities Instead of Text
TypeSafe AI, founded by RLHF co-inventor Diogo Almeida, has released Jev, a transformer-based model that outputs probabilities rather than text. Developers report it running 5 to 20 times cheaper and faster than LLMs for classification tasks.
Comments
Loading...