Tencent releases Hy4 flash: 220B MoE, 12B active parameters, 1M-token context
Tencent has released Hy4 flash, a multimodal Mixture-of-Experts model with 220B total parameters, 12B active per token, and a 1,048,576-token context window. Tencent calls it its most cost-effective Hy model to date and claims a per-task cost of 1/20 that of Hy4 on its internal WorkIndex benchmark.
Tencent has released Hy4 flash, a Mixture-of-Experts model with 220B total parameters (including 64B Engram parameters) and 12B active parameters per token. It accepts text and image inputs, generates text, and supports a context window of up to 1M tokens (1,048,576). Tencent describes it as its most cost-effective Hy model to date. API pricing is not yet disclosed.
Architecture
The 42-layer language backbone has one dense feed-forward layer and 41 MoE layers. Identity Hyper-Connections (iHC) with 4 residual streams carry information between layers. Hidden size is 3,072 and vocabulary size is 120,832.
Tencent lists three changes relative to Hy4:
- Hybrid attention: sparse attention (IndexPool) and sliding-window attention (SWA) are interleaved at a 1:3 ratio. Twelve layers use sparse attention and 30 use SWA. Sparse layers select up to 2,048 tokens, and SWA uses a 512-token causal window.
- LatentMoE: routed experts operate in a lower-dimensional latent space (1,536). Each token activates 16 of 512 routed experts, plus one shared expert.
- Engram: learned bigram and trigram embeddings retrieved by multi-head hash lookups (8 hash heads per order, 63.4B embedding table parameters).
Parameter breakdown: 155.5B excluding Engram, ViT and DFly2; 63.5B Engram; 0.43B vision encoder; 0.85B DFly2. Activated parameters per token are 11.3B for text plus 0.43B for vision. The vision encoder is adapted from SigLIP2-So400m-Patch16-Res512 and connects to the backbone through an MLP projector.
A built-in 5-layer drafter, DFly2 (0.85B parameters), reuses the target model's KV cache and sparse indices for long-context speculative decoding.
Base model benchmarks
Tencent reports few-shot results for Hy4 flash Base against Hy4 Base, Hy3 preview Base, DeepSeek-V4-flash-Base and DeepSeek-V4-Pro-Base. Hy4 flash Base scores:
- MMLU 88.7, MMLU-Pro 71.1, MMLU-Redux 87.6
- SuperGPQA 57.8, C-Eval 92.0, CMMLU 91.6
- SimpleQA 37.5, U-MATH 69.8, MBPP-plus 73.7
- MMMU-Pro (Standard, 10 options) 52.0, RealWorldQA 71.1, TextVQA 73.4
For comparison, DeepSeek-V4-flash-Base scores 89.7 on MMLU, 52.5 on SuperGPQA, 57.4 on U-MATH and 76.8 on MBPP-plus. Hy4 Base scores higher than Hy4 flash Base on most listed rows, including MMLU (91.6) and SimpleQA (57.2). Multimodal results are reported only for the two Hy4 base models.
Tencent notes that its base model saw a large amount of synthetic instruction data during mid-training and occasionally fails to follow the expected answer format in few-shot evaluations, which it says may distort the scores.
Cost and efficiency claims
All figures below come from Tencent.
- KV cache size is 1/24 of Hy3 and 1/6 of Hy4.
- Reinforcement learning cuts average end-to-end output tokens by 50%. Post-training began with self-distillation from Hy4.
- Megakernels, 4-bit quantization and DFly2 push decoding speed past 150 tokens/s.
- On WorkIndex, Tencent's internal benchmark built around typical WorkBuddy tasks, Hy4 flash scores 42.1, against 39.7 for Hy4 preview and 51.4 for Hy4.
- Average cost per task is 1/13 of Hy4 preview, 1/20 of Hy4, and 1/8 of DeepSeek-V4.1-Flash-1017 (Max tier, peak-hour pricing).
WorkIndex has 8 evaluation sets: slide creation, slide editing, human-AI slide co-writing, Markdown human-AI co-writing, document editing, spreadsheet repair, audio/video processing, and visual understanding. Tencent also tracked 7 public agent benchmarks: DeepSWE, ProgramBench, Terminal-Bench 4.0, APEX-Agents, AutomationBench, HLE and OSWorld 2.1.
In a user blind evaluation, Tencent says Hy4 flash reached user satisfaction close to Claude Haiku 5.5 while recording the lowest cost per task.
Deployment
The Hugging Face page covers Transformers, an OpenAI-compatible API, vLLM and SGLang deployment, DFly2 speculative decoding, quantization, and fine-tuning. A GGUF version is linked.
What this means
Hy4 flash is built to lower serving cost: the KV cache reduction, 12B active parameters out of 220B, and a 1M-token window point to a model aimed at long-context workloads. Tencent recommends it for everyday productivity tasks such as deep search, document processing, slide creation, simple frontend demos and computer use. For complex work such as long-horizon software engineering, it recommends Hy4 for its higher intelligence ceiling. On WorkIndex, Hy4 scores 51.4 against 42.1 for Hy4 flash.
Most of the cost and quality evidence is internal to Tencent, including WorkIndex and the cost-per-task ratios, and the base-model tables carry the format-following caveat above. Independent testing and published API pricing will be needed to check the cost claims.
Related Articles
Tencent releases Hy4 Flash Base: 220B-parameter MoE, 12B active, 1M-token context
Tencent has published Hy4 Flash Base on Hugging Face: a 220B-parameter mixture-of-experts model with 12B active parameters per token, text and image input, and a 1M-token context. Tencent calls it its most cost-effective Hy model to date.
Tencent releases Hy4 flash weights in three variants under a custom license with revenue and user caps
Tencent has released downloadable weights for Hy4 flash in three variants: instruction-tuned, FP8 and base. The checkpoints ship under Tencent Hy License 1.0, which requires a separate license above US$50 million in revenue or 100 million monthly active users. Tencent says the larger Hy4 model's weights are still being prepared.
SenseNova-Loop-9B released: 9B parameters, 56 layer executions, 512K context
SenseNova has released SenseNova-Loop-9B, a roughly 9B-parameter multimodal model that runs 56 decoder-layer executions per forward pass from 32 distinct layers. It supports up to 512K tokens of context and, per the model card, needs about 15 GiB of GPU memory at 256K context with 4-bit weights and an 8-bit KV cache.
Microsoft releases Aion 1.0 Instruct, a 2.3B on-device SLM with 32K context under MIT license
Microsoft's Windows Applied Sciences Group has published Aion 1.0 Instruct, a roughly 2.29B-parameter instruction-tuned small language model with a 32,768-token context window under the MIT license. A further optimized version is slated to become the Windows OS SLM by the end of January 2027.
Comments
Loading...