model releaseTencent

Tencent releases Hy4 Flash Base: 220B-parameter MoE, 12B active, 1M-token context

TL;DR

Tencent has published Hy4 Flash Base on Hugging Face: a 220B-parameter mixture-of-experts model with 12B active parameters per token, text and image input, and a 1M-token context. Tencent calls it its most cost-effective Hy model to date.

4 min read
0

How Hy4 Flash Base ranks now

BenchmarkScoreRankSource
MMLU-Pro71.1%62 of 76Hugging Face model card, Nov 2026 ↗

Rank is against every model in our database with a sourced score on that benchmark, each counted once with its best score.

All Hy4 Flash Base specs and scores →

Tencent has published Hy4 Flash Base on Hugging Face, a 220B-parameter mixture-of-experts model that activates 12B parameters per token and supports a context window of 1M tokens (1,048,576). Tencent describes it as its most cost-effective Hy model to date. Pricing is not yet disclosed.

Architecture

The model accepts text and image inputs and generates text. The total of 220B parameters includes 64B Engram parameters. The model card breaks this down as 155.5B (excluding Engram, ViT and the DFly2 drafter), 63.5B Engram, 0.43B vision encoder and 0.85B DFly2. Activated parameters per token are 11.3B for text plus 0.43B for vision.

The 42-layer language backbone has one dense feed-forward layer and 41 MoE layers, with identity Hyper-Connections (iHC) using 4 residual streams. Hidden size is 3,072 and vocabulary size is 120,832. The vision encoder is adapted from SigLIP2-So400m-Patch16-Res512 and connects to the backbone through an MLP projector.

Tencent lists three architectural changes relative to Hy4:

  • Hybrid attention: sparse attention (IndexPool) and sliding-window attention (SWA) are interleaved at a 1:3 ratio. 12 layers use sparse attention and 30 use SWA. Sparse layers select up to 2,048 tokens, and SWA uses a 512-token causal window.
  • LatentMoE: routed experts operate in a lower-dimensional latent space (1,536). Each token activates 16 of 512 routed experts, plus one shared expert.
  • Engram: learned bigram and trigram embeddings retrieved through multi-head hash lookups, with 8 hash heads per n-gram order.

A built-in 5-layer drafter, DFly2 (0.85B parameters), reuses the target model's KV cache and sparse indices for long-context speculative decoding.

Base model benchmarks

Tencent reports few-shot results for Hy4 Flash Base against Hy4 Base, Hy3 preview Base, DeepSeek-V4-flash-Base and DeepSeek-V4-Pro-Base.

Benchmark Hy4 Flash Base Hy4 Base DeepSeek-V4-flash-Base DeepSeek-V4-Pro-Base
MMLU (5-shot) 88.7 91.6 89.7 91.1
MMLU-Pro (5-shot) 71.1 73.9 68.6 73.4
SuperGPQA (5-shot) 57.8 62.7 52.5 54.8
C-Eval (5-shot) 92.0 95.0 91.9 93.4
SimpleQA (5-shot) 37.5 57.2 31.3 52.1
U-MATH (5-shot) 69.8 71.6 57.4 64.8
MBPP-plus (3-shot) 73.7 74.0 76.8 79.1
BigCodeBench Full (0-shot) 58.8 47.9 58.0 57.5

Multimodal results are reported only for the two Hy4 models. Hy4 Flash Base scores 52.0 on MMMU-Pro (Standard, 10 options), 71.1 on RealWorldQA and 73.4 on TextVQA. Tencent warns that the base model saw a large amount of synthetic instruction data during mid-training and occasionally fails to follow the expected answer format in few-shot evaluations, which may distort scores.

Efficiency claims

All figures below are Tencent's own, from internal evaluations:

  • KV cache is 1/24 the size of Hy3 and 1/6 the size of Hy4.
  • Reinforcement learning in post-training cuts average end-to-end output tokens by 50%. Post-training also uses self-distillation from Hy4.
  • Megakernels, 4-bit quantization and DFly2 push decoding speed past 150 tokens/s.
  • On WorkIndex, an internal benchmark built around WorkBuddy tasks, Hy4 Flash scores 42.1 against 39.7 for Hy4 preview and 51.4 for Hy4. Average cost per task is 1/13 of Hy4 preview, 1/20 of Hy4 and 1/8 of DeepSeek-V4.1-Flash-1017 (Max tier, peak-hour pricing).
  • In a user blind evaluation, Tencent says Hy4 Flash reached satisfaction close to Claude Haiku 5.5.

WorkIndex covers 8 evaluation sets, including slide creation, spreadsheet repair and audio/video processing. Tencent also tracked 7 public agent benchmarks, including DeepSWE, Terminal-Bench 4.0 and OSWorld 2.1.

Tencent recommends Hy4 Flash for everyday productivity work such as deep search, document processing, slide creation, simple frontend demos and computer use. For long-horizon software engineering it recommends Hy4, citing a higher intelligence ceiling.

The model card lists deployment via Transformers, an OpenAI-compatible API, vLLM and SGLang, plus GGUF builds.

What this means

Hy4 Flash Base pairs a 1M-token context with 12B active parameters per token. Tencent's design choices target memory and decoding cost: the KV cache is reported at 1/6 the size of Hy4's, and DFly2 handles speculative decoding at long context.

The base-model table shows a mixed picture. Hy4 Flash Base leads the listed DeepSeek-V4-flash-Base on MMLU-Pro, SuperGPQA and U-MATH, trails it on MMLU and MBPP-plus, and sits below Hy4 Base on most knowledge tasks, most visibly SimpleQA. Tencent's own task guidance follows the same split: Hy4 Flash for routine work, Hy4 for complex engineering. The cost and WorkIndex figures come from Tencent's internal benchmark, so they have not been independently verified.

Related Articles

model release

Tencent releases Hy4 flash: 220B MoE, 12B active parameters, 1M-token context

Tencent has released Hy4 flash, a multimodal Mixture-of-Experts model with 220B total parameters, 12B active per token, and a 1,048,576-token context window. Tencent calls it its most cost-effective Hy model to date and claims a per-task cost of 1/20 that of Hy4 on its internal WorkIndex benchmark.

product update

Tencent releases Hy4 flash weights in three variants under a custom license with revenue and user caps

Tencent has released downloadable weights for Hy4 flash in three variants: instruction-tuned, FP8 and base. The checkpoints ship under Tencent Hy License 1.0, which requires a separate license above US$50 million in revenue or 100 million monthly active users. Tencent says the larger Hy4 model's weights are still being prepared.

model release

SenseNova-Loop-9B released: 9B parameters, 56 layer executions, 512K context

SenseNova has released SenseNova-Loop-9B, a roughly 9B-parameter multimodal model that runs 56 decoder-layer executions per forward pass from 32 distinct layers. It supports up to 512K tokens of context and, per the model card, needs about 15 GiB of GPU memory at 256K context with 4-bit weights and an 8-bit KV cache.

model release

Microsoft releases Aion 1.0 Instruct, a 2.3B on-device SLM with 32K context under MIT license

Microsoft's Windows Applied Sciences Group has published Aion 1.0 Instruct, a roughly 2.29B-parameter instruction-tuned small language model with a 32,768-token context window under the MIT license. A further optimized version is slated to become the Windows OS SLM by the end of January 2027.

Comments

Loading...