Tencent Open-Sources Hy4 Preview: 770B-Parameter MoE Model with 1M-Token Context

Tencent Releases Hy4 Preview, a 770B-Parameter MoE Model

Tencent's Hy Team has released Hy4 preview, a new flagship Mixture-of-Experts (MoE) language model with 770 billion total parameters and 49 billion activated per token. The weights are open-sourced under the Apache 2.0 license on Hugging Face, ModelScope, GitCode, and CNB, alongside an FP8-quantized version for faster inference.

Architecture

Hy4 preview uses a 78-layer backbone. The first layer runs a standard dense feed-forward network, while the remaining 77 layers use MoE blocks with 256 routed experts and 1 shared expert. Each token activates the top-8 routed experts plus the shared expert. A separate Multi-Token Prediction (MTP) layer — 10 billion total parameters, 0.7 billion activated — is built in for speculative decoding.

The attention mechanism, called Gated DeepSeek Sparse Attention (Gated DSA), incorporates an IndexCache for cross-layer sparse index reuse, an approach Tencent says draws inspiration from DeepSeek and GLM architectures. The residual pathway uses "identity Hyper-Connections" (iHC) across 4 residual streams to widen inter-layer information flow. The model has a hidden size of 6144, 64 attention heads, a vocabulary of 120,832 tokens, and supports a 1-million-token context length.

Benchmark Claims

Tencent did not publish standard third-party benchmark scores (e.g., MMLU, HumanEval) in the release notes. Instead, the company ran a blind side-by-side evaluation using 163 internal experts rating outputs across 203 engineering tasks. According to Tencent, Hy4 preview scored 2.99 on average versus GLM 5.3's 2.92 (46.8% wins, 12.8% ties, 40.4% losses) and versus Kimi K3's 2.94 (51.2% wins, 7.9% ties, 40.9% losses). These figures come from an internal, non-public evaluation and have not been independently verified.

Tencent positions the model for software engineering, office/document automation, game prototyping, and scientific research tasks, citing co-development with internal teams and products like CodeBuddy and WorkBuddy.

Known Limitations

Tencent describes Hy4 preview as an early release with "real headroom left in both pre-training and post-training." The company acknowledges the model spends longer than necessary reasoning through complex tasks and tends to over-verify its own work. The model defaults to a "high" reasoning effort with deep chain-of-thought; users can disable this via a reasoning_effort: no_think parameter for direct responses.

Deployment

Hy4 preview and its FP8 variant support deployment via vLLM and SGLang, with official Docker images provided (vllm/vllm-openai:hy4-preview, lmsysorg/sglang:hy4-preview). Pricing not yet disclosed — the model is distributed as open weights rather than through a hosted API.

What This Means

Hy4 preview is a substantial scale-up from Tencent's prior Hy3 preview, targeting the same open-weight frontier occupied by GLM and Kimi model families. The 770B-parameter size with only 49B activated keeps inference costs manageable relative to a dense model of similar capacity, while the 1M-token context and native speculative decoding layer signal a push toward long-horizon agentic and coding workloads. However, the reported benchmark wins come exclusively from Tencent's internal, unpublished evaluation rather than standardized public benchmarks, so independent verification is needed before drawing conclusions about how it stacks up against GLM 5.3 or Kimi K3 on broader tasks. The self-acknowledged over-verification and reasoning-length issues suggest this is a genuine preview, not a finished production model.