analysisTencent

Tencent Hy team says next model will be substantially larger, with Flash-Lite and 24GB-class options planned

TL;DR

Tencent's Hy team said in a public Reddit AMA that its next model will be substantially larger and that it will keep investing in large-scale reinforcement learning. It also described Flash-Lite, Hy4-mini and models for 24GB GPUs as roadmap items, not released products.

2 min read
0

Tencent's Hy team said its next Hy model will be substantially larger and that it will continue investing in large-scale reinforcement learning. The statements came in a public Reddit question-and-answer session advertised with Shunyu Yao, held in the early hours of November 21 in Shanghai, a day after the release of Hy4 flagship weights.

The answer details below come from a Chinese-language compilation published by Zhidx; 36Kr and the public Reddit AMA are also cited as sources. The AMA post confirms the session and its participants. These are development intentions, not a newly released checkpoint, a confirmed parameter count or a dated launch commitment.

Smaller models on the roadmap

At the smaller end, the team described Flash-Lite as planned work and discussed models aimed at devices with 24GB of GPU memory. Hy4-mini was also described as being on the roadmap, and a dense model in the 24-billion-to-31-billion-parameter range was under consideration. The answers do not establish that these are the same product, or that any proposed configuration is available now.

The 24GB figure refers to video memory. By itself it does not specify total system RAM, quantization, context length or achievable speed. Pricing is not yet disclosed, and no release dates were given for any of the smaller models.

Attention design: GQA, MLA and DSA

As summarized by Zhidx, the team discussed trade-offs between grouped-query attention (GQA) and multi-head latent attention (MLA):

  • MLA offered better model quality at a comparable key-value-cache size.
  • MLA's compute intensity complicated decoding.
  • Introducing the DSA sparse-attention mechanism reduced that burden and changed the team's preferred balance.

Mixture-of-experts sparsity

The team cautioned against assuming that greater mixture-of-experts sparsity always improves deployment. According to the summary, an excessively sparse model can occupy substantial GPU memory without fully using available compute. Reducing active parameters while holding total size fixed can also harm quality. These are design explanations attributed to the team, not universal benchmark results for every implementation.

What this means

The AMA gives a clearer account of Hy's research priorities and engineering choices. It is not a new release, and it does not make Hy4 weights newly downloadable.

For developers, the two stated directions point different ways: a substantially larger next model backed by large-scale reinforcement learning, and smaller options (Flash-Lite, Hy4-mini, a possible 24B-to-31B dense model) aimed at constrained hardware. The 24GB target is only useful for planning once quantization, context length and speed figures are published.

The attention and sparsity answers explain why the team weighs decoding cost and GPU memory use alongside model quality. Those are the same constraints that determine whether a model fits the 24GB class the team described.

Related Articles

model release

Tencent releases Hy4 flagship weights: 770B-parameter MoE, 1,048,576-token context, conditional license

Tencent released downloadable weights for its flagship Hy4 on November 20, with a System Card. The multimodal MoE has a 770-billion-parameter language backbone, 49 billion active parameters per text token and a 1,048,576-token context window, under a license with commercial conditions for model-as-a-service operators.

product update

Tencent opens WorkBuddy Computer Use beta for Windows and macOS apps that lack APIs

Tencent announced a public beta of WorkBuddy's Computer Use feature on November 20. It lets the office agent operate desktop applications that have no API, connector or browser version, on Windows and macOS, and requires WorkBuddy client 5.11.0 or later.

model release

Tencent releases Hy4 flash: 220B MoE, 12B active parameters, 1M-token context

Tencent has released Hy4 flash, a multimodal Mixture-of-Experts model with 220B total parameters, 12B active per token, and a 1,048,576-token context window. Tencent calls it its most cost-effective Hy model to date and claims a per-task cost of 1/20 that of Hy4 on its internal WorkIndex benchmark.

model release

Tencent releases Hy4 Flash Base: 220B-parameter MoE, 12B active, 1M-token context

Tencent has published Hy4 Flash Base on Hugging Face: a 220B-parameter mixture-of-experts model with 12B active parameters per token, text and image input, and a 1M-token context. Tencent calls it its most cost-effective Hy model to date.

Comments

Loading...