Huawei Ascend validates LoRA fine-tuning of 2.8T-parameter Kimi K3.1 Pro on 128-card Atlas 950 SuperPoD

Huawei's Ascend computing team says it has adapted Moonshot AI's Kimi K3.1 Pro for LoRA fine-tuning and validated training on a 128-card Atlas 950 SuperPoD. The claim comes from Guandian and a Chinese-language technical write-up published on November 6 and 7, as reported by Pandaily. This is a training result, separate from the day-zero inference support Ascend announced after Moonshot released the model's weights on November 4.

The model

Kimi K3.1 Pro is an open-weight, natively multimodal mixture-of-experts model with about 2.8 trillion total parameters. The reports describe:

• 896 routed experts, with 16 activated per token
• A hybrid attention design in which roughly three quarters of attention blocks use KDA linear attention alongside gated sparse attention
• A context window of up to one million tokens

The BF16 weights alone occupy roughly 5.6TB, before gradients, optimizer states and long-sequence activations are counted.

Parallelism and memory

Ascend says it combined FSDP2 sharding of parameters, gradients and optimizer states with expert parallelism, which distributes experts across accelerators. Validation used an FSDP2 plus EP64 configuration on the 128-card system.

Chunked loss computation along the sequence and vocabulary dimensions was used to contain memory for the model's large vocabulary.

Operator and kernel work

The team says it built large fused forward and backward kernels for KDA attention and used fused AscendC operators for sparse attention. Ascend claims more than twice the performance of the earlier small-operator approach.

Other changes it describes:

• New residual structures were brought up through the Triton Ascend framework and the BiSheng compiler.
• MoE routing steps such as token permutation and grouped matrix multiplication were fused and overlapped with communication.
• A selective caching mechanism called Swap Core keeps the outputs of chosen matrix-multiplication operators in a tiered store spanning HBM and DDR memory.

Ascend says Swap Core raised average training performance by 15% in the current test configuration. It did not publish absolute throughput, and the figure has not been independently verified.

Reported loss curves

The team reported loss curves from the 128-card runs:

Run	Starting loss	Reported loss later	Steps
Training from random initialization	near 13.5	about 1.7	more than 400
LoRA fine-tuning from open weights	near 1.85	roughly 0.27	more than 500

Ascend also says an agent-driven development workflow allowed the model to be brought up within days.

What this means

The reported result shows that a 2.8-trillion-parameter MoE model with a one-million-token context window can be trained, both from random initialization and with LoRA, on a 128-card Atlas 950 SuperPoD using FSDP2 plus EP64. The 5.6TB BF16 weight footprint explains why the team's work centers on sharding, expert distribution and tiered HBM and DDR caching rather than on LoRA itself.

The evidence is limited to Ascend's own reporting. The 15% gain from Swap Core and the more-than-twofold operator speedup are company claims, and no absolute throughput figures were given. The loss curves show that training converged in the reported runs, but they are not downstream task evaluations. Builders considering Ascend hardware for Kimi K3.1 Pro fine-tuning should wait for throughput numbers and independent reproduction before treating the performance claims as established.