Artificial Analysis Index v5 drops Kimi K3.1 Pro from 4th to 9th; Gemini 4 Carbon takes #1 at 59

Artificial Analysis released Intelligence Index v5 on Tuesday, and Moonshot AI's Kimi K3.1 Pro dropped from 4th place to 9th on the new version, according to a Gizmodo report. Google's Gemini 4 Carbon now ranks #1 at a score of 59.

What changed in v5

Artificial Analysis called v5 "our biggest upgrade since v4 launched in January." According to the firm, 60% of the new test uses private questions, or "held-out answers." Gizmodo reads both changes as an effort to curb "benchmaxxing," the practice of preparing models for evaluations using old questions available in public repositories such as GitHub and Hugging Face.

Kimi K3.1 Pro's rise and fall

Kimi K3.1 Pro, from Beijing-based Moonshot AI, debuted last week and placed 4th on Intelligence Index v4. It also took the top spot on Artificial Analysis' Terminal Bench 4.0, which focuses on computer use.

The Decoder noted that had Moonshot released the model's weights at debut, it would have been the highest-scoring open-weight model by a wide margin, and the highest-scoring Chinese model. "That would shrink the gap between open and proprietary models from 12 to just 3 points," The Decoder wrote.

The weights were not released at debut. Moonshot has still not published them and will not do so until November 4. On v5, the model sits 9th. Artificial Analysis said in its announcement that "The open weights gap widens from 3 points to 12."

Gemini 4 Carbon takes the top spot

Artificial Analysis said Gemini 4 Carbon is the new #1 on Intelligence Index v5 with a score of 59, "driven by leading performance in complex reasoning." The firm also said it is second only to Claude Opus 5.5 on the Artificial Analysis Coding Agent Index.

Google announced Gemini 4 Carbon on Wednesday. The model was evaluated but has not been released. Google's blog post leads with the ranking: the second sentence reads, "Carbon ranks #1 on the recently updated v5.0 of Artificial Analysis Intelligence Index at 59."

Other explanations and context

Gizmodo notes there may be other reasons for the lower scores of Moonshot and other Chinese models. Physicist Stephen Hsu wrote on X that he believes U.S. models are outperforming Chinese ones in math and science. "I've been talking to a lot of Chinese labs recently and they have not prioritized this area at all," he wrote.

Gizmodo says Kimi K3.1 Pro's brief run created none of the panic that followed DeepSeek's release in January of last year. It adds that the result was still a setback for critics of the Chinese open-weight approach. Anthropic has expressed concern about open weights, writing in a report on one such model that because it was "released as an open-weight model, users can reconfigure it to remove its refusals with little change in its capabilities." Anthropic also criticizes model distillation, which Gizmodo says open-weight models often rely on to raise their capabilities. Nvidia CEO Jensen Huang has said distillation is just competition.

Gizmodo describes Artificial Analysis as a private firm, not a standardizing body or academic institution, founded, it appears, in 2023.

What this means

The same model moved from 4th to 9th when the benchmark version changed, and the open-weights gap went from 3 points to 12 according to Artificial Analysis. A model's rank depends heavily on how the test is built, including how much of it is private. Teams comparing models should record which index version a ranking comes from.

Kimi K3.1 Pro's weights are not yet public and are due November 4, so the open-weights comparison rests on a model that outside parties cannot yet inspect. Hsu's comment, which is his own view and not an Artificial Analysis finding, offers one possible explanation for the gap: uneven priorities across labs in math and science.

Gemini 4 Carbon's #1 ranking is based on an evaluation of a model that Google has announced but not released.