GPT-6 Astra edits 0.016% of Qwen3.5-0.8B weights, lifting MATH-500 from 7.9% to 48.1%

MIT CSAIL researchers report that GPT-6 Astra edited 0.016% of the weights of Qwen3.5-0.8B and raised its MATH-500 score from 7.9% to 48.1%, with no gradient-based training. The work, published November 2, 2026 by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, is titled "Sparks of Weight-Space Programming." The authors describe the results as very early.

The approach

The authors define Weight-Space Programming as a general-purpose LLM reading and writing a network's weights directly, around a natural-language spec. This happens not through an optimizer and not with gradient descent, and sometimes with no environment feedback at all. Astra was not allowed to write training loops. According to the authors, it reasons about what each part of a network computes, or should compute, and changes the weights accordingly.

Headline result: Qwen3.5-0.8B

Asked to improve Qwen3.5-0.8B at math without gradients, Astra identified an issue in the weights that causes reasoning loops, which Qwen's own model card warns about. It fixed the issue by editing 0.016% of the weights in a Python kernel.

• MATH-500: 7.9% to 48.1%, with the same sampling settings.
• IFEval: raised by 3x, which the authors say the fix achieves out of the box.

Smaller experiments

Reading weights. In Experiment 1, the authors trained PPO policies on four grid games (Breakout, Pong, Pac-Man, Freeway). All four used the same 10×10 grid, five actions and an MLP of 500 → 32 → 32 → 5 with tanh (17,253 weights). Training ran for 1 billion steps, with the best of 3 seeds per game. Astra received the raw weights as text, with no game named and no options given. It named the correct game for all four. The authors say it keyed on features such as where each input channel's first-layer weights concentrate. For Breakout, the agent's weights lie along the bottom row, like a paddle. Astra was 99% confident on Breakout.

In Experiment 2, three DDPM diffusion models generated 12×12 grayscale circles, squares and triangles. Each was an MLP of 160 → 64 → 64 → 144 with ReLU (23,824 weights), trained with Adam for 40,000 steps at batch size 512. Astra received only the weights, with no shape named, and was told not to compute a forward pass. It identified each shape, but drew every one as an outline although the models generate filled shapes. The authors read this mistake as evidence that it does not rely on computing the forward pass.

Fixing weights. In Experiment 4, a trained control policy never got a car over a strip. Astra edited 3.5% of the weights in one shot, taking success from 0% to 100%.

Writing weights. The authors say Astra wrote small game-playing networks from scratch, without environment feedback, that outplay networks trained with PPO for 100 million steps. Experiment 5 covers three MinAtar games and Experiment 6 covers Go. Experiment 7 asks Astra which of two language models was trained to be sycophantic.

What this means

The results are from the authors' own experiments on small networks and one 0.8B-parameter language model. The authors describe them as early and have not presented them as a replacement for existing training methods. The Qwen3.5-0.8B edit is notable for its size and mechanism: 0.016% of weights changed, tied to a failure mode Qwen's model card already documents, and the gain carried over to IFEval without targeting it.

The authors argue that reasoning-driven optimization now reaches differentiable networks, and that this may matter for both optimization and interpretability. In their words, neural network weights "will soon be natively optimized by LLMs that deliberately reason about their weights." That is the authors' projection, not an established result. The Astra experiments use a single model, so whether other LLMs show the same ability is not addressed in the material above.