Simon Willison's Pelican Benchmark Shows GPT-6 Astra Outperforming GPT-5.6 Sol at Every Reasoning Level

What happened

Developer Simon Willison published a comparison grid testing OpenAI's GPT-6 Astra against GPT-5.6 Sol, Terra, and Luna using his long-running informal benchmark: generating SVG images of a pelican riding a bicycle. Willison says he gained access to Astra on September 4, 2026, and immediately ran it through the test at five reasoning levels — low, medium, high, xhigh, and max. Notably, Astra does not support a "reasoning=none" setting, unlike some other models in OpenAI's lineup.

The results

According to Willison, the quality gap was stark. The best GPT-5.6 Sol output — he preferred the xhigh setting over max — still read as "a bunch of abstract shapes." Every single Astra output, even at the lowest reasoning level, reportedly looked better than that. The Astra "max" setting produced what Willison called "really good" results, though he noted that Astra settings below max still struggled to consistently place the pelican's legs on both sides of the bicycle frame.

On cost, Willison reports Astra pricing at $10 per million input tokens and $50 per million output tokens, versus $5/$30 for Sol — roughly double. However, he claims Astra uses significantly fewer tokens at each reasoning level, narrowing the effective cost gap. Per Willison's numbers, an Astra "low" generation cost 9.55 cents and beat every Sol output at any reasoning level and any price point up to a dime.

The token-count anomaly

Willison also flagged an unexplained pattern in input token counts: Astra and Luna both consumed 16 input tokens for the same prompt, while Sol and Terra both consumed 26. He speculated this could indicate that Astra and Luna share more architectural lineage than OpenAI has publicly disclosed — though this remains Willison's own inference, not a confirmed fact from OpenAI.

Context

This is not a formal or standardized benchmark. Willison's pelican-SVG test is a widely-followed but informal proxy he has used across many model generations to gauge relative visual/spatial reasoning and instruction-following in code generation. It has no official scoring rubric, and results depend on subjective visual judgment of a single rendered image per model per setting.

What this means

Willison's test is anecdotal, not a controlled benchmark, and covers a single narrow task — SVG code generation for one specific prompt. But the pattern is consistent with what's expected from a next-generation flagship model: better output quality at every configuration, paired with higher per-token pricing offset by lower token consumption. The token-count coincidence between Astra and Luna is the more interesting finding for engineers tracking OpenAI's model family relationships, though it requires independent verification before drawing architectural conclusions. No official OpenAI benchmark data, model card, or pricing page accompanies this report — all figures originate from Willison's own testing.