OpenAI's GPT-5.6 Sol Adds Five Reasoning Effort Settings, Follows DeepSeek-R1 RLVR Training Method

OpenAI released GPT-5.6 Sol last week, a new reasoning model family that comes in three sizes with roughly five to six reasoning-effort settings each. The release arrives nearly two years after OpenAI's o1 model first popularized LLM-based reasoning, and four months after DeepSeek-R1 detailed the reinforcement learning with verifiable rewards (RLVR) training recipe.

According to Sebastian Raschka, who analyzed the release, reasoning models have "become a standard part of modern model releases." The GPT-5.6 Sol family includes settings ranging from low to ultra effort, with the Ultra setting using four subagents to accelerate work at a similar effort level to Max.

Training Method Explained

The RLVR training approach, detailed in DeepSeek-R1's January 2025 release, provides a binary reward signal (0=incorrect, 1=correct) for verifiable domains like mathematics and code. The method uses symbolic math checkers like SymPy or WolframAlpha for math problems, and compilers or unit tests for code verification.

Notably, the intermediate reasoning trace itself is not used for training. DeepSeek-R1's paper reported that incorporating this information wasn't helpful for model training, so only the final answer and response format determine the reward.

Despite training only on output rewards, models learn to write intermediate explanations, backtrack, and self-correct. These self-correction moments are called "Aha" moments in the research.

Pure RL Versus SFT-First Approaches

DeepSeek-R1 demonstrated that reasoning behavior can be achieved with pure reinforcement learning through its R1-Zero variant, which applied RLVR directly to the pretrained base model without supervised fine-tuning (SFT). While R1-Zero proved the concept, it produced a weaker model than the full R1, which was trained from an SFT checkpoint.

Other models like Tülu 3 and Kimi K1.5 (both published on arXiv on January 22, 2025, the same day as DeepSeek-R1) applied reinforcement learning on top of supervised fine-tuned models. The term RLVR itself was coined two months earlier in the Tülu 3 paper.

Inference Scaling Adds Performance

Beyond training improvements, inference compute scaling offers another way to boost performance. Reasoning models inherently use more compute during inference by outputting longer token sequences compared to conventional LLMs.

Common inference scaling techniques include self-consistency, implemented as majority voting where the model is queried multiple times and the final answer is selected by vote. This can be applied to both conventional LLMs and reasoning models.

DeepSeekMath-V2 demonstrated extreme inference scaling by applying these techniques on top of a math-specialized reasoning model to achieve state-of-the-art performance on olympiad-level problems.

What This Means

The GPT-5.6 Sol release confirms that adjustable reasoning effort is becoming a standard feature in frontier models. The ability to dial compute up or down during inference gives users control over the cost-performance tradeoff. While OpenAI has not disclosed pricing or specific benchmark scores for the new models, the five-tier effort system suggests a wider range of use cases than previous releases. Most modern LLMs are now effectively reasoning models, trained using variants of the RLVR method that DeepSeek-R1 popularized.