IBM's ALTK-Evolve lifts GPT-4.1 to 67.5% Pass^4 on τ²-bench Retail without weight updates

IBM Research reports that ALTK-Evolve, a learned-memory method, lifted a basic ReAct customer-service agent built on GPT-4.1 to 67.5% Pass^4 on τ²-bench retail. According to IBM, that is above every retail entry in its October 12, 2026 snapshot of the τ²-bench leaderboard. The result has not been independently verified.

How ALTK-Evolve works

ALTK-Evolve analyzes an agent's attempts on training tasks. It then uses the same model, GPT-4.1, to distill successes and failures into short behavioral rules, such as how to authenticate a customer, confirm a transaction, or process a return. The rules are appended to the system prompt as a plain markdown list after the domain policy.

IBM says the pipeline needs no weight updates and no human annotation. The scaffold is τ²-bench's stock two-phase THINK → ACT ReAct agent. Policy, tools and scaffold are unchanged.

Benchmark setup

τ²-bench, from Sierra, pairs an agent with an LLM-simulated customer, a set of domain tools and a written policy. A task counts as solved only if the final database state matches the expected outcome. IBM used the retail and airline domains with these settings:

• Agent model: gpt-4.1
• User simulator: gpt-5.5
• Trials: 4 per task
• Held-out test tasks: 40 retail, 20 airline
• Headline metric: Pass^4, the share of tasks solved on all four trials, which is what the leaderboard reports

Results

IBM tested three configurations. Baseline uses no guidelines. "sup3" keeps only guidelines extracted from at least 3 distinct training tasks (31 retail, 14 airline). "Lossless" keeps every retained guideline (292 retail, 101 airline).

Domain	Config	Mean Accuracy	Pass^4
Retail	baseline	78.1%	57.5%
Retail	sup3	78.8%	60.0%
Retail	lossless	82.5%	67.5%
Airline	baseline	57.5%	35.0%
Airline	sup3	60.0%	45.0%
Airline	lossless	65.0%	40.0%

In retail, more guidance helped on every metric. The lossless set added 4.4 points of mean accuracy and 10.0 points of Pass^4. In airline, lossless led on mean accuracy (+7.5 points), but the compact sup3 set was more reliable, with Pass^4 up 10.0 points against 5.0 for lossless. IBM concludes that guideline volume should be tuned per domain rather than hard-coded.

IBM says the gains come from better-informed decisions rather than longer conversations. It reports that the cost lands in prompt input tokens, not in extra turns. The cost table is truncated in the source material available to us, so per-conversation cost figures are not yet confirmed here.

Caveats

• The test sets are small: 40 retail tasks and 20 airline tasks. On 20 tasks, one task equals 5 percentage points of Pass^4.
• The leaderboard comparison is against a specific October 12, 2026 snapshot, and IBM did not claim a top airline position.
• GPT-4.1 handles both guideline extraction and the agent, and the user simulator is gpt-5.5, so results may not transfer to other model pairings.

What this means

The result suggests that, for policy-bound tasks, a cheap, non-reasoning model plus a curated prompt-level memory can match or beat entries built on more capable models. That matters for teams weighing prompt-side learning against fine-tuning or a larger model. Because the mechanism is a markdown block in the system prompt, it is inspectable and easy to roll back, which are real advantages in regulated customer-service settings.

The airline numbers show the limits. Gains did not scale monotonically with guideline count, and the small sample makes the differences noisy. Teams should treat support-count thresholds as a parameter to validate on their own data, and independent replication on the full leaderboard protocol would strengthen the claim.