researchMicrosoft

Microsoft Research's TokenEconomizer plugin cuts coding agent costs 18.4% to 30.9% on SWE-Bench

TL;DR

TokenEconomizer, an open-source plugin from UC Berkeley, Microsoft Research and Microsoft Security, learns from an agent's past sessions in a repository to cut token costs. Across 277 SWE-Bench tasks and five agent and model combinations, mean cost per task fell 18.4% to 30.9%, according to the authors.

4 min read
0

TokenEconomizer, an open-source plugin from UC Berkeley, Microsoft Research, and Microsoft Security, lowered mean cost per task by 18.4% to 30.9% across 277 SWE-Bench bug-fixing tasks, according to a Microsoft Research blog post and accompanying paper. The plugin learns from an agent's completed sessions in a repository and applies that knowledge to later sessions, without modifying the agent or its underlying model.

The paper is titled TokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents.

Where the tokens go

The authors first profiled Claude Code on 60 SWE-Bench tasks, running each repository's tasks in sequence. They report three recurring sources of waste:

  • Rediscovering facts. After a repository had at least five completed tasks, 30% of tool calls repeated an action from an earlier task in that repository. Only 7% matched an action in another repository's history.
  • Unused tools. The agent received 23 tool descriptions on every turn, but 16 were never called.
  • Failure loops. The 17% of tasks that failed accounted for 32% of spending.

The authors also tested three open-source plugins, each built around a fixed cost-saving strategy. Each lowered costs in some repositories but not others.

How it works

After each session, TokenEconomizer reads the session log and updates what it knows about the repository. It then acts through three mechanisms, using only interfaces the agent already provides:

  • Micro-skills: one-line, repository-specific notes. They are added to the agent's starting context only after several sessions confirm a pattern, and are retired if later evidence contradicts them.
  • Tool pruning: removes tools that repository history suggests the agent is unlikely to use. Core tools for running commands and editing files are always retained, and removed tools can be restored.
  • Failure-loop intervention: a judge model decides whether the agent is truly stuck. If so, the agent gets a short note suggesting a different approach. In the open-source plugin, if the agent is still stuck after the note, TokenEconomizer alerts the developer rather than stopping the work itself.

Results

The evaluation covered 277 SWE-Bench tasks from six repositories and five agent and model combinations. TokenEconomizer started each repository with no prior session history. The authors report:

  • Mean cost per task fell 18.4% to 30.9%. The post says TokenEconomizer saved the most in all 30 repository-combination pairs without performance loss.
  • Pass@1 rose in four of the five combinations, and task success rates remained broadly stable.
  • Lowering the model's reasoning effort saved similar or larger amounts but led to substantial performance degradation.
  • Early tasks can cost more while the plugin learns, but with Claude Code, cumulative costs fell below the default in every repository by the sixth task.
  • Each mechanism saves costs on its own, and combining all three gives the largest reduction.
  • On sequential web development (Web-Bench), costs were 16.0% to 21.2% lower.
  • On security auditing, where every task is in a new repository, costs were 6.4% to 19.6% lower.

Reported savings include the cost of TokenEconomizer's own model calls.

Availability

The plugin is open source and designed to run alongside existing coding agents. According to the authors, it requires no API key of its own and keeps what it learns on the developer's machine. The paper, code, and project page are linked from the Microsoft Research post. Authors include Aaron J. Li, Weijia Xu, Chandan Singh, Zixiao Chen, Jeevana Priya Inala, Bin Yu, Ion Stoica, Michel Galley, and Jianfeng Gao.

What this means

The profiling data points to the same conclusion for repeated work in one repository: much agent spend goes to repeated actions, unused tool descriptions, and failed tasks. Because only 7% of tool calls matched another repository's history, versus 30% within the same repository, the authors argue that cost control should adapt to each repository rather than apply one fixed strategy. That fits their finding that three fixed-strategy plugins helped in some repositories and not others.

The comparison with reduced reasoning effort matters for anyone weighing cost against quality. That approach saved similar or larger amounts but degraded performance, while TokenEconomizer's savings came with broadly stable success rates in the authors' tests.

All headline figures come from the authors' own evaluations on SWE-Bench, Web-Bench, and security auditing tasks. The plugin works through interfaces agents already expose, so adopting it does not require changing the agent or model.

Related Articles

analysis

Nadella says Claude Code's CLI caught Microsoft off guard and the IDE "kind of faded away"

Microsoft CEO Satya Nadella said Anthropic's Claude Code moved coding from the editor to the command line, and that within about six months the IDE "kind of faded away." In an hour-long interview with Gergely Orosz, he also covered verification of AI-written code, agent risk, local models and Microsoft's workforce changes.

research

Amazon releases Mitra-v2, an open tabular foundation model that leads the single-model field on TabArena

Amazon has released Mitra-v2, a tabular foundation model for classification and regression that, according to Amazon, leads the single-model field on the TabArena benchmark of 51 datasets. Weights and code are available under Apache-2.0 on Hugging Face.

research

Together AI says CUDA Graphs and PDL recover 98% of megakernel performance without custom kernels

Together AI reports that CUDA Graphs and PDL on top of default Torch Inductor kernels recover 98% of its earlier megakernel performance. With "multi-kernels" and agent loops, it claims up to 1.7x over its old megakernels, reaching over 1100 TPS for Llama 1B on H100. The work led to Together Compiler, a framework for continual AI inference optimization.

changelog

VS Code 1.146 deprecates the Local harness for Copilot agent sessions, retirement set for February 2027

Visual Studio Code 1.146 deprecates the Local harness, the older software that coordinates the model, context, and tools for GitHub Copilot agent sessions in the editor. GitHub says the harness retires in February 2027.

Comments

Loading...