IBM Research's ALTK-Evolve Matches or Beats ACE Agentic Memory at Up to 86% Lower Token Cost

IBM Research has published benchmark results showing its ALTK-Evolve agentic memory system matches or beats a comparable system called ACE (Agentic Context Engineering) on task accuracy while using dramatically fewer tokens per task — in one case, roughly one-seventh the inference cost.

Both systems let an LLM agent learn from its own past trajectories without weight updates or human labels, converting failures on multi-step tasks into reusable "lessons." Both also agree on a core design principle: don't compress those lessons into a short summary. ACE tracks failure modes it calls "brevity bias" and "context collapse," keeping a rich, itemized playbook with helpful/harmful counters per bullet. ALTK-Evolve keeps a similar record via "support counts" — a tally of how many independent episodes produced each guideline.

The systems diverge on delivery. ACE injects its entire consolidated playbook into the model's context on every single step, regardless of model or task. ALTK-Evolve treats delivery as an adjustable parameter: a small fixed core of high-support guidelines, extended with a handful of task-specific lessons selected via cosine similarity or LLM-guided ranking, or the full consolidated set when a model has the context headroom to use it.

Benchmark results

IBM Research ran both systems in-house on AppWorld's test_normal split (168 tasks), using the same ReAct code agent as the base, and reported the following:

• DeepSeek-V3.2: ALTK-Evolve scored 89.3 Task Goal Completion (TGC) / 80.4 Scenario Goal Completion (SGC) using 263K tokens per task, versus ACE's 80.4 TGC / 73.2 SGC at 634K tokens per task — roughly 40% of ACE's token cost with higher accuracy.
• gpt-oss-120b: ALTK-Evolve scored 56.0 TGC / 37.5 SGC at 116K tokens per task, versus ACE's 54.8 TGC / 35.7 SGC at 777K tokens per task — about one-seventh the token cost, with IBM describing the accuracy difference as within run-to-run noise (a repeat run of ALTK-Evolve landed at 54.8, matching ACE).

A breakdown by task difficulty shows the accuracy source varies by model. On gpt-oss-120b, ACE's full playbook led on Easy and Medium tasks, but ALTK-Evolve's selective retrieval won on Hard tasks and the aggregate score. On DeepSeek-V3.2, ALTK-Evolve led Easy, Hard, and Overall, with ACE only edging out Medium.

IBM says it applies different delivery configurations depending on model strength: the full consolidated guideline set for the stronger DeepSeek-V3.2, and selective retrieval for the weaker gpt-oss-120b, arguing that a large injected context can overwhelm a weaker model rather than help it.

On the memory-building side, ACE uses a Generator → Reflector → Curator loop with incremental delta updates and embedding-based de-duplication. ALTK-Evolve clusters near-duplicate lessons and merges them in a support-conserving way, so a merged guideline retains the combined evidence count. It also extracts typed guidelines — strategy, recovery, and optimization — with provenance tracking back to source trajectories at subtask granularity, which IBM says enables transfer of a lesson learned in one app to another.

IBM Research notes the ACE comparison numbers are its own in-house runs of the ACE agent on identical AppWorld splits and base models, not figures published by ACE's original authors. The company says a follow-up post will detail how much guideline injection helps across different model capability levels. The ALTK-Evolve library, including the extraction, consolidation, and retrieval pipeline, is available along with a technical report detailing the full method and ablations.

What this means

This is an infrastructure-layer optimization for agentic memory rather than a new model release — no new weights, just a smarter way to decide what context to feed an existing LLM at inference time. The headline number that matters for builders is the token bill: if ALTK-Evolve's claims hold up outside IBM's own benchmark runs, teams running memory-augmented agents at scale could cut inference costs by 60-85% without sacrificing accuracy, simply by retrieving relevant guidelines instead of always injecting a full playbook. The finding that weaker models perform worse with more context, while stronger models benefit from more, is a practical signal for anyone tuning agent memory systems — the optimal delivery strategy isn't universal, it depends on how much context the underlying model can actually use productively. As with any single-lab benchmark comparison, independent replication on AppWorld or other agentic benchmarks would strengthen the claims.