IBM's CUGA harness passes 81.1% of AppWorld tasks on GLM-5.3-flash; Hermes 46.7%, Deep Agents 30.0%

IBM Research reports that changing only the agent harness moved task completion on AppWorld from 30% to 81% with the same open-weight model. In its controlled comparison, IBM's open-source CUGA (ConfigUrable Generalist Agent) passed 81.1% of tasks, against 46.7% for Hermes and 30.0% for Deep Agents. IBM built CUGA and published the results on the Hugging Face blog on November 3, 2026.

The experiment

IBM ran three harnesses on AppWorld, a benchmark of multi-step tasks across 9 apps and 457 APIs, including email, payments and file storage. Every harness used GLM-5.3-flash on the same 90 tasks in 30 scenarios, with one attempt per task. All runs used the Harbor evaluation framework, and IBM's setup is public in harbor-eval, its Harbor fork.

Harness	TGC (tasks passed)	SGC (scenarios with all 3 variants passed)
CUGA	81.1 (73/90)	66.7 (20/30)
Hermes	46.7 (42/90)	26.7 (8/30)
Deep Agents	30.0 (27/90)	6.7 (2/30)

The setups were not identical. CUGA used its AppWorld configuration: AppWorld instructions, task decomposition, verification before execution and automatic reflection. Hermes used Harbor's default setup with AppWorld tools over MCP. Deep Agents used AppWorld tools over MCP plus LangChain's built-in middleware, configured for 469 tools (tool selection, context summarization, a to-do list, retries and call limits). IBM also added two small fixes so malformed GLM replies did not end runs.

IBM says the SGC metric is the most useful, because a scenario passes only when all three of its variants pass, which rewards consistency over a lucky run.

What CUGA does differently

IBM attributes the gap to the harness, since the model was constant. CUGA runs a loop of prepare, call the model, execute and repeat. The AppWorld features IBM lists:

• Plan: task decomposition one app at a time, focused tool sets per step rather than all 469 tools, and a tool shortlister (find_tools).
• Act: CodeAct, where the model writes Python against the app APIs, run in a restricted sandbox with a time limit.
• Check before acting: verification of each data-changing code block before it runs.
• Check after acting: automatic reflection, plus a pagination audit that flags list calls where the next page was never requested.

Policy enforcement

IBM also describes CUGA policies, covered in its ACM CAIS 2026 paper, Governance by Construction for Generalist Agents. They act at four points: Intent Guard (user intent), Playbook (agent guidance), Tool Approval and Tool Guide with ToolGuard (tool execution), and Output Formatter (final response). IBM says guidance steers the model, while approvals and ToolGuards are enforced by the runtime. In its Policy Playground demo, for the message "Pay everything I owe.", the agent without policies paid $90, including a denied claim in collections. The agent with policies held the payment for approval, then paid only the $10 that was due.

IBM also claims that, with hosted models, CUGA held #1 on WebArena (February to September 2025) and on AppWorld (July 2025 to February 2026).

Stated limits

IBM lists three: the sample is 90 tasks on one benchmark, the setups were not identical, and the runs did not use policies, so they do not measure policy enforcement. IBM also says the result is not a verdict on Hermes (Nous Research, built for always-on personal assistants) or Deep Agents (LangChain, built for long multi-step tasks such as research), since neither was built for multi-app API work. AppWorld uses consumer apps, so IBM calls it a proxy for enterprise automation.

IBM cites a separate analysis, via best-of-Agent-Harnesses, in which GLM-5.2 went from 23% to 52% on SWE-bench Pro when only the harness changed. Community lists track nearly 200 harnesses.

What this means

The result is a vendor-run comparison: IBM authored CUGA and chose the benchmark, and CUGA had a task-specific AppWorld configuration. It is best read as evidence that harness design, with the model held fixed, can account for a large share of agent performance on multi-app API tasks. The comparison covers 90 tasks and a single attempt per task. The policy features, which IBM presents as the main enterprise differentiator, were not exercised in the benchmark, so the AppWorld numbers say nothing about them. Teams evaluating harnesses should test on their own workloads and models, which is the question IBM itself says matters.