AWS Launches Framework-Agnostic Agent Evaluation via OpenTelemetry in Bedrock AgentCore

What's new

Amazon has extended Bedrock AgentCore Evaluations to work with any agent framework that emits OpenTelemetry-compliant telemetry, removing a long-standing constraint where evaluation tooling only worked with agents built using a specific SDK or tracing pattern.

According to AWS, the service now supports Strands Agents, LangGraph, the OpenAI Agents SDK, LlamaIndex, Google ADK, and the Claude Agent SDK out of the box, with most frameworks covered through both OpenTelemetry and OpenInference instrumentation. Coverage also extends to any instrumentation library that follows the same naming conventions, even those not explicitly listed.

How it works

AgentCore Evaluations reconstructs an agent's session from telemetry stored in Amazon CloudWatch, routed there by the AWS Distro for OpenTelemetry (ADOT) running on Bedrock AgentCore runtime. A session is grouped by a session.id attribute; within it, each user turn is a separate trace made up of spans.

The service reads three span roles to score an interaction:

• Invoke agent spans — carry the user prompt and final agent response
• Inference spans — carry the message history and model reply for each model call
• Execute tool spans — carry the tool name, input parameters, and result

Other span types — retrieval, reranking, guardrail checks, memory operations, orchestration steps — are passed through as additional context but aren't required by the evaluators. AWS says this design is forward-compatible: unfamiliar span kinds are skipped rather than causing errors, so new span types added to future conventions won't break existing evaluation pipelines.

Framework detection happens automatically through a scope.name attribute that every OpenTelemetry instrumentation library stamps on its spans. Any library whose scope name falls under the opentelemetry.instrumentation.* or openinference.instrumentation.* prefixes is read through a generic path, without requiring configuration changes to agent code. AWS notes that a scope name outside these prefixes — even if its spans otherwise follow the conventions — will not be picked up.

Once a session is reconstructed, the same evaluators apply regardless of framework: GoalSuccessRate, Correctness, Helpfulness, and custom LLM-as-a-judge metrics score every agent identically.

Requirements

AWS lists two conditions for end-to-end evaluation. First, spans must carry a session.id attribute matching the runtimeSessionId used to invoke the agent — handled automatically by ADOT on AgentCore runtime. Second, the underlying data source must include actual message content, not just span metadata. For newly created agents using AgentCore's default unified observability setup, message content and spans live in the same log group, satisfying this requirement without extra setup.

What this means

This is a plumbing fix, not a new capability in the sense of new evaluation metrics or scoring methods. The value is in decoupling evaluation from framework lock-in: teams that mix LangGraph for orchestration, LlamaIndex for retrieval, and Claude Agent SDK for specific tasks no longer need separate evaluation pipelines or custom instrumentation for each.

The approach also signals where AWS is betting the ecosystem is headed — toward OpenTelemetry and OpenInference as de facto standards for agent observability, rather than proprietary tracing formats tied to individual SDKs. Any instrumentation author can opt into AgentCore Evaluations simply by conforming to the scope-name convention, which effectively outsources framework support to the broader open-source instrumentation community instead of requiring AWS to build and maintain per-framework adapters.

The practical limitation is that agents without OpenTelemetry-compliant instrumentation, or those using AgentCore runtime without unified observability enabled, will need configuration changes to benefit. Teams running custom or in-house agent frameworks with non-standard telemetry naming won't get automatic coverage unless they adopt the conventions.