AWS Launches AgentCore Optimization: Automated Performance Loop for Production AI Agents

Amazon Web Services released AgentCore Optimization in preview, introducing an automated performance loop that generates configuration recommendations from production traces, validates them through batch evaluation and A/B testing, and enables continuous agent optimization without manual prompt tuning cycles.

Core Capabilities

The system introduces three linked components:

Recommendations: Analyzes production traces and evaluation outputs to optimize system prompts or tool descriptions based on a specified evaluator. The service reflects on OpenTelemetry-compatible traces and generates targeted improvements for metrics like goal success rate, tool selection accuracy, helpfulness, and safety.

Batch Evaluation: Tests recommendations against pre-defined datasets and reports aggregate scores to catch regressions on known use cases. Teams can wire batch evaluation into CI/CD pipelines to block configuration changes that fail existing test cases. The system supports simulated datasets using LLM-backed actors when hand-authored scenarios are insufficient.

A/B Testing: Runs controlled comparisons between agent versions through AgentCore Gateway, splitting live production traffic at configurable percentages and reporting results with confidence intervals and p-values for statistical significance.

How the Loop Works

AgentCore Observability captures every model call, tool invocation, and reasoning step as OpenTelemetry traces. Evaluations score those traces across multiple dimensions using built-in evaluators, ground-truth comparisons, or custom LLM-as-judge scoring.

Developers point the Recommendations API at the CloudWatch Log group containing agent traces, select a reward signal (the evaluator to optimize for), and choose whether to optimize the system prompt or tool descriptions. The service generates a recommendation without modifying tool implementations.

Configurations ship as immutable, versioned bundles keyed by runtime ARN, containing model ID, system prompt, and tool descriptions. Agents read their active configuration dynamically at runtime through the AgentCore SDK, making prompt or model changes configuration updates rather than code deployments.

Developers create one bundle for the current configuration and another for the recommendation, then validate offline through batch evaluation before running an A/B test against live traffic. When data provides adequate confidence in the new version's performance, developers stop the test and promote the winning variant.

Production Use Cases

According to AWS, Yoshiharu Okuda, Head of Generative AI Business Strategy at NTT DATA, stated that processes requiring weeks of manual prompt tuning evolved into rapid, repeatable cycles through AgentCore. Masashi Shimizu, Senior Managing Director at Nomura Research Institute, claims what took weeks of manual iteration is now a repeatable cycle that compounds with each improvement.

The current preview is developer-triggered by design. Developers choose when to generate recommendations, which evaluator to target, and whether to promote results.

What This Means

AWS is positioning AgentCore as infrastructure for the complete agent lifecycle, moving beyond build and deploy to systematic optimization. The automated recommendation system addresses the quality drift problem where agents degrade as models update and usage patterns shift, replacing manual trace analysis and hypothesis-driven fixes with data-backed optimization.

The integration of A/B testing at the infrastructure layer is significant. Most teams currently test agent changes through manual review or basic success metrics. Statistical significance testing with confidence intervals brings production rigor to agent development, though effectiveness depends on traffic volume and metric sensitivity.

The roadmap indicates AWS plans to automate more of the loop: recommendations targeting multiple evaluators simultaneously, automatic recommendation triggers when evaluators drop below thresholds, and expansion to optimize agent skills based on production usage patterns. The current design keeps humans in the approval loop while automating the evidence gathering.