AWS ships aws-ai-ml skill so Kiro, Claude Code and Codex can benchmark SageMaker inference endpoints

Amazon SageMaker AI has released the aws-ai-ml skill, an agent skill that gives MCP-compatible coding agents, including Kiro, Claude Code and Codex, the ability to benchmark inference endpoints, recommend deployment configurations, compare performance runs and generate executable SageMaker Python SDK v3 code. It is distributed through the Agent Toolkit for AWS. This is a tooling release, not a new model.

What the skill does

According to AWS, the skill turns any agent that supports the Model Context Protocol (MCP) into a SageMaker AI inference optimization assistant. The user describes a goal in natural language, such as a latency target, a cost ceiling or a model to evaluate. The agent asks clarifying questions and produces code the user can review, modify and run in their own environment. AWS says the code is grounded in real benchmark and measured performance data.

The documented capabilities include:

• Benchmarking an existing endpoint. The agent generates a Python notebook that uses the Workload.synthetic() and start_benchmark() APIs from the SageMaker Python SDK. It first confirms the endpoint is safe to load-test, since benchmarking sends real traffic to a live endpoint.
• Performance reporting. Reports cover throughput (requests per second, output tokens per second), latency (p50, p99, time-to-first-token, inter-token latency) and concurrency. AWS states these are measured values from real load, not estimates. The agent may also suggest improvements such as prefill decoding.
• Instance-type selection. Users can supply a fine-tuned model in Amazon S3 (by S3 URI), a SageMaker JumpStart model ID (the example given is huggingface-reasoning-qwen3-8b) or a Hugging Face Hub model name. The source text cuts off while describing the Hugging Face Hub path, including gated-model handling.

Setup

There are two routes:

1. Local install. This requires AWS CLI 2.35 or later and uv. Run aws configure agent-toolkit to detect agents, install skills and configure the AWS MCP Server. Then run npx skills add aws/agent-toolkit-for-aws/skills/aws-ai-ml. According to AWS, Kiro and Claude Code can also discover and load skills at runtime through the AWS MCP Server without a local install.
2. SageMaker Studio JupyterLab. Users select a pre-configured image that ships with the skill. The space must be set to Private, because skills only sync on private spaces. First boot takes 5–10 minutes. AWS recommends a fresh space, since a reused space with a locally modified skill may not pick up the image. Kiro authentication uses kiro-cli login.

The skill generates code that runs under the user's own AWS credentials. Those credentials need permission to call SageMaker AI APIs, including creating endpoints and running benchmark and recommendation jobs. AWS says no additional IAM configuration is needed for the skill itself. AWS claims users can go from zero to a working conversation in 10 minutes.

Pricing for the skill itself is not stated in the source. Costs from endpoints and benchmark jobs the generated code creates would be incurred in the user's account.

What this means

The notable choice here is packaging. AWS is shipping SageMaker expertise as a skill for third-party agents, including Claude Code and Codex, rather than only through its own Kiro. That puts AWS inference tuning where developers already work.

The design also addresses a real problem. Picking an instance family and serving container for a given model is hard, and recommendations backed by measured throughput and latency are more useful than generic advice from a general-purpose coding agent. Because the output is SDK v3 code rather than an opaque UI action, teams can audit and version-control what the agent does.

The main caveat is operational. Load tests hit live endpoints, so the agent's safety check matters, and teams should still gate production benchmarking. AWS has not published comparative data on how much the skill improves cost or latency outcomes, so those gains remain unquantified.