product updateAmazon Web Services

Amazon Bedrock adds programmatic tool calling to reduce latency and token usage in multi-step workflows

TL;DR

Amazon Bedrock now supports programmatic tool calling (PTC), a technique that allows LLMs to generate Python code for multi-step tool orchestration rather than making sequential API calls. AWS offers three implementation paths: self-hosted Docker sandboxes on ECS, managed execution via Amazon Bedrock AgentCore Code Interpreter, and Anthropic SDK-compatible proxy integration.

2 min read
0

Amazon Bedrock adds programmatic tool calling to reduce latency and token usage in multi-step workflows

Amazon Web Services has introduced programmatic tool calling (PTC) for Amazon Bedrock, enabling language models to generate Python code that orchestrates multiple tool invocations in a single inference cycle rather than requiring sequential round trips.

How it works

Traditional tool calling requires models to invoke tools one at a time, with each call requiring a full inference round trip. For a query like "Which engineering team members exceeded their Q3 travel budget?", a model using standard tool calling would need to make 20+ separate API calls to retrieve team members and expense records, passing thousands of intermediate data points through its context window.

With PTC, the model generates Python code once that handles all tool calls, data processing, and filtering within a sandboxed execution environment. Using asyncio.gather(), the code can execute tool calls in parallel. Only the final processed result returns to the model's context.

According to AWS, this approach reduces both latency and token consumption for workflows involving multiple tool calls, data aggregation, or numerical calculations.

Three implementation options

AWS provides three ways to implement PTC on Bedrock:

Self-hosted Docker sandbox on Amazon ECS: Offers maximum control with model-agnostic support for Claude, Qwen, MiniMax, Llama, Nova, and other Bedrock models. Developers can customize the sandbox environment, install domain-specific Python packages, and keep code execution within their AWS account. The architecture uses an orchestrator (ECS task or Lambda) that calls the InvokeModel API via Boto3 and manages Docker sandbox lifecycle.

Managed solution via Amazon Bedrock AgentCore Code Interpreter: A fully managed sandbox environment that handles execution without requiring custom infrastructure.

Anthropic SDK-compatible proxy: Designed for teams already using Anthropic's SDK who want PTC functionality while maintaining their existing developer workflow.

System prompt engineering

The self-hosted implementation relies on injecting tool definitions into the system prompt rather than using the standard tool_config parameter. The prompt instructs models to write Python code with specific rules: each execute_code call runs in a fresh stateless environment, tool calls must use await, and all operations must complete in a single code block.

The orchestrator intercepts tool calls through IPC over stdin/stderr, executes them externally, and injects results back into the sandbox.

What this means

PTC addresses a legitimate bottleneck in agentic workflows where sequential tool calling creates compounding latency. The model-agnostic self-hosted option is particularly significant—it extends a pattern originally introduced by specific providers to any model available on Bedrock. This matters for enterprises already committed to AWS infrastructure who want to avoid vendor lock-in at the model level.

The technique works best for workflows involving data aggregation, filtering operations, or scenarios where intermediate data shouldn't enter the model's context for privacy reasons. However, it requires models capable of generating correct async Python code and adds complexity around sandbox security and resource management.

Related Articles

product update

Nous Research brings open-source Hermes Agent to Microsoft Teams, adds Collective Wisdom skill sharing

Nous Research says companies will soon be able to run its open-source Hermes Agent directly in Microsoft Teams, with agents sandboxed in Azure Container Apps. A new feature, Collective Wisdom, lets employees share reviewed skills with colleagues, and the shared knowledge stays inside the company.

product update

BioMap launches AI drug safety system with xTrimoTox, reports 0.9637 AUROC on liver injury

BioMap launched an AI-assisted nonclinical drug safety assessment system on November 19. It combines virtual-cell models, toxicity prediction and evaluation services, and the company reports AUROC of 0.9637 for drug-induced liver injury on a selected benchmark task.

product update

Kimi Code CLI v2.7.0 adds native Amazon Bedrock provider with standard AWS credentials

Kimi Code CLI v2.7.0, Moonshot AI's open source coding agent, adds a native bedrock provider. It resolves the standard AWS credential chain, including named profiles and SSO, so users no longer need a separate Amazon Bedrock API key. AWS demonstrates the setup with Kimi K3.1 Pro Highspeed.

product update

OpenAI tests ChatGPT dots with a small group of Plus users, rolls out to all Business Standard

OpenAI has started testing ChatGPT dots with a small number of ChatGPT Plus subscribers and is rolling it out to all ChatGPT Business Standard users. Dots launched in September for Pro, Business Premium and Enterprise plans. Usage allowances for the new tiers have not yet been clarified.

Comments

Loading...