AWS Ships Multi-Turn RL Infrastructure for Amazon Nova on SageMaker HyperPod

AWS has released infrastructure for deploying multi-turn reinforcement learning (RL) to train Amazon Nova models on Amazon SageMaker HyperPod. The system addresses the training challenge of enterprise agents that execute multi-step workflows requiring sequential decision-making across multiple turns.

Technical Architecture

The infrastructure operates as an event-driven pipeline with three compute layers:

• SageMaker HyperPod (EKS): Training pods and vLLM generation replicas run on P5 instances, generating responses and performing GRPO (Group Relative Policy Optimization) weight updates
• ECS on AWS Fargate: Reward workers execute custom environments, receive model responses via SQS, and return reward signals
• Amazon Nova Forge SDK: Routes messages between the model and reward environment while tracking conversation state

The system uses a two-phase deployment model. A one-time AWS CDK deployment provisions the foundation (VPC, EKS/HyperPod, ECS, S3, IAM), while each training run spins up ephemeral resources. AWS Step Functions orchestrates runs, triggered by Amazon EventBridge when data lands in S3.

Infrastructure Requirements

Minimum specifications:

• 10 ml.p5.48xlarge instances (12-14 recommended for production)
• Amazon Nova Forge subscription
• Python 3.12+
• AWS CDK v2
• Docker for Lambda container image builds

Operating costs: $786-$1,180 per hour when running, with deployment taking 30-40 minutes.

Configuration Parameters

Key settings in cdk.json:

• instance_type: ml.p5.48xlarge (default) or ml.p5en.48xlarge
• instance_count: 10 (default)
• nova_model: NOVA_MICRO, NOVA_LITE, NOVA_LITE_2, or NOVA_PRO
• generation_replicas: 4 (default)
• global_batch_size: 64 samples per training step
• max_steps: 10 (default)

The system supports two training methods: RFT_MULTITURN_FULL and RFT_MULTITURN_LORA. Users can override parameters at deploy time via CDK command-line flags.

Training Approach

Unlike standard RLHF, which optimizes single responses in isolation, multi-turn RL optimizes over entire interaction sequences. According to AWS, this approach teaches agents tool orchestration, error recovery, and multi-step reasoning through trial and error—capabilities that supervised fine-tuning, RAG, and continued pre-training typically do not provide on their own.

The infrastructure includes a built-in Wordle environment for validation, with support for custom Bring Your Own Orchestrator (BYOO) environments.

Availability

The infrastructure is available now via GitHub at aws-samples/nova-multi-turn-rl-infra. AWS also offers multi-turn RL as a fully managed, serverless capability in SageMaker AI for users who don't require full control over the training stack.

What This Means

This release gives enterprises infrastructure to train agents on complex, multi-step workflows without managing the orchestration layer themselves. The $786-$1,180 hourly cost puts this firmly in the enterprise category, but the event-driven architecture and ephemeral resource model prevent idle GPU waste. The two-phase deployment separates foundation from compute, reducing iteration cycles for teams that need custom reward environments or specific instance configurations beyond what AWS's managed service provides.