AWS Details Custom Reward Function Design for Multi-Turn RL on Amazon Nova Forge

Amazon Web Services has published a detailed technical guide on building custom reward functions for multi-turn reinforcement learning (RL) with Amazon Nova Forge, its customization platform for Amazon Nova models. The post, part of a series on Nova Forge infrastructure, focuses on reinforcement fine-tuning (RFT) using Group Relative Policy Optimization (GRPO) and explains how to design reward logic that holds up across multi-step, agentic tasks like tool calling, code execution, and error recovery.

What's new

Nova Forge now offers a generally available serverless multi-turn RL option for teams that don't want to manage their own training environment. For teams that need more control, Nova Forge's Bring Your Own Orchestration (BYOO) capability lets developers run custom reward logic inside their own environment — for example, on Amazon Elastic Container Service (ECS) — while Nova Forge handles rollout coordination, message passing, and conversation state across turns.

This matters because multi-turn conversations and long-running scoring exceed the 15-minute invocation limit of AWS Lambda, which is used for single-turn RFT via a reward_lambda_arn parameter. For multi-turn tasks, developers instead set rollout.delegate: true, and Nova Forge delegates each rollout episode to a customer-managed container that runs a user simulator, executes code, calls a verifier, and returns an aggregate_reward_score plus optional per-component metrics via metrics_list.

Technical details

According to AWS, RFT differs from supervised fine-tuning (SFT) by learning from evaluation signals on the model's own outputs rather than curated, annotated examples. For each conversation, GRPO samples K rollouts from the current model, ranks them using the reward function, and updates model weights based on the normalized reward (advantage) within that group. AWS states that a reward component contributes to learning only when it varies across completions in a group — a term that returns the same value for every rollout produces no gradient signal at all.

The guide recommends combining three reward types: episode-level outcome rewards (e.g., whether unit tests passed), turn-level behavioral rewards (e.g., asking before acting, avoiding loops), and explicit penalties for failure modes like guessing or stalling. AWS cites research from Chu et al. (2025) showing RL improves out-of-distribution generalization across task variants from a shared checkpoint, while SFT degrades under equal-compute post-training — though this is presented as external research, not an AWS-run benchmark.

The post also warns of a real production pitfall: a highest-weighted reward component that silently contributed zero learning signal during an actual training run, despite healthy-looking training curves. AWS recommends instrumenting each reward component individually to verify it is actually driving learning rather than assuming a plausible-looking reward is working correctly.

Requirements

Following the guide requires a Nova Forge subscription (which includes the Nova Customization SDK and multi-turn RFT APIs), a SageMaker HyperPod cluster, a customer-managed ECS environment, an S3 bucket for rollouts and checkpoints, and example code from AWS's aws-samples/sample-nova-multi-turn-rl-infra GitHub repository. Custom environments are opt-in via a cdk.json configuration flag; the default stack uses a built-in Wordle environment.

What this means

This is a documentation and infrastructure update, not a new model release — no new Nova model weights or version are being introduced. The practical value here is aimed squarely at teams already using Nova Forge to customize Nova models for agentic, multi-step tasks (tool use, code execution, multi-turn dialogue) where naive single-shot fine-tuning falls short. The warning about reward components silently contributing no gradient is the most operationally useful detail: it's a failure mode that produces normal-looking loss curves while quietly training nothing, and catching it requires deliberate per-component instrumentation rather than trusting aggregate metrics. Teams building production RL pipelines on any platform, not just Nova Forge, should take note of the same debugging principle.