AWS introduces rDPO unlearning technique to reduce false content moderation in Amazon Nova models by 53 percentage points

AWS has developed Reverse Direct Preference Optimization (rDPO), a novel unlearning technique that enables Amazon Nova models to process legitimate business-critical content while maintaining safety alignment. The approach addresses over-deflection issues where models refuse to handle requests like cybersecurity simulations or legal document processing due to overly broad content moderation.

How rDPO works

rDPO reverses the preference pair in the Direct Preference Optimization (DPO) objective. According to AWS, traditional unlearning methods like Negative Preference Optimization (NPO) only teach models to forget deflection behavior without guiding them toward quality alternatives. rDPO simultaneously moves the model away from refusal responses while steering it toward generating high-quality outputs in approved policy areas.

The technique trains Low-Rank Adaptation (LoRA) adapters that modify specific model behaviors without retraining from scratch. AWS reports that rDPO achieves convergence at approximately 30 training steps, with training accuracy reaching nearly 1.0.

Measured deflection reductions

AWS tested the approach across five evaluation categories, measuring deflection rate—the percentage of prompts the model refuses to answer:

• Safety: 86.51% baseline → 32.77% customized (53.74 percentage point reduction)
• Security: 91.61% → 45.73% (45.88 pp reduction)
• Sensitive Content: 79.02% → 33.58% (45.44 pp reduction)
• Fairness: 51.84% → 23.83% (28.01 pp reduction)
• Red Team Prompts: 98.10% → 47% (51.1 pp reduction)

The customized models now process the majority of previously-blocked legitimate requests while maintaining alignment in non-targeted areas.

Customizable Content Moderation Settings

The technology powers Amazon Nova Customizable Content Moderation Settings (CCMS), which allows approved customers to adjust safeguards across four responsible AI pillars: Safety, Sensitive Content, Fairness, and Security. Essential controls for child safety and privacy remain non-configurable.

Customers receive a custom model variant identified by a unique ARN when importing the LoRA adapter. At inference time, the adapter steers the core model away from deflecting approved content categories while Nova's output moderation guardrails are automatically configured for the customer's approved policies.

Training efficiency comparison

According to AWS's internal benchmarks, rDPO demonstrated superior training efficiency compared to NPO. The training rewards for target responses in rDPO continued growing throughout training, while NPO's rewards for chosen responses dropped. AWS attributes this to the strong RAI alignment in the base model, which makes it difficult for NPO to effectively move away from safe responses while guiding toward quality alternatives.

What this means

This marks a notable technical approach to the over-deflection problem that has limited enterprise adoption of aligned models. By using LoRA adapters rather than full model retraining, the technique offers a practical path for organizations with legitimate use cases that conflict with broad content policies. The 53 percentage point reduction in safety deflections suggests the approach effectively distinguishes between genuine threats and contextually appropriate content. However, AWS has not disclosed which customers can access CCMS, the approval process, or whether general capabilities benchmarks were maintained—critical details for evaluating real-world viability.