AWS introduces rDPO unlearning technique to reduce false content moderation in Amazon Nova models by 53 percentage point
AWS has developed Reverse Direct Preference Optimization (rDPO), a novel unlearning technique that reduces over-deflection in Amazon Nova models by up to 53 percentage points. The approach allows organizations to selectively adjust content moderation safeguards while preserving general model capabilities through LoRA adapters.
AWS introduces rDPO unlearning technique to reduce false content moderation in Amazon Nova models by 53 percentage points
AWS has developed Reverse Direct Preference Optimization (rDPO), a novel unlearning technique that enables Amazon Nova models to process legitimate business-critical content while maintaining safety alignment. The approach addresses over-deflection issues where models refuse to handle requests like cybersecurity simulations or legal document processing due to overly broad content moderation.
How rDPO works
rDPO reverses the preference pair in the Direct Preference Optimization (DPO) objective. According to AWS, traditional unlearning methods like Negative Preference Optimization (NPO) only teach models to forget deflection behavior without guiding them toward quality alternatives. rDPO simultaneously moves the model away from refusal responses while steering it toward generating high-quality outputs in approved policy areas.
The technique trains Low-Rank Adaptation (LoRA) adapters that modify specific model behaviors without retraining from scratch. AWS reports that rDPO achieves convergence at approximately 30 training steps, with training accuracy reaching nearly 1.0.
Measured deflection reductions
AWS tested the approach across five evaluation categories, measuring deflection rate—the percentage of prompts the model refuses to answer:
- Safety: 86.51% baseline → 32.77% customized (53.74 percentage point reduction)
- Security: 91.61% → 45.73% (45.88 pp reduction)
- Sensitive Content: 79.02% → 33.58% (45.44 pp reduction)
- Fairness: 51.84% → 23.83% (28.01 pp reduction)
- Red Team Prompts: 98.10% → 47% (51.1 pp reduction)
The customized models now process the majority of previously-blocked legitimate requests while maintaining alignment in non-targeted areas.
Customizable Content Moderation Settings
The technology powers Amazon Nova Customizable Content Moderation Settings (CCMS), which allows approved customers to adjust safeguards across four responsible AI pillars: Safety, Sensitive Content, Fairness, and Security. Essential controls for child safety and privacy remain non-configurable.
Customers receive a custom model variant identified by a unique ARN when importing the LoRA adapter. At inference time, the adapter steers the core model away from deflecting approved content categories while Nova's output moderation guardrails are automatically configured for the customer's approved policies.
Training efficiency comparison
According to AWS's internal benchmarks, rDPO demonstrated superior training efficiency compared to NPO. The training rewards for target responses in rDPO continued growing throughout training, while NPO's rewards for chosen responses dropped. AWS attributes this to the strong RAI alignment in the base model, which makes it difficult for NPO to effectively move away from safe responses while guiding toward quality alternatives.
What this means
This marks a notable technical approach to the over-deflection problem that has limited enterprise adoption of aligned models. By using LoRA adapters rather than full model retraining, the technique offers a practical path for organizations with legitimate use cases that conflict with broad content policies. The 53 percentage point reduction in safety deflections suggests the approach effectively distinguishes between genuine threats and contextually appropriate content. However, AWS has not disclosed which customers can access CCMS, the approval process, or whether general capabilities benchmarks were maintained—critical details for evaluating real-world viability.
Related Articles
AWS releases I/Q, I/R and Isabelle Assistant to connect AI agents to the Isabelle proof checker via MCP
AWS has made three tools in its AutoCorrode verification suite publicly available: I/Q, I/R and Isabelle Assistant. They connect AI agents to the Isabelle proof assistant through the Model Context Protocol, so generated proofs are checked by computer. Amazon Science authors demonstrate the setup using Claude Opus 4.6.
Hugging Face releases Qwen3.8-27B-pi-adapted LoRA, lifting SWE-bench Pro hard-51 pass@1 from 39.2% to 64.7%
Hugging Face released Qwen3.8-27B-pi-adapted, an Apache-2.0 LoRA adapter trained with reinforcement learning for the pi coding agent. According to MarkTechPost, it raises SWE-bench Pro hard-51 pass@1 from 39.2% to 64.7% and Terminal-Bench 2.1 from 39.5% to 55.8%.
OpenAI outlines four-part approach to combat adversarial distillation
OpenAI describes a layered approach to adversarial distillation. It combines model training, safeguards, detection, and enforcement to protect against capability extraction.
Anthropic mathematician Levent Alpöge claims Claude-assisted proof of the Riemann hypothesis, posts it in Lean
Levent Alpöge, a mathematician employed by Anthropic, says he has proven the Riemann hypothesis, first proposed in 1859, working with Claude. He posted the proof coded in Lean, and experts have yet to confirm that it is correct.
Comments
Loading...