NVIDIA releases LoRA/DoRA fine-tuning guide for Cosmos Predict 2.5 to generate synthetic robot training data
NVIDIA published a technical guide for parameter-efficient fine-tuning of its Cosmos Predict 2.5 world model using LoRA and DoRA adapters. The method allows teams to adapt the 2B-parameter model to robot manipulation tasks on a single 80GB GPU, generating synthetic training trajectories from just 92 demonstration videos.
NVIDIA releases LoRA/DoRA fine-tuning guide for Cosmos Predict 2.5 to generate synthetic robot training data
NVIDIA published a technical guide for parameter-efficient fine-tuning of its Cosmos Predict 2.5 world model using LoRA and DoRA adapters. The method allows teams to adapt the 2B-parameter model to robot manipulation tasks on a single 80GB GPU, generating synthetic training trajectories from just 92 demonstration videos.
Technical implementation
Cosmos Predict 2.5 is a video generation model that produces physically plausible videos from text, images, or video clips. The fine-tuning approach injects small trainable adapter modules into the frozen base model, avoiding the cost and catastrophic forgetting risks of full fine-tuning.
The implementation targets three components: a VAE for encoding videos to latents, a text encoder, and a DiT (Diffusion Transformer) for latent-space diffusion. All base weights remain frozen. LoRA adapters are injected only into the DiT's attention projections (to_q, to_k, to_v, to_out.0) and feedforward layers (ff.net.0.proj, ff.net.2). Trainable LoRA parameters are upcast to float32 for numerical stability under bf16 mixed precision.
Training approach
The model uses rectified flow, training to predict the velocity that linearly transports noise to clean data. At timestep t, the model constructs a noisy interpolation xt = σt·noise + (1−σt)·clean and learns to predict the target velocity noise − clean via MSE loss. The first two video frames serve as conditioning and receive no noise.
NVIDIA's reference training uses 92 robot manipulation videos with text prompts describing pick-and-place tasks, evaluated against 50 (prompt, image) pairs. The VideoDataset loader samples random contiguous windows of frames from longer videos each epoch for temporal augmentation.
Hardware and configuration
Minimum requirements: one 80GB GPU for single-GPU training. NVIDIA recommends 8× H100s for faster iteration. The guide provides a training script using diffusers and accelerate libraries with support for both single- and multi-GPU configurations.
Switching from LoRA to DoRA requires only setting use_dora=True in the LoraConfig. DoRA decomposes each weight into magnitude and direction before applying the low-rank update, with no other training loop changes required.
The optimizer is AdamW with linear warmup over scheduler_warm_up_steps, peaking at scheduler_f_max × learning_rate, then linear decay to scheduler_f_min × learning_rate. Checkpoints save as pytorch_lora_weights.safetensors files every specified number of epochs.
What this means
Parameter-efficient fine-tuning addresses a critical bottleneck in robot learning: collecting real-world demonstration data is slow and expensive. This guide provides a practical path to generate synthetic training data by adapting a general-purpose world model to specific robotic tasks with modest compute requirements. The approach keeps adapter files small and portable, enabling teams to swap different domain adapters at inference time without maintaining separate full model copies. The 92-video training set demonstrates that meaningful domain adaptation is possible with limited data when starting from a capable foundation model.
Related Articles
NVIDIA releases Nemotron-Labs-Sigma: 14 open 3B and 8B continuous diffusion language model checkpoints
NVIDIA released Nemotron-Labs-Sigma, a set of 14 open continuous diffusion language model checkpoints at 3B and 8B parameters. According to MarkTechPost, Sigma denoises text in a learned 16-D embedding space, distills to few-step samplers, and decodes up to 4.20x faster than autoregressive decoding.
Horizon Robotics unveils LoFT, a 514.5M-parameter video geometry model; paper, code and weights still pending
Horizon Robotics announced LoFT on November 18, a video-geometry foundation model with approximately 514.5 million parameters that predicts camera pose, camera intrinsics and per-frame depth from RGB video. The project page still lists the paper, code and Hugging Face resources as Coming soon.
Hugging Face releases Qwen3.8-27B-pi-adapted LoRA, lifting SWE-bench Pro hard-51 pass@1 from 39.2% to 64.7%
Hugging Face released Qwen3.8-27B-pi-adapted, an Apache-2.0 LoRA adapter trained with reinforcement learning for the pi coding agent. According to MarkTechPost, it raises SWE-bench Pro hard-51 pass@1 from 39.2% to 64.7% and Terminal-Bench 2.1 from 39.5% to 55.8%.
NVIDIA: Sparse attention indexer and Top-k take 47% of a DeepSeek V3.2 prefill layer at 128K tokens
NVIDIA published a co-design primer for model-level sparse attention. It reports that the indexer and Top-k take 47% of a DeepSeek V3.2 prefill layer at 128K tokens, and that reusing the prior decode step's selection speeds up Top-k by 2.1x to 2.4x.
Comments
Loading...