NVIDIA releases four open-weight Cosmos 3 Nano checkpoints, 16B parameters each, for driving and robot simulation

NVIDIA has added four open-weight checkpoints to its Cosmos 3 model family, aimed at interactive world simulation and multi-sensor, multi-view driving scenes. All four are post-trained examples of Cosmos 3 Nano and have 16B trainable parameters. They are released under OpenMDW1.1, which permits commercial and non-commercial use under its terms. NVIDIA also published a technical report and post-training recipes.

The four checkpoints

• Cosmos3-Nano-Transfer-Auto: a bidirectional model for offline synthetic driving data. It generates synchronized video across up to 11 camera views, including fisheyes, and supports joint RGB and LiDAR generation. Per-camera World Scenario Map (WSM) controls set scene structure and traffic motion, while text prompts guide appearance, so developers can vary weather, lighting and time of day. Joint LiDAR generation also needs aligned HD-map range-map controls.
• Cosmos3-Nano-Sim-Auto: the causally distilled streaming variant of Transfer-Auto. It generates successive small chunks of synchronized camera video and LiDAR in a few denoising steps, using generated history and updated driving controls for long, interactive rollouts. Closed-loop use requires a simulation runtime and a WSM renderer to update controls from driving inputs.
• Cosmos3-Nano-Sim-Transfer: turns a physics simulator's spatial controls (depth, segmentation, edges, blur) and a text prompt into RGB observations a policy can consume. The simulator keeps physics and ground-truth state; the model handles appearance.
• Cosmos3-Nano-Sim-Bimanual: predicts video from robot or camera actions. It supports interactive two-arm simulation for YAM and AgiBot embodiments, driven by a policy or human-operated joysticks. The mixed-action example uses 59 control values, padded to 64 during training.

How streaming generation works

According to NVIDIA, Cosmos 3 uses a Mixture-of-Transformers backbone that connects separate text-processing and visual-generation networks through attention. The generator predicts each short segment from the prompt, available controls and generated history. Temporal-causal attention lets it read history and controls while blocking future target observations, and a key/value cache carries attention information into the next prediction.

The model works on compressed video latents. After the initial image, each latent time step represents four RGB frames. Transfer predicts one step per chunk, and the optimized Bimanual setup in the report predicts two. Smaller chunks let new controls affect generation more often. NVIDIA describes a four-stage training recipe that starts with Stage 0, bidirectional adaptation, which learns the domain and its control-to-video relationships.

Adaptation path and requirements

NVIDIA positions Transfer-Auto as the base for adapting Cosmos to a custom sensor rig and driving domain. Developers post-train it on their own data for camera layout, LiDAR configuration and target environment. The adapted model can generate data offline, or go through causal training and few-step distillation for streaming simulation. NVIDIA says this is the path used to create Sim-Auto.

For Bimanual, the actions must follow the published schema, which defines coordinate frames, rotations, gripper calibration, validity masks and control domains. Timing must be aligned: T transitions need T+1 observations. Released inference actions are already normalized. Requirements are Linux, an NVIDIA GPU, Docker and NVIDIA Container Toolkit. The published vLLM-Omni setup was tested on one GB200, and NVIDIA states this is not a minimum-memory specification. Hosted pricing is not yet disclosed.

What this means

The release splits simulation into distinct roles. Sim-Transfer keeps physics in a conventional simulator and uses the model only for rendering. Sim-Bimanual generates the next observation directly from the scene and robot actions. Sim-Auto ties camera and LiDAR output together, which matters where visual and 3D sensing have to agree.

Open weights, post-training recipes and a documented path from Transfer-Auto to Sim-Auto let teams adapt the models to their own sensor configurations. NVIDIA itself notes that these problems remain hard. It says the remaining work is to measure where the models faithfully support policy learning and evaluation, understand their limits, and improve where they fall short.