NVIDIA Cosmos3-H-Dreams runs real-time generative cholecystectomy simulation on a single RTX PRO 6000

NVIDIA has released Cosmos3-H-Dreams, a world foundation model that generates interactive cholecystectomy video from tool-action inputs and runs in real time on a single NVIDIA RTX PRO 6000 GPU. The model was published October 20, 2026. According to NVIDIA, it is the first real-time generative surgical simulator trained on clinical cholecystectomy recordings to support live kinematic control.

What was released

Cosmos3-H-Dreams succeeds the earlier Cosmos-H-Dreams, which was built on Cosmos-Predict2.5 and targeted dVRK tabletop suturing. The new model trains on video and robot kinematics recorded during clinical procedures on the CMR Surgical Versius platform. It ships with two checkpoints:

• Four-step student: four denoising steps per generated video block.
• Two-step student: trained separately from the four-step model, not simply the same model run with fewer sampling steps. NVIDIA says fewer steps mean faster inference at the cost of lower visual quality.

Technical details

• Base model: Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on Open-H-Embodiment. NVIDIA specialized a bidirectional teacher on curated cholecystectomy video and kinematics.
• Action space: 28 dimensions covering pose and gripper information for both arms, plus energy controls, thumbstick axes and thumbstick buttons.
• Instruments: a left Cadiere grasper and a right monopolar hook. Camera commands are deliberately limited.
• Training data format: 832 x 480 video with actions aligned at 10 Hz. NVIDIA notes this is distinct from runtime generation throughput.
• Horizon: the teacher was first adapted on 24 future frames, then extended to 72 frames (2.4 and 7.2 seconds at 10 Hz).
• Distillation: causal teacher forcing, then Self-Forcing, which trains the student on its own generated context using the teacher and a learned critic. This targets error accumulation during long rollouts.
• Inference: FlashDreams, NVIDIA's accelerated streaming-inference library, with an adapted FadeMem module that consolidates the growing key/value cache during long rollouts.

NVIDIA has not disclosed a parameter count for the distilled students, a latency or frames-per-second figure, quantitative benchmark scores, or pricing in the source material. The training data cutoff is also not stated.

Validation caveats

NVIDIA is explicit that visual realism is "a starting point for validation, not proof of biomechanical accuracy." Its demonstrations show two rollouts from the same initial frame diverging under different VR-headset-recorded action sequences. NVIDIA says these show physically accurate tissue deformation, but that is a company claim supported by qualitative video, not by published surgeon evaluation or quantitative metrics.

Customization

NVIDIA provides a recipe for teacher fine-tuning and Self-Forcing distillation covering data preparation, public teacher-training tools and the distillation stages. Adapting the model to other datasets or embodiments requires synchronized video and compatible action representations. Controller integrations must match the training action conventions and normalization.

NVIDIA's stated research directions include modeling full procedures, combining generative and numerical simulation, surgeon-led evaluation, support for more instruments, and closed-loop training of autonomous surgical policies.

What this means

The main shift is a move from tabletop tasks to clinical anatomy with occlusion, smoke and tissue deformation, running on one workstation-class GPU. That makes learned simulators plausible as data and test environments for surgical robotics policies, but only if they are validated. The missing pieces are the ones NVIDIA itself flags: no published surgeon assessment, no quantitative controllability or temporal-consistency metrics, and no measured latency. Until those exist, the model is best treated as a research platform for visual simulation, not a validated physics substitute. The released distillation recipe is the more durable contribution, since it lets other groups build real-time students on their own surgical data.