NVIDIA brings generative AI to robotic surgery

NVIDIA has introduced Cosmos-H-Dreams, a generative AI simulator that predicts in real time how a surgical scene would change based on a robot’s movements. The system can take an initial image and a sequence of actions, generate the next frames, and continue the simulation while the robot or a person keeps interacting with it.

The goal is not to operate on patients. It is to create a virtual environment where surgical robots and their controllers can train, be tested, and make mistakes without putting physical equipment at risk or relying on limited laboratory hours.

From programmed simulations to learned scenarios

Traditional simulators require developers to manually describe objects, physical rules, and the consequences of each movement. World models, by contrast, learn these relationships from videos synchronized with a robot’s movements.

The previous version, Cosmos-H-Surgical-Simulator, could take an image of the surgical context and a future robot trajectory to generate a video of the likely outcome. This makes it possible to evaluate a policy, meaning the system that decides which movement the robot should make, without running every test on real hardware.

Cosmos-H-Dreams brings that capability to an interactive environment. Instead of generating a complete simulation all at once, it produces small video blocks and updates them as new actions arrive.

More speed for testing more movements

The model is specialized for suturing tasks on a table with the da Vinci Research Kit (dVRK). NVIDIA trained it on successful demonstrations as well as episodes involving mistakes, such as dropped needles, failed throws, and improperly tied knots.

That detail matters. A simulator that only reproduces perfect movements cannot show how a robot will respond when a policy makes a mistake. To train and evaluate autonomous systems, it must also show the consequences of bad decisions.

NVIDIA says the system goes from approximately 10 frames per second with the previous simulator’s standard inference to around 160 frames per second with Cosmos-H-Dreams, using a single RTX PRO 6000 GPU.

The acceleration combines two components. First, NVIDIA distills the large model, called the teacher, into a smaller model that needs only a few generation steps. It then runs that model with FlashDreams, a library that reduces latency through techniques such as memory caching, model compilation, and optimized CUDA execution.

A person or a policy can control it

The system offers several ways to interact with it:

• A web client receives frames over WebRTC and lets users send commands from the keyboard.
• A Meta Quest headset can translate controller movements into actions for the robot and display the scene through WebXR.
• A learned policy can exchange actions and observations with the simulator in a closed loop.

NVIDIA also worked with CMR Surgical and Cambridge Consultants to integrate a demonstration into the surgeon controller of the Versius robot. This does not make the model an approved medical system or mean that it controls real operations.

What changes in practice

For people developing surgical robotics, the main benefit is being able to run more experiments with less access to hardware. The model can help to:

• Generate synthetic data to train robots.
• Test policies before deploying them on physical equipment.
• Create rare failures on demand.
• Quickly compare different movement strategies.
• Investigate how latency and errors affect a surgical task.

But generating plausible videos is not enough. The simulator must preserve the position of the instruments, keep the scene stable over long periods, and predict consequences that match what is observed on the real robot. NVIDIA proposes evaluating factors such as tool-tip accuracy, gripper behavior, stability when there is no movement, and the difference between simulated and real outcomes.

Cosmos-H-Dreams is a research and development platform, not a diagnostic system, a tool that replaces intraoperative imaging, or a controller for physical surgical robots. Its importance lies in creating a faster, more flexible testing space where robotic policies can learn before they face the real world.

The next step will be determining how much of what happens in the simulation holds up outside it. If fidelity and stability improve, these models could connect training, data generation, robot training, and evaluation within a single virtual environment.