NVIDIA unveils physical AI models for surgery

NVIDIA and a community of 35 organizations have introduced Open-H-Embodiment, an open dataset for training robots that can act in healthcare environments, not just interpret images or detect diseases.

The project brings together 778 hours of data with synchronized information on vision, force and movement. It includes simulations, lab exercises such as suturing and real clinical procedures, as well as data from commercial robots and research platforms.

The initiative focuses mainly on surgical robotics, though it also includes ultrasound and colonoscopy tasks. Its data is published under the CC-BY-4.0 license, allowing other teams to use and adapt it as long as they follow its terms.

From analyzing images to manipulating instruments

Until now, much of medical AI has been trained to recognize signals: locate a lesion, classify an image or segment an organ. That can support decisions, but it is not enough for a robot to perform a physical task.

A surgical system must coordinate cameras, instruments, forces and movements in real time. It also has to deal with soft tissue, reflections, blood, smoke and differences between robots. That is why traditional datasets based solely on images leave out an essential part of the problem.

Open-H-Embodiment aims to fill that gap with data from nine types of robots and 32 datasets. The collaboration includes universities, hospitals, robot manufacturers and software companies from different countries.

Two models for training and simulating robots

The dataset comes with two open-source models.

The first is GR00T-H, a vision-language-action model derived from the Isaac GR00T family. It was trained on approximately 600 hours of Open-H-Embodiment data and is designed to learn surgical robotics tasks from demonstrations.

In practice, the model receives visual information and an instruction, then generates actions to control a robot. To work with different platforms, it uses several mechanisms:

• An adapter specific to each robot's movement characteristics.
• Actions relative to the instrument's end effector, rather than commands tied to a specific machine.
• Information about the instruments and controls included in the task instructions.
• A configuration that does not use the robot's internal position data during execution.

A prototype of GR00T-H completed a suture from start to finish in the SutureBot benchmark, a lab test for evaluating surgical dexterity. The result demonstrates an experimental capability, not authorization to operate autonomously on patients.

The second model, Cosmos-H-Surgical-Simulator, generates surgical video from a robot's actions and movements. Its goal is to serve as a learned simulator: it can predict how tissue deforms and how instruments interact without having to run every test in the real world.

According to NVIDIA, generating 600 trajectories took about 40 minutes in simulation, compared with two days using physical laboratory methods. The model was fine-tuned with around 10,000 GPU-hours on 64 A100 cards and uses a shared 44-dimensional space to represent actions.

What changes for medical robotics

For researchers and manufacturers, shared data can reduce the time needed to train and compare systems. It also makes it easier to test a model on different robots, which matters because each platform has different arms, controls and capabilities.

For patients, the effect will not be immediate. These models still need clinical validation, safety controls and human supervision before taking part in real procedures. Their most immediate use is in research, training and the automation of narrowly defined tasks.

The project's next step will be to add data explaining why the robot makes a decision, what it is trying to do, what the outcome was and how it responded to a failure. The goal is to move from robots that imitate movements to systems capable of planning, adapting and explaining their actions during lengthy procedures.

The key question will be whether open models maintain their performance outside the lab and across different equipment. In surgery, learning to move an instrument is only the beginning: the system must also know when to stop.