NVIDIA launches Cosmos 3, an open model for robots

NVIDIA has introduced Cosmos 3, an open model designed to help robots, autonomous vehicles, and intelligent systems understand the physical world and act within it. The new version combines tasks that previously required several separate models into a single system.

Cosmos 3 can analyze text, images, video, audio, and actions. It can also generate images and videos, predict what will happen next in a scene, and suggest actions based on the position of objects and the basic laws of motion.

One model to understand and act

In earlier versions, NVIDIA split these functions across specialized models: some generated virtual worlds, others interpreted scenes, controlled generation, or produced instructions for robots.

Cosmos 3 brings these capabilities together in an architecture called Mixture-of-Transformers (MoT). In simple terms, it uses specialized parts for each type of information, but connects them within the same model. That lets it move from describing a scene to simulating how it will evolve or deciding what action a robot should take.

For example, it could receive an image of a table with several objects and an instruction such as «place the cup to the left of the book». It can also generate a video of a vehicle encountering debris on the road or create synthetic scenes to train a robot in a warehouse.

"Cosmos 3 gives you the foundation to simulate and understand the physical world."

Two versions for different uses

The launch includes two models available on Hugging Face:

• Cosmos 3 Nano: has 8 billion parameters for the reasoning component and another 8 billion for the generator. It is designed for more efficient inference and can run on workstations with hardware such as the RTX PRO 6000 GPU.
• Cosmos 3 Super: has 32 billion parameters for each of those functions. It is intended for research and the generation of large volumes of synthetic data, using NVIDIA Hopper and Blackwell GPUs.

The company is also publishing model cards, licenses, scripts for adapting Cosmos 3 to proprietary data, and an integration with Diffusers, the Hugging Face library used to create images and videos with generative models.

Synthetic data for training robots

Synthetic datasets are one of the important parts of the launch. They are generated in simulations and can be used to train or evaluate AI systems without having to record every situation in the real world.

NVIDIA is publishing data covering several areas:

• Robot scenes and warehouse operations
• Physical interactions and spatial reasoning
• Synthetic human motion
• Autonomous driving scenarios

For example, this makes it possible to create thousands of risky situations in a warehouse or simulate rare events on the road. Teams can then use those examples to improve a system before testing it with real people, vehicles, or robots.

What this changes for you

Cosmos 3 is not an assistant designed for general users or a robot you can buy off the shelf. It is a foundation for developers who need to teach a machine how an environment works and what consequences an action may have.

A manufacturer could use it to simulate how a robotic arm picks up different objects. A transport company could generate difficult driving scenarios. In both cases, the goal is to reduce reliance on costly physical testing and expand the number of situations used to train the system.

Even so, NVIDIA says some projects will need additional training with proprietary data. A robot that folds clothes, for example, will need to adapt to its arm, camera, workspace, and the types of garments it will encounter.

Cosmos 3 brings video generation and action control closer together in a single tool, but simulations do not completely replace real-world testing. The next thing to watch is how much of what the system learns in those synthetic worlds holds up when the robot encounters unexpected objects, people, and errors.