Into the Omniverse: Physical AI Open Models and Frameworks Advance Robots and Autonomous Systems

Editor's note: This post is part of Into the Omniverse, a series focused on how developers, 3D practitioners and enterprises can transform their workflows using the latest advancements in OpenUSD and NVIDIA Omniverse.

Open source has become essential for driving innovation in robotics and autonomy. By providing access to critical infrastructure - from simulation frameworks to AI models - NVIDIA is enabling collaborative development that accelerates the path to safer, more capable autonomous systems.

At CES earlier this month, NVIDIA introduced a new suite of open physical AI models and frameworks to accelerate the development of humanoids, autonomous vehicles and other physical AI embodiments. These tools span the entire robotics development lifecycle - from high-fidelity world simulation and synthetic data generation to cloud-native orchestration and edge deployment - giving developers a modular toolkit to build autonomous systems that can reason, learn and act in the real world.

OpenUSD provides the common framework that standardizes how 3D data is shared across these physical AI tools, enabling developers to build accurate digital twins and reuse them seamlessly from simulation to deployment. NVIDIA Omniverse libraries, built on OpenUSD, serve as the source of ground truth simulation that feeds the entire stack.

From Labs to the Show Floor At CES 2026, developers brought the NVIDIA physical AI stack out of the lab and onto the show floor, debuting machines ranging from heavy equipment and factory assistants to social and service robots.

The stack taps into NVIDIA Cosmos world models; NVIDIA Isaac technologies, including the new Isaac Lab-Arena open source framework for policy evaluation; the NVIDIA Alpamayo open portfolio of AI models, simulation frameworks and physical AI datasets for autonomous vehicles; and the NVIDIA OSMO framework to orchestrate training across compute environments.

Caterpillar's Cat AI Assistant, powered by NVIDIA Nemotron open models for agentic AI and running on the NVIDIA Jetson Thor edge AI module, brings natural language interaction directly into the cab of heavy vehicles. Operators can ask Hey Cat -style questions and get step by step guidance, as well as adjust safety parameters by voice.

Behind the scenes, Caterpillar uses Omniverse libraries to build factory and job site digital twins that can help simulate layouts, traffic patterns and multi machine workflows. These insights are fed back into equipment and fleets before changes are deployed to job sites, making AI assisted operations safer and more efficient.

LEM Surgical showcased its Dynamis Robotic Surgical System, which is FDA-cleared and in routine clinical use for spinal procedures. The next-generation system uses NVIDIA Jetson AGX Thor for compute, NVIDIA Holoscan for real-time sensor processing and NVIDIA Isaac for Healthcare to train its autonomous arms.

LEM Surgical also uses NVIDIA Cosmos Transfer - an open, fully customizable world model that enables physically based synthetic data generation - to generate synthetic training data and the NVIDIA Isaac Sim framework for digital twin simulation. Designed as a dual-arm humanoid surgical robot for hard-tissue surgery, the Dynamis system mimics human surgeon dexterity and enables complex spinal procedures with enhanced precision, alleviating strenuous physical demands on surgeons and surgical assistants.

LEM Surgical showcase. NEURA Robotics is building cognitive robots on a full NVIDIA stack, using Isaac Sim and Isaac Lab to train its 4NE1 humanoid and MiPA service robots in OpenUSD based digital twins before deployment in domestic settings and workplaces. The company used NVIDIA Isaac GR00T Mimic to post train the Isaac GR00T foundation model for its platforms.

In addition, NEURA Robotics is collaborating with SAP and NVIDIA to integrate SAP's Joule agents with its robots, using the Mega NVIDIA Omniverse Blueprint to simulate and refine robot behavior in complex, realistic operational scenarios before those agents and behaviors are deployed into the company's Neuraverse ecosystem, as well as in real world fleets.

AgiBot uses NVIDIA Cosmos Predict 2 as the world modeling backbone for its Genie Envisioner (GE-Sim) platform - allowing the platform to generate action conditioned videos grounded in strong visual and physical priors. Combining this data with Isaac Sim and Isaac Lab, as well as post training on AgiBot's own data, lets policies developed in Genie Envisioner transfer more reliably to Genie2 humanoids and compact Jetson Thor-powered tabletop robots.

Intbot is using the NVIDIA Cosmos Reason 2 open model to give its social robots a sixth sense for the real world - using the model's reasoning capabilities to identify simple social cues and safety context that go beyond simple scripted tasks. In its Cosmos Cookbook recipe, Intbot demonstrates how reasoning vision language models can aid robots in deciding when to speak and how to more naturally interact with humans.

How Robotics Developers Are Using New Toolkits and Frameworks NVIDIA recently introduced Agile, an Isaac Lab-based engine for humanoid loco manipulation that packages a full, sim to real verified workflow for training robust reinforcement learning policies on platforms like the Unitree G1 and LimX Dynamics TRON.

Robotics developers can use Agile's built in task configurations, Markov Decision Process mathematical models for decision-making, training utilities and deterministic evaluation tools to tune policies. Developers can then stress test these policies in Isaac Lab and transfer locomotion and whole body behaviors to real-world robots more reliably and efficiently.

Hugging Face and NVIDIA are bringing together their robotics communities by integrating NVIDIA Isaac GR00T N models and simulation frameworks into the LeRobot ecosystem.

LINK:	https://blogs.nvidia.com/blog/physical-ai-open-models-robot-autonomous...
	See more stories from nvidia

More from Nvidia