Sony Pixel Power calrec Sony

Hugging Face and NVIDIA to Accelerate Open-Source AI Robotics Research and Development

06/11/2024

At the Conference for Robot Learning (CoRL) in Munich, Germany, Hugging Face and NVIDIA announced a collaboration to accelerate robotics research and development by bringing together their open-source robotics communities.

Hugging Face's LeRobot open AI platform combined with NVIDIA AI, Omniverse and Isaac robotics technology will enable researchers and developers to drive advances across a wide range of industries, including manufacturing, healthcare and logistics.

Open-Source Robotics for the Era of Physical AI The era of physical AI - robots understanding physical properties of environments - is here, and it's rapidly transforming the world's industries.

To drive and sustain this rapid innovation, robotics researchers and developers need access to open-source, extensible frameworks that span the development process of robot training, simulation and inference. With models, datasets and workflows released under shared frameworks, the latest advances are readily available for use without the need to recreate code.

Hugging Face's leading open AI platform serves more than 5 million machine learning researchers and developers, offering tools and resources to streamline AI development. Hugging Face users can access and fine-tune the latest pretrained models and build AI pipelines on common APIs with over 1.5 million models, datasets and applications freely accessible on the Hugging Face Hub.

LeRobot, developed by Hugging Face, extends the successful paradigms from its Transformers and Diffusers libraries into the robotics domain. LeRobot offers a comprehensive suite of tools for sharing data collection, model training and simulation environments along with designs for low-cost manipulator kits.

NVIDIA's AI technology, simulation and open-source robot learning modular framework such as NVIDIA Isaac Lab can accelerate the LeRobot's data collection, training and verification workflow. Researchers and developers can share their models and datasets built with LeRobot and Isaac Lab, creating a data flywheel for the robotics community.

Scaling Robot Development With Simulation Developing physical AI is challenging. Unlike language models that use extensive internet text data, physics-based robotics relies on physical interaction data along with vision sensors, which is harder to gather at scale. Collecting real-world robot data for dexterous manipulation across a large number of tasks and environments is time-consuming and labor-intensive.

Making this easier, Isaac Lab, built on NVIDIA Isaac Sim, enables robot training by demonstration or trial-and-error in simulation using high-fidelity rendering and physics simulation to create realistic synthetic environments and data. By combining GPU-accelerated physics simulations and parallel environment execution, Isaac Lab provides the ability to generate vast amounts of training data - equivalent to thousands of real-world experiences - from a single demonstration.

Generated motion data is then used to train a policy with imitation learning. After successful training and validation in simulation, the policies are deployed on a real robot, where they are further tested and tuned to achieve optimal performance.

This iterative process leverages real-world data's accuracy and the scalability of simulated synthetic data, ensuring robust and reliable robotic systems.

By sharing these datasets, policies and models on Hugging Face, a robot data flywheel is created that enables developers and researchers to build upon each other's work, accelerating progress in the field.

The robotics community thrives when we build together, said Animesh Garg, assistant professor at Georgia Tech. By embracing open-source frameworks such as Hugging Face's LeRobot and NVIDIA Isaac Lab, we accelerate the pace of research and innovation in AI-powered robotics.

Fostering Collaboration and Community Engagement The planned collaborative workflow involves collecting data through teleoperation and simulation in Isaac Lab, storing it in the standard LeRobotDataset format. Data generated using GR00T-Mimic, will then be used to train a robot policy with imitation learning, which is subsequently evaluated in simulation. Finally, the validated policy is deployed on real-world robots with NVIDIA Jetson for real-time inference.

The initial steps in this collaboration have already been taken, having shown a physical picking setup with LeRobot software running on NVIDIA Jetson Orin Nano, providing a powerful, compact compute platform for deployment.

Combining Hugging Face open-source community with NVIDIA's hardware and Isaac Lab simulation has the potential to accelerate innovation in AI for robotics, said Remi Cadene, principal research scientist at LeRobot.

This work builds on NVIDIA's community contributions in generative AI at the edge, supporting the latest open models and libraries, such as Hugging Face Transformers, optimizing inference for large language models (LLMs), small language models (SLMs) and multimodal vision-language models (VLMs), along with VLM's action-based variants of vision language action models (VLAs), diffusion policies and speech models - all with strong, community-driven support.

Together, Hugging Face and NVIDIA aim to accelerate the work of the global ecosystem of robotics researchers and developers transforming industries ranging from transportation to manufacturing and logistics.

Learn about NVIDIA's robotics research papers at CoRL, including VLM integration for better environmental understanding, temporal navigation and long-horizon planning. Check out workshops at CoRL with NVIDIA researchers.
LINK: https://blogs.nvidia.com/blog/hugging-face-lerobot-open-source-robotic...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

06/09/2026

Dolby and MagentaTV Bring Fans Closer to the FIFA World Cup 2026 in Germany with Dolby Vision and Dolby Atmos

June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...

04/08/2026

Dalet Announces Commercial Availability of Dalia, Bringing Media-Aware Agentic AI to Enterprise Productions

Dalet, a leading technology and service provider for media-rich organizations, t...

04/07/2026

Detective Conan: Fallen Angel of the Highway Opens in Dolby Cinemas Across Japan, Presented in Dolby Atmos and Dolby ...

April 7 2026, 19:00 (PDT) Detective Conan: Fallen Angel of the Highway Opens in...

18/06/2026

The Gauge: Poland | May 2026

The average daily TV screen time in May was 3 hours and 36 minutes, marking a clear decrease of 15 minutes compared to April. This trend proved to be much stron...

18/06/2026

Gracenote and PubMatic Bring Curated Live Sports and Content-Level Deals to Programmatic CTV

Integration embeds Gracenote content intelligence, including contextual segments...

18/06/2026

AJA Debuts Io Xpand Thunderbolt 5 Expansion Chassis for...

AJA Video Systems unveiled Io Xpand, a high-performance Thunderbolt 5-enabled PCIe expansion chassis for AJA KONA and Corvid video and audio I/O cards. As dema...

18/06/2026

Providius Announces Providius Direct for Faster AV-over-I...

New NVRT-driven workflow enables on-demand traffic mirroring and guided troubleshooting for AV teams, integrators, and support organizations ahead of InfoComm 2...

18/06/2026

Mavis Studio Brings New Muscle to Mobile Production on iP...

At InfoComm 2026, Mavis announced a major update to Mavis Studio, its live production app for the iPad, with new features designed to make professional AV produ...

18/06/2026

Harmonic Completes Divestiture of Video Business to Media...

Transaction Positions Harmonic as a Pure-Play Broadband Company Harmonic Inc. (NASDAQ: HLIT), the worldwide leader in virtualized broadband solutions, today a...

18/06/2026

ACT Entertainment and NETGEAR Partner to Power the Futur...

ACT Entertainment and NETGEAR have announced a strategic partnership that establishes verified interoperability between NETGEAR Switches and key technologies fr...

18/06/2026

IBC2026 creates new pathways from innovation to real-worl...

IBC2026 is set to bring the global media, entertainment and technology community together at the RAI Amsterdam from 11 14 September, enabling industry players f...

18/06/2026

PLIANT TECHNOLOGIES DEBUTS CREWCOM FLEX THE WORLDS FIRST...

Pliant Technologies is proud to introduce CrewCom Flex , the world's first frameless matrix intercom system and the only complete matrix IP-based intercom s...

18/06/2026

XR Sports Alliance Strengthens its Capabilities with New...

The XR Sports Alliance (XRSA) has announced that a new cohort of members has joined the strategic initiative: ActionStreamer, Antigravity, Creative Artists Agen...

18/06/2026

Hollywood Filmmakers Launch AI Production Platform Cascade

Share Copy link Facebook X Linkedin Bluesky Email...

18/06/2026

ATVA Blasts Deltavision Media for Demanding 'Egregious' Retrans Fees

Share Copy link Facebook X Linkedin Bluesky Email...

18/06/2026

MediaKind Completes Merger with Harmonic's Video Business

Share Copy link Facebook X Linkedin Bluesky Email...

18/06/2026

AJA Unveils Io Xpand Thunderbolt 5 Expansion Chassis At InfoComm 2026

Share Copy link Facebook X Linkedin Bluesky Email...

18/06/2026

Linda May Han Oh Receives Guggenheim Fellowship

Linda May Han Oh Receives Guggenheim Fellowship The Berklee professor, bassist, and composer will use the fellowship to debut Dreams of Knowing, an interdisci...

18/06/2026

FOX Advertising and Toonstar Team to Create New Opportunities for Brands in Digital-First Animation

FOX Advertising and Toonstar Team to Create New Opportunities for Brands in Digi...

18/06/2026

Arqiva secures WTA Tier-4 accreditation

Arqiva's Crawley Court and Chalfont Grove teleports re-certified at highest World Teleport Association standard 18 June 2026, Winchester, UK - Arqiva, the ...

18/06/2026

Apple announces changes to iOS in Brazil - UPDATE - Posted on 18 June 2026

Apple today announced changes impacting iOS apps in Brazil that reflect a recent agreement with Brazil's competition regulator, the Conselho Administrativo ...

18/06/2026

Sync and Stream: GeForce NOW Connects to Members' Game Libraries Across Devices

Play favorite titles from popular game libraries, keep progress synced and jump ...

18/06/2026

At Cannes Lions, NVIDIA Partners Reshape Advertising and Marketing With AI

The digital era gave the advertising and marketing industry speed; the AI era is giving it autonomous operations. For companies building next-generation techn...

17/06/2026

EVS Achieves EcoVadis Gold Medal, Ranking in Top 5% of Companies Globally

EVS has announced it has received the EcoVadis Gold Medal for sustainability performance, ranking among the top 5% of companies globally in the Technology/Mid-S...

17/06/2026

Chyron Releases Weather 2.4 with Updated DataFlow Module

Chyron has released Weather 2.4, an update to its weather suite for broadcasters and meteorologists. The release focuses on enhancements to the DataFlow module,...

17/06/2026

VSiN Launches Best Bets TV, a 24/7 FAST Channel

VSiN, The Sports Betting Network, has announced the launch of Best Bets TV, a free ad-supported streaming TV (FAST) channel. The 24/7 channel is currently avail...

17/06/2026

Snapchat and Team Whistle Launch World Cup Creator Program Across Miami, New York, and Los Angeles

DAZN's Team Whistle and Snap Inc. have announced a creator program centered ...

17/06/2026

LiveU Supporting Broadcast and Public Safety Operations for Summer of Soccer

LiveU is providing video transmission technology for broadcasters, production companies, and public safety agencies across North America's busy Summer of So...

17/06/2026

NATAS Announces Two New Board Members and COO Promotion

The National Academy of Television Arts and Sciences (NATAS) has announced that Laurens Grant and Jacob Ullman have joined its Board of Directors. Chief of Staf...

17/06/2026

Akta Brings AI-First Cloud Video Workflows to Oracle Cloud Infrastructure

Akta, the AI-First SaaS video platform for modern broadcast and streaming operations, today announced that its video platform is now generally available on Orac...

17/06/2026

SVG Students to Watch: Nicholas Stafford, Texas Southern University

This recent graduate from Houston found inspiration in technical directing and now eyes a future career in sports production...

17/06/2026

InfoComm 2026: Audio-Technica Brings New Wireless, Networked Audio, and Certified Conferencing Solutions

Audio-Technica (booth C7959) arrives at InfoComm 2026 in Las Vegas with a slate ...

17/06/2026

SNS Positions AI Suite as On-Premise Answer to Video Indexing and Search Demands

SNS has published a guide addressing growing demand for AI-powered video indexing, transcription, facial recognition, and searchable metadata across media libra...

17/06/2026

Providius Announces Providius Direct for Network Troubleshooting on NETGEAR Pro AV Switches

Providius has announced Providius Direct, a workflow for investigating network i...

17/06/2026

NEP Launches NEP Platform Software Orchestration System, Selects Bridge Technologies VB440

NEP Group has announced the commercial availability of NEP Platform, a software ...

17/06/2026

BBright White Paper: SRT vs. RIST for Professional Video Contribution and Distribution

A new white paper examining Secure Reliable Transport (SRT) and Reliable Interne...

17/06/2026

Bango Research: 51% of Gen Z Say Highlights Are Replacing Live Games

New research from subscription bundling platform Bango finds that younger sports fans are increasingly consuming sport through highlights, clips, and social med...

17/06/2026

SMPTE Opens Entire Standards Library to Public at No Cost

SMPTE has announced that its complete Standards catalog is now freely available to the global media technology community, including all published SMPTE Standard...

17/06/2026

MediaKind Completes Merger with Harmonic's Video Business, Creating Independent Video Infrastructure Heavyweight

Harmonic has completed the sale of its Video Business to MediaKind for $145 mill...

17/06/2026

Omaha Productions to Produce 2026 World Series of Poker Main Event Coverage for ESPN

Omaha Productions will produce the 2026 World Series of Poker (WSOP) in Las Vega...

17/06/2026

SoundBridge 3.1.0 now available

New features, changes & bug fixes SoundBridge have just released another update for their remote collaboration-focused DAW - reviewed here in SOS March 2026...

17/06/2026

Fryette launch the Valvulator Mini

Valve-based front end for digital & modelling rigs The latest addition to Fryette's product range delivers a packed-down, pedalboard-friendly version of...

17/06/2026

The Biggest UK Pro Audio Show In 20 Years!

GearExpo UK - 27 June 2026 Sound On Sound are proud to announce GearExpo UK, a major new recording and music technology exhibition in London! This is the bi...

17/06/2026

Genelec introduce the 9402A System Management Device

SAM monitoring line-up gains Dante and AES67 support The latest expansion of Genelec's UNIO monitoring ecosystem introduces a new device that provides D...

17/06/2026

The R&SPR300 portable receiver from Rohde & Schwarz sets new standards in spectrum monitoring

The R&S PR300 portable receiver from Rohde & Schwarz sets new standards in spect...

17/06/2026

Elt Group and Rohde & Schwarz sign a cooperation agreement to explore commercial opportunities in electronic warfare and defense

Elt Group and Rohde & Schwarz sign a cooperation agreement to explore commercial...

17/06/2026

NABLF Graduates Its 2026 Broadcasting Leadership Training Class

Share Copy link Facebook X Linkedin Bluesky Email...

17/06/2026

ABC and ESPN Score Most-Watched NBA Finals Since 1998

Share Copy link Facebook X Linkedin Bluesky Email...

17/06/2026

Smartly, Roku Bring Social Performance Tools to CTV

Share Copy link Facebook X Linkedin Bluesky Email...