Sony Pixel Power calrec Sony

Hugging Face and NVIDIA to Accelerate Open-Source AI Robotics Research and Development

06/11/2024

At the Conference for Robot Learning (CoRL) in Munich, Germany, Hugging Face and NVIDIA announced a collaboration to accelerate robotics research and development by bringing together their open-source robotics communities.

Hugging Face's LeRobot open AI platform combined with NVIDIA AI, Omniverse and Isaac robotics technology will enable researchers and developers to drive advances across a wide range of industries, including manufacturing, healthcare and logistics.

Open-Source Robotics for the Era of Physical AI The era of physical AI - robots understanding physical properties of environments - is here, and it's rapidly transforming the world's industries.

To drive and sustain this rapid innovation, robotics researchers and developers need access to open-source, extensible frameworks that span the development process of robot training, simulation and inference. With models, datasets and workflows released under shared frameworks, the latest advances are readily available for use without the need to recreate code.

Hugging Face's leading open AI platform serves more than 5 million machine learning researchers and developers, offering tools and resources to streamline AI development. Hugging Face users can access and fine-tune the latest pretrained models and build AI pipelines on common APIs with over 1.5 million models, datasets and applications freely accessible on the Hugging Face Hub.

LeRobot, developed by Hugging Face, extends the successful paradigms from its Transformers and Diffusers libraries into the robotics domain. LeRobot offers a comprehensive suite of tools for sharing data collection, model training and simulation environments along with designs for low-cost manipulator kits.

NVIDIA's AI technology, simulation and open-source robot learning modular framework such as NVIDIA Isaac Lab can accelerate the LeRobot's data collection, training and verification workflow. Researchers and developers can share their models and datasets built with LeRobot and Isaac Lab, creating a data flywheel for the robotics community.

Scaling Robot Development With Simulation Developing physical AI is challenging. Unlike language models that use extensive internet text data, physics-based robotics relies on physical interaction data along with vision sensors, which is harder to gather at scale. Collecting real-world robot data for dexterous manipulation across a large number of tasks and environments is time-consuming and labor-intensive.

Making this easier, Isaac Lab, built on NVIDIA Isaac Sim, enables robot training by demonstration or trial-and-error in simulation using high-fidelity rendering and physics simulation to create realistic synthetic environments and data. By combining GPU-accelerated physics simulations and parallel environment execution, Isaac Lab provides the ability to generate vast amounts of training data - equivalent to thousands of real-world experiences - from a single demonstration.

Generated motion data is then used to train a policy with imitation learning. After successful training and validation in simulation, the policies are deployed on a real robot, where they are further tested and tuned to achieve optimal performance.

This iterative process leverages real-world data's accuracy and the scalability of simulated synthetic data, ensuring robust and reliable robotic systems.

By sharing these datasets, policies and models on Hugging Face, a robot data flywheel is created that enables developers and researchers to build upon each other's work, accelerating progress in the field.

The robotics community thrives when we build together, said Animesh Garg, assistant professor at Georgia Tech. By embracing open-source frameworks such as Hugging Face's LeRobot and NVIDIA Isaac Lab, we accelerate the pace of research and innovation in AI-powered robotics.

Fostering Collaboration and Community Engagement The planned collaborative workflow involves collecting data through teleoperation and simulation in Isaac Lab, storing it in the standard LeRobotDataset format. Data generated using GR00T-Mimic, will then be used to train a robot policy with imitation learning, which is subsequently evaluated in simulation. Finally, the validated policy is deployed on real-world robots with NVIDIA Jetson for real-time inference.

The initial steps in this collaboration have already been taken, having shown a physical picking setup with LeRobot software running on NVIDIA Jetson Orin Nano, providing a powerful, compact compute platform for deployment.

Combining Hugging Face open-source community with NVIDIA's hardware and Isaac Lab simulation has the potential to accelerate innovation in AI for robotics, said Remi Cadene, principal research scientist at LeRobot.

This work builds on NVIDIA's community contributions in generative AI at the edge, supporting the latest open models and libraries, such as Hugging Face Transformers, optimizing inference for large language models (LLMs), small language models (SLMs) and multimodal vision-language models (VLMs), along with VLM's action-based variants of vision language action models (VLAs), diffusion policies and speech models - all with strong, community-driven support.

Together, Hugging Face and NVIDIA aim to accelerate the work of the global ecosystem of robotics researchers and developers transforming industries ranging from transportation to manufacturing and logistics.

Learn about NVIDIA's robotics research papers at CoRL, including VLM integration for better environmental understanding, temporal navigation and long-horizon planning. Check out workshops at CoRL with NVIDIA researchers.
LINK: https://blogs.nvidia.com/blog/hugging-face-lerobot-open-source-robotic...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

01/06/2026

Dolby Sets the New Standard for Premium Entertainment at CES 2026

January 6 2026, 05:30 (PST) Dolby Sets the New Standard for Premium Entertainment at CES 2026 Throughout the week, Dolby brings to life the latest innovatio...

02/05/2026

Dalet Flex LTS Delivers Smarter Search, Faster Editing, and an AI-Ready Foundation for Modern Media

Dalet, a leading technology and service provider for media-rich organizations, t...

01/05/2026

NBCUniversal's Peacock to Be First Streamer to Integrate Dolby's Full Suite of Premium Picture and Sound Innovations

January 5 2026, 18:30 (PST) NBCUniversal's Peacock to Be First Streamer to ...

01/04/2026

DOLBY AND DOUYIN EMPOWER THE NEXT GENERATON OF CREATORS WITH DOLBY VISION

January 4 2026, 18:00 (PST) DOLBY AND DOUYIN EMPOWER THE NEXT GENERATON OF CREATORS WITH DOLBY VISION Douyin Users Can Now Create And Share Videos With Stun...

26/03/2026

Audio-Technica and Extron Announce Microsoft Teams-Certified Enterprise Solutions

Extron and Audio-Technica have announced that their Design Solutions devices, Ce...

26/03/2026

SPORTEL Asia Returns to Singapore with 370 Participants from 35 Countries

SPORTEL Asia took place March 24-25 at the Orchard Hotel Singapore, drawing 370 participants representing more than 220 companies from 35 countries. Approximate...

26/03/2026

NAB 2026: LTN to Showcase Satellite-to-IP Migration Solutions

LTN will demonstrate satellite-to-IP migration solutions for broadcasters, cable networks, MVPDs, and station groups at NAB Show 2026 (Booth W2100). The C-Ban...

26/03/2026

Seattle Mariners Upgrade Center Field Video Display at T-Mobile Park with Daktronics

The Seattle Mariners have partnered with Daktronics to install an upgraded outfi...

26/03/2026

Esports Nations Cup 2026 Names National Team Partners Across 100+ Countries

The Esports Foundation (EF) has awarded National Team Partner status to organizations and individuals across more than 100 countries and territories for the Esp...

26/03/2026

NAB 2026: Telestream Expands Vantage Integration with Adobe Premiere, Media Encoder, and Frame.io

Telestream has announced expanded integration between its Vantage workflow platf...

26/03/2026

NAB 2026: FOR-A Brings Nippon TV's viztrick AiDi AI Production Tool to Market with NVIDIA

FOR-A Corporation has announced a partnership with Nippon TV and NVIDIA to distr...

26/03/2026

CBS Sports Reaches New Long-Term Multiplatform Rights Agreement with the WNBA

CBS Sports and the WNBA have expanded their partnership through a new long-term, multiplatform rights agreement. CBS Sports will feature up to 20 regular-season...

26/03/2026

Zac Brown Band to Perform Sunday Night Baseball Open on NBC and Peacock

Zac Brown Band will star in the Sunday Night Baseball show open on NBC and Peacock, NBC Sports announced. The open was filmed at American Family Field, home of ...

26/03/2026

2026 MLB Season Opens With Full Slate of National Games

Major League Baseball opens the 2026 season with nationally televised games across multiple media partners. Fans can visit MLB.com/Watch for a complete schedule...

26/03/2026

DIRECTV Adds Nine MLB Team Channels for 2026 Season

DIRECTV has announced an agreement to carry nine new MLB-operated team channels, covering the Cincinnati Reds, Detroit Tigers (Detroit SportsNet), Kansas City R...

26/03/2026

Nationals.TV Now Available on Charter, Comcast, Cox, DIRECTV, Verizon Fios and More

Nationals.TV, the Washington Nationals' local broadcast channel, is now avai...

26/03/2026

MLB.TV Launches on the ESPN App Starting Opening Day

Starting Thursday, March 26, MLB.TV will be available on the ESPN App, giving fans access to more than 2,000 out-of-market regular season games. Fans can also r...

26/03/2026

What's Next for Spotify and NAVER in Korea After a Stockholm Summit

Spotify's partnership with NAVER, Korea's leading digital platform, is built around one simple goal: making it easier for listeners in Korea to discover...

26/03/2026

EastWest DrumX: Early access

Exclusive to ComposerCloud for first month Sample library and virtual instrument gurus EastWest have announced the upcoming release of a new virtual drum i...

26/03/2026

Dynamic Range Day 2026

Friday 27 March Friday 27 March will mark Dynamic Range Day 2026, the annual day of online activity that was started in 2010 by mastering engineer Ian Sheph...

26/03/2026

ASMR Choir from SRM Sounds

Includes four separate instruments Max Richter's SRM Sounds venture have just announced their fifth release, ASMR Choir. Captured at the main live room ...

26/03/2026

IMPORTANT FEEDBACK: PESP 6

The National Film and Video Foundation (NFVF), an agency of the Department of Sport, Arts and Culture (DSAC), has been tasked with administering the Sixth Presi...

26/03/2026

Why YEP Changed My Career (And Why It Might Change Yours)

By Matt Klein Hello to everyone reading this - whether you're a current YEP, HPA member, or just YEP-curious. I'm Matt Klein, one of the co-chairs of t...

26/03/2026

CBS Sports Inks New WNBA Deal

Share Copy link Facebook X Linkedin Bluesky Email...

26/03/2026

Apogee Insight Acquires PMA Research

Share Copy link Facebook X Linkedin Bluesky Email...

26/03/2026

NBC Announces Exclusive Coverage of Sail4th 250 Tall Ships in NYC

Share Copy link Facebook X Linkedin Bluesky Email...

26/03/2026

UK Group of media tech companies take the spotlight at NA...

28 participating companies, from start-ups to blue-chips, lead the UK Group as part of the GREAT Britain and Northern Ireland presence across all the Halls at N...

26/03/2026

Arkona Unveils BLADE//planner and Major Usability Enhancements at NAB 2026

Arkona Unveils BLADE//planner and Major Usability Enhancements at NAB 2026 Brie Clayton March 26, 2026 0 Comments New graphical configuration tool and...

26/03/2026

Bitfocus showcases complete control at NAB Show 2026

Bitfocus showcases complete control at NAB Show 2026 Brie Clayton March 26, 2026 0 Comments Continuing development drives advances in security, availa...

26/03/2026

Allen Media Group To Deploy Anoki ContextIQ

Share Copy link Facebook X Linkedin Bluesky Email...

26/03/2026

LG Announces New Premium FAST Channels

Share Copy link Facebook X Linkedin Bluesky Email...

26/03/2026

IABM to Host Breakfast Event at 2026 NAB Show

Share Copy link Facebook X Linkedin Bluesky Email...

26/03/2026

Nexstar Defends Tegna Deal in Calif. Court Filing

Share Copy link Facebook X Linkedin Bluesky Email...

26/03/2026

Nevion introduces powerful new Panel Builder to enhance VideoIPath broadcast control capability

Nevion introduces powerful new Panel Builder to enhance VideoIPath broadcast con...

26/03/2026

2026 Oscar Nominated Films Powered by Blackmagic Design

2026 Oscar Nominated Films Powered by Blackmagic Design Brie Clayton March 25, 2026 0 Comments DaVinci Resolve Studio used on 27 of this year's no...

26/03/2026

Leader to present full suite of advanced Test & Measurement solutions at NAB Show 2026

Leader to present full suite of advanced Test & Measurement solutions at NAB Sho...

26/03/2026

Boston Conservatory to Present New England and Collegiate Premiere of Groundbreaking Opera Time to Act

Boston Conservatory to Present New England and Collegiate Premiere of Groundbrea...

26/03/2026

HBO Max included for Sky and NOW customers from 26 March, bringing more world-class entertainment in one subscription

Thursday 26 March 2026 HBO Max included for Sky and NOW customers from 26 March...

26/03/2026

Sky Sets Iconic Costumes Dancing Together Like Never Before in Brand Campaign

Thursday 26 March 2026 Sky Sets Iconic Costumes Dancing Together Like Never Before' in Brand CampaignTurn on cookies to view this content. Go to Privacy o...

26/03/2026

CS live Builds Future-Ready OB Van With Riedel MediorNet and hi Control System

Wuppertal March 26, 2026 CS live Builds Future-Ready OB Van With Riedel MediorNet and hi Control SystemRiedel Communications today announced that Czech-based ...

26/03/2026

Have a Wish Worth Dying For? Netflix's First Korean YA Horror Series If Wishes Could Kill' Premieres April 24

Back to All News Have a Wish Worth Dying For? Netflix's First Korean YA Hor...

26/03/2026

'The Truthers,' Starring Jose Coronado and Stphanie Magnin, Premieres on Netflix on 24 July

Back to All News The Truthers, Starring Jose Coronado and St phanie Magnin, Pre...

26/03/2026

Harmonic Delivers Winning Live Sports Streaming Innovations

Enhancements to Harmonic's Sports Streaming Solution Introduce Server-Side Multiview, Contextual Metadata-Driven Advertising, Dynamic In-Stream Ads and Robu...

26/03/2026

ABC Business Media Report: Trusted Data, Real Insight

Open the report Our 2025 report brings together results from 79 business magazine titles across 22 market sectors, published by 49 media owners. Over the year,...

26/03/2026

Into the Omniverse: NVIDIA GTC Showcases Virtual Worlds Powering the Physical AI Era

Editor's note: This post is part of Into the Omniverse, a series focused on ...

26/03/2026

Fiat to sponsor The Louise Duffy Show on RT Radio 1

RT Commercial today announced Fiat as sponsor of The Louise Duffy Show on RT Radio 1, Weekdays 3pm 4pm. The Louise Duffy Show is the home of daytime music...

26/03/2026

Game On: Five New Titles Now Streaming on GeForce NOW

That gaming backlog won't clear itself - GeForce NOW is here to help. Stream the latest titles straight from the cloud across a variety of devices. This we...

26/03/2026

Phantom C-Series High-Speed Cameras Set a New Standard for Automotive Crash and Safety Imaging

Wayne, N.J., March 26, 2026 Phantom High-Speed announces the latest product li...