Sony Pixel Power calrec Sony

Hugging Face and NVIDIA to Accelerate Open-Source AI Robotics Research and Development

06/11/2024

At the Conference for Robot Learning (CoRL) in Munich, Germany, Hugging Face and NVIDIA announced a collaboration to accelerate robotics research and development by bringing together their open-source robotics communities.

Hugging Face's LeRobot open AI platform combined with NVIDIA AI, Omniverse and Isaac robotics technology will enable researchers and developers to drive advances across a wide range of industries, including manufacturing, healthcare and logistics.

Open-Source Robotics for the Era of Physical AI The era of physical AI - robots understanding physical properties of environments - is here, and it's rapidly transforming the world's industries.

To drive and sustain this rapid innovation, robotics researchers and developers need access to open-source, extensible frameworks that span the development process of robot training, simulation and inference. With models, datasets and workflows released under shared frameworks, the latest advances are readily available for use without the need to recreate code.

Hugging Face's leading open AI platform serves more than 5 million machine learning researchers and developers, offering tools and resources to streamline AI development. Hugging Face users can access and fine-tune the latest pretrained models and build AI pipelines on common APIs with over 1.5 million models, datasets and applications freely accessible on the Hugging Face Hub.

LeRobot, developed by Hugging Face, extends the successful paradigms from its Transformers and Diffusers libraries into the robotics domain. LeRobot offers a comprehensive suite of tools for sharing data collection, model training and simulation environments along with designs for low-cost manipulator kits.

NVIDIA's AI technology, simulation and open-source robot learning modular framework such as NVIDIA Isaac Lab can accelerate the LeRobot's data collection, training and verification workflow. Researchers and developers can share their models and datasets built with LeRobot and Isaac Lab, creating a data flywheel for the robotics community.

Scaling Robot Development With Simulation Developing physical AI is challenging. Unlike language models that use extensive internet text data, physics-based robotics relies on physical interaction data along with vision sensors, which is harder to gather at scale. Collecting real-world robot data for dexterous manipulation across a large number of tasks and environments is time-consuming and labor-intensive.

Making this easier, Isaac Lab, built on NVIDIA Isaac Sim, enables robot training by demonstration or trial-and-error in simulation using high-fidelity rendering and physics simulation to create realistic synthetic environments and data. By combining GPU-accelerated physics simulations and parallel environment execution, Isaac Lab provides the ability to generate vast amounts of training data - equivalent to thousands of real-world experiences - from a single demonstration.

Generated motion data is then used to train a policy with imitation learning. After successful training and validation in simulation, the policies are deployed on a real robot, where they are further tested and tuned to achieve optimal performance.

This iterative process leverages real-world data's accuracy and the scalability of simulated synthetic data, ensuring robust and reliable robotic systems.

By sharing these datasets, policies and models on Hugging Face, a robot data flywheel is created that enables developers and researchers to build upon each other's work, accelerating progress in the field.

The robotics community thrives when we build together, said Animesh Garg, assistant professor at Georgia Tech. By embracing open-source frameworks such as Hugging Face's LeRobot and NVIDIA Isaac Lab, we accelerate the pace of research and innovation in AI-powered robotics.

Fostering Collaboration and Community Engagement The planned collaborative workflow involves collecting data through teleoperation and simulation in Isaac Lab, storing it in the standard LeRobotDataset format. Data generated using GR00T-Mimic, will then be used to train a robot policy with imitation learning, which is subsequently evaluated in simulation. Finally, the validated policy is deployed on real-world robots with NVIDIA Jetson for real-time inference.

The initial steps in this collaboration have already been taken, having shown a physical picking setup with LeRobot software running on NVIDIA Jetson Orin Nano, providing a powerful, compact compute platform for deployment.

Combining Hugging Face open-source community with NVIDIA's hardware and Isaac Lab simulation has the potential to accelerate innovation in AI for robotics, said Remi Cadene, principal research scientist at LeRobot.

This work builds on NVIDIA's community contributions in generative AI at the edge, supporting the latest open models and libraries, such as Hugging Face Transformers, optimizing inference for large language models (LLMs), small language models (SLMs) and multimodal vision-language models (VLMs), along with VLM's action-based variants of vision language action models (VLAs), diffusion policies and speech models - all with strong, community-driven support.

Together, Hugging Face and NVIDIA aim to accelerate the work of the global ecosystem of robotics researchers and developers transforming industries ranging from transportation to manufacturing and logistics.

Learn about NVIDIA's robotics research papers at CoRL, including VLM integration for better environmental understanding, temporal navigation and long-horizon planning. Check out workshops at CoRL with NVIDIA researchers.
LINK: https://blogs.nvidia.com/blog/hugging-face-lerobot-open-source-robotic...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

04/08/2026

Dalet Announces Commercial Availability of Dalia, Bringing Media-Aware Agentic AI to Enterprise Productions

Dalet, a leading technology and service provider for media-rich organizations, t...

04/07/2026

Detective Conan: Fallen Angel of the Highway Opens in Dolby Cinemas Across Japan, Presented in Dolby Atmos and Dolby ...

April 7 2026, 19:00 (PDT) Detective Conan: Fallen Angel of the Highway Opens in...

05/06/2026

LSUs Sarah Ramundt on Celebrating the One-Year Anniversary of The Brand

The university-wide initiative has pushed their creative content to a new level Although collegiate athletics at a single institution can contain numerous spo...

05/06/2026

SVG GameDay, Ep. 18: New York Islanders Brooklyn Boyars - A Career Filled With Championships

In-venue and creative video staffers at the professional and collegiate level ha...

05/06/2026

Synamedia Enters New Chapter Following Lumine Group Acquisition Agreement

Synamedia has announced that Lumine Group has agreed to acquire its Video Network business. The company is positioning the transition as the start of a new phas...

05/06/2026

Ateme, BCE, and Scaleway Announce Sovereign Cloud Media Supply Chain Partnership

Ateme, Broadcasting Center Europe (BCE), and Scaleway have announced a strategic partnership to deliver a cloud-based media supply chain covering ingest through...

05/06/2026

Audinate Adds Three Models to Dante AVIO Install Adapter Line

Audinate has announced three new additions to its Dante AVIO Install adapter series: a 4-Channel Analog Input, a 4-Channel Analog Output, and a 2-Ch In/2-Ch Out...

05/06/2026

Sportradar Extends Data and AV Betting Rights Agreement for Wimbledon Beyond 2026

Sportradar Group AG has announced a multi-year extension of its exclusive global...

05/06/2026

M6 Group to Broadcast FIFA World Cup 2026 in 4K UHD via Fransat Satellite Platform

The M6 Group will broadcast FIFA World Cup 2026 matches live and in Ultra High D...

05/06/2026

The Athletic Adds PGA TOUR Highlights to Golf Coverage

The Athletic has announced that PGA TOUR highlights will be integrated into its golf coverage beginning with the Memorial Tournament presented by Workday. PGA T...

05/06/2026

Op-Ed: Beyond the Stadium: Why Real-Time Visual Intelligence Will Define World Cup 2026 Security

When the FIFA World Cup arrives in North America in 2026, it will bring more tha...

05/06/2026

FIFA+ Launches Exclusively on DAZN

FIFA and DAZN have announced the launch of FIFA exclusively on DAZN, consolidating FIFA's content portfolio within DAZN's sports platform. The move fol...

05/06/2026

Telemundo Announces Group Stage Broadcast Schedule and Talent Assignments for FIFA World Cup 2026

Telemundo's exclusive Spanish-language coverage of the FIFA World Cup 2026 G...

05/06/2026

Peacock to Stream Telemundos FIFA World Cup 2026 Coverage in Dolby Vision and Dolby Atmos

Dolby Laboratories and NBCUniversal have announced that Peacock will stream Tele...

05/06/2026

Formula 1 Extends Las Vegas Grand Prix Through 2037

Formula 1 has announced a 10-year extension to keep the Las Vegas Grand Prix on the F1 calendar through 2037. Las Vegas Grand Prix, Inc., Clark County, and the ...

05/06/2026

Inside SailGP New York: How Riedel Provides Tech Backbone for Challenging Production Effort

The broadcast-engineering team overcomes wind, speed, and salt water - and dista...

05/06/2026

Virtual Eye Powers Broadcast Graphics Across Golf Productions in Busy Three-Event Week

Deploying both onsite and remote crews, the company is providing calibrated-came...

05/06/2026

ESPN Pivots From REMI to Full On-Site Production for UFL Playoff Game on Daytonas Blank Canvas

With Inter&Co Stadium unavailable, ESPN's UFL team rebuilt its broadcast pla...

05/06/2026

Free Registration for SVG Regional Sports Production Summit Ends Today!

One of the most exciting and informative events on the SVG annual event calendar is the Regional Sports Production Summit, an annual gathering of industry profe...

05/06/2026

FOX Returns to Saratoga Race Course for 158th Running of Belmont Stakes

For the race's third year at the historic racetrack, the broadcaster has added cameras and will incorporate multiple drones The 158th edition of the Belmon...

05/06/2026

Ratings Roundup: NBCs Spurs-Thunder Game 7 Best Since 2016, ESPNs Stanley Cup Final Opener Hits Seven-Year High

Ratings Roundup is a rundown of recent rating news and is derived from press rel...

05/06/2026

Sonuscore release LUX Orchestral Strings Essentials

Built from the same recordings as flagship library Sonuscore's LUX Orchestral Strings has been met with widespread praise since its launch in late 2025,...

05/06/2026

RME unveil the ADI-2 Pro EX

High-end converter, interface & headphone amp upgraded Said to represent the next evolution of RME's all-in-one reference converter concept, the all-new...

05/06/2026

Soundgas Sounds: Latest prize draw announced

Win a Soundgas Type 636P & Type G preamps Soundgas, one of the UK's leading vintage and boutique audio equipment specialists have just announced the lau...

05/06/2026

New leadership of Technology Systems Division at Rohde & Schwarz

New leadership of Technology Systems Division at Rohde & Schwarz On July 1, 2026, Hansj rg Herrbold and Andreas H gele will take over as Executive Vice Presid...

05/06/2026

Hitachi and Intel announce strategic collaboration to accelerate AI transformation across key industries

Hitachi and Intel announced a strategic collaboration to explore opportunities t...

05/06/2026

Nielsen, Mediaocean Announce New Integration to Power The Next Phase of Data-Driven Linear, Advanced Audience Measurement

MRI-Simmons and S&P Global Mobility are expanding advanced audience capabilities...

05/06/2026

Latest Nielsen data reveals insurance advertising climbs as Australians weigh cost, cover and loyalty

New Nielsen data shows insurance ad spend grew 11%, while consumers remain highl...

05/06/2026

Nielsen Adds New Audience Data Partnerships

Share Copy link Facebook X Linkedin Bluesky Email...

05/06/2026

ASG Promotes Joe Marchitto to Western Regional CTO

ASG Promotes Joe Marchitto to Western Regional CTO Brie Clayton June 5, 2026 0 Comments Appointment to Support Engineering Alignment and Client Experi...

05/06/2026

Stargate Studios Colombia Uses DaVinci Resolve Studio for Vertical Microdramas

Stargate Studios Colombia Uses DaVinci Resolve Studio for Vertical Microdramas Brie Clayton June 5, 2026 0 Comments End to end post in one platform al...

05/06/2026

People Need to Come First When We Use AI

People Need to Come First When We Use AI Andy Marken June 5, 2026 0 Comments It's just surviving. Life's very existence requires destruction....

05/06/2026

Fahad Haider Joins NESN as VP, Operations and Engineering

Share Copy link Facebook X Linkedin Bluesky Email...

05/06/2026

GatesAir Opens New Brazil Office to Back DTV+ Rollout

Share Copy link Facebook X Linkedin Bluesky Email...

05/06/2026

Supreme Court Upholds FCC's Authority to Levy Fines

Share Copy link Facebook X Linkedin Bluesky Email...

05/06/2026

Montclair State University Will Run NJ PBS

Share Copy link Facebook X Linkedin Bluesky Email...

05/06/2026

Merkhet's Sam Matheny Urges Congress to Expedite BPS Deployment

Share Copy link Facebook X Linkedin Bluesky Email...

05/06/2026

Sinclair Launches NextGen TV Campaign in Columbus, Ohio

Share Copy link Facebook X Linkedin Bluesky Email...

05/06/2026

NAB Releases New Keep the Game On' Spot

Share Copy link Facebook X Linkedin Bluesky Email...

05/06/2026

NCTC, ACAC Going to Disney World for Independent Show

Share Copy link Facebook X Linkedin Bluesky Email...

05/06/2026

Nielsen Announces New Integrations for Improved Measurement

Share Copy link Facebook X Linkedin Bluesky Email...

05/06/2026

Frequency Launches In-Scene Advertising to Accelerate Str...

Frequency, the engine powering many of the world's leading streaming television channels, today announced the launch of In-Scene Advertising, a new monetiza...

05/06/2026

Berklee Study Reveals Video Has Become Essential to Music Careers

Berklee Study Reveals Video Has Become Essential to Music Careers Survey findings show social platforms have become the primary source of music for video cont...

05/06/2026

VEON's Kyivstar to Expand Digital Mobility Ecosystem with Acquisition of E-wings by Uklon

05 Jun 2026 VEON's Kyivstar to Expand Digital Mobility Ecosystem with Acqui...

05/06/2026

RT Radio 1 Folk Awards 2026 Date Announcement

RT Radio 1 Folk Awards to take place on Tuesday 10th November 2026, Vicar Street, Dublin Moya Brennan, D nal Lunny, Mary Black and Christy Moore among previou...

04/06/2026

Sony's New PTZ Cameras Deliver 4K 60p; New STARVIS Sensor Meets Low-Light Demands

Sony Electronics is introducing the SRG-AS10, a 4K 60p-compatible PTZ auto-frami...

04/06/2026

SVG Students To Watch: Alex Albert, Texas A&M University

This recent grad from Spring, TX, led creative-video output for the Aggies' men's basketball team last season and has been producing video and creating ...

04/06/2026

USGA Brings AI Recaps, 3D Range Tracking, and Predictive Shot Tracing to U.S. Womens Open

For the first time at a women's golf major, every player in the field will r...

04/06/2026

Panasonic PT-RQ45 Projectors Power Lille Video Mapping Festival Opera House Installation

Three Panasonic PT-RQ45 40,000-lumen 3-Chip DLP projectors made their first live...