The gap between reality and simulation just became narrower.
In his keynote at GTC, NVIDIA founder and CEO Jensen Huang announced NVIDIA Omniverse Replicator, an engine for generating synthetic data with ground truth for training AI networks. In a demo, Huang showed the power of Omniverse Replicator when applied to autonomous vehicle development using DRIVE Sim.
DRIVE Sim is a simulation tool built on Omniverse that takes advantage of the many capabilities of the platform. Data generated by DRIVE Sim is used to train deep neural networks that make up the perception systems in autonomous vehicles. For the NVIDIA DRIVE team, synthetic data has been an effective, critical component of its AV development workflow.
The deep neural networks powering an autonomous vehicle's perception are composed of two parts: an algorithmic model, and the data used to train that model. Engineers have dedicated significant time to refining algorithms. However, the data side of the equation is still underdeveloped due to the limitations of real-world data, which is incomplete and time consuming and costly to collect.
This imbalance often leads to a plateau in DNN development, hindering progress where the data cannot meet the demands of the model. With synthetic data generation, developers have more control over data development, tailoring it to the specific needs of the model.
While real-world data is a critical component for AV training, testing and validation, it presents significant challenges. The data used to train these networks is collected by sensors on a vehicle fleet during real-world drives. Once captured, the data must be labeled with the ground truth. Annotation is done by hand by thousands of labelers - a process that is time consuming, costly and can be inaccurate.
Augmenting real-world data collection with synthetic data removes these bottlenecks while allowing engineers to take a data-driven approach to DNN development, significantly accelerating AV development and improving real-world results.
The Sim-Real Domain Gap Problem Synthetic data generation is a well-known tool for AI training - researchers have been experimenting with video games such as Grand Theft Auto to create data as early as 2016.
Unlike video games, however, the quality of perception DNNs is heavily influenced by the fidelity of the data to the real world - training on datasets that don't translate to the physical world can actually degrade a network's performance.
This sim-to-real gap presents itself in two ways primarily. An appearance gap corresponds to pixel-level differences between the simulated image and the real image, which is caused by how the simulator generates the data. The renderer, sensor model, fidelity of 3D assets and material properties can all contribute to this.
A content gap can be caused by the lack of real-world content diversity and by differences between sim and real-world contexts. These inconsistencies occur when the context of a scene doesn't match reality. For example, the real world contains dirty roads, dented cars, and emergency vehicles on roadsides, which all must be reproduced in simulation. Another important factor is the behavior of actors, such as traffic and pedestrians - realistic interactions are key to realistic data output.
Applying Omniverse Replicator to Narrow the Sim-Real Domain Gap Omniverse Replicator is designed to narrow the appearance and content gaps.
To narrow the appearance gap, DRIVE Sim takes advantage of Omniverse's RTX path-tracing renderer to generate physically based sensor data for cameras, radars and lidars and ultrasonics sensors. Real-world effects are captured in the sensor data, including phenomena such as LED flicker, motion blur, rolling shutter, lidar beam divergence and doppler effect. These details even include high-fidelity vehicle dynamics, which are important because, for example, the movement of a vehicle during a lidar scan impacts the resulting point cloud.
The other half of this sensor equation is materials. Materials in DRIVE Sim are physically simulated for accurate beam reflections. DRIVE Sim includes a built-in lidar materials library and a forthcoming radar and ultrasonics materials library.
DRIVE Sim's sensor capabilities include path-traced camera, radar and lidar models that capture real-world effects such as motion blur, LED flicker, rolling shutter and doppler effect.
DRIVE Sim uses the RTX path-tracer to render these early morning and night time scenes with incredible fidelity. One of the primary ways to address the content gap is with more diverse assets at the highest levels of fidelity. DRIVE Sim leverages Omniverse's capabilities to connect to a wide range of content creation tools. However, generating proper scenes also requires that the context is correct.
Under the hood, Omniverse Replicator organizes data for rapid scene manipulation using a capability called domain randomization. DRIVE Sim includes tools for this and scene construction that create a large amount of diverse data while maintaining real-world context. Since Omniverse Replicator is also time accurate and deterministic, the dataset can be created in repeatable way.
DRIVE Sim provides tools for generating randomized scenes in a controlled and repeatable way that adds variety and diversity to the data generated. (Click image to play.) Real Results DRIVE Sim has already produced significant results in accelerating perception development with synthetic data at NVIDIA.
One example is the migration to the latest NVIDIA DRIVE Hyperion sensor set. The NVIDIA DRIVE Hyperion 8 platform includes sensors for complete production AV development. However, before these sensors were even available, the NVIDIA DRIVE team was able to bring up DNNs for the platform using synthetic data. DRIVE Sim generated millions of images and ground-trut
Most recent headlines
04/08/2024
Dalet, a leading technology and service provider for media-rich organizations, is excited to announce Santiago Solanas as its new Chief Executive Officer (CEO)....
03/06/2024
Dalet, a leading technology and service provider for media-rich organizations, a...
01/06/2024
Busy this week catching up from the Memorial Day holiday? Our TV Tech weekly wrap-up of all our coverage of new products, services and deployments will get you ...
01/06/2024
Young Sherlock, with Guy Ritchie directing and executive producing, is a go at Prime Video. Hero Fiennes Tiffin plays Sherlock. Prime Video calls it the origin...
01/06/2024
The Los Angeles Sparks-Indiana Fever game on Friday featuring the top two WNBA draft picks, Caitlin Clark and Cameron Brink, was the most watched WNBA game on I...
01/06/2024
Lionsgate, which increased its majority stake in 3 Arts Entertainment in January, said it named Brian Weinstein co-CEO of 3 Arts....
01/06/2024
Netflix is launching four video games based on unscripted series this year. They...
01/06/2024
ABC, CBS, NBC and Fox affiliates switched from their regular programming to carry live coverage of the verdicts in the porn star hush-money trial of former Pres...
31/05/2024
A U.S. Marine fires an FIM-92 Stinger missile during a training exercise in Yuma...
31/05/2024
NEW YORK The global body that sets technical standards for digital advertising h...
31/05/2024
In a new analysis of what services might make up a good bundle and what services might benefit the most from bundling, Antenna has released new data that highli...
31/05/2024
Global SVOD revenues will increase by $116.6 billion by 2031, a growth rate of 13.3% per year over the next eight years, according to a new report from MIDiA Re...
31/05/2024
WASHINGTON, D.C. The FCC Media Bureau has announced that effective August 1, 2024 the Commission's TVStudy software will incorporate the most recent U.S. Ce...
31/05/2024
NBCUniversal has announced that its FAST channel TNBC will be rebranding to NBC Comedy Vault on June 1 and that the free streaming service will officially launc...
31/05/2024
PARIS Ateme has announced that the Indonesian public broadcaster TVRI is making the transition to 4K UHD OTT streaming using Ateme technologies....
31/05/2024
The proposal had received support from James Cameron, Andy Serkis, Paul Greengrass, Richard Curtis, and Jeremy Irons
By Matthew Corrigan
Published: May 31, 2...
31/05/2024
According to multiple reports, the new bid from Skydance offers a better deal to Paramounts Class B shareholders, who were said to be unhappy with the companys ...
31/05/2024
The awards recognise women and companies who are stand-out performers, innovators and leaders in the broadcast media technology industry
By Matthew Corrigan
...
31/05/2024
NEW YORK The global body that sets technical standards for digital advertising has expanded the reach of its measurement software development kit to include Sam...
31/05/2024
The rise of digital platforms is fundamentally changing the broadcast newsroom and its investment in technology according to Jon Roberts, director of technology...
31/05/2024
In a new analysis of what services might make up a good bundle and what services might benefit the most from bundling, Antenna has released new data that highli...
31/05/2024
WASHINGTON, D.C. Fifty state broadcasters associations, including the District of Columbia and the Commonwealth of Puerto Rico, have sent a letter to Congressio...
31/05/2024
EVS has announced the appointment of Richard Katz as senior vice president of operations for the North and Latin American (NALA) region. This appointment is par...
31/05/2024
CommScope has announced that it was selected by Casa Systems, Inc. as the highest to acquire Casa's cable business assets and that the two companies have en...
31/05/2024
Robert De Niro won't be receiving the Service to America Leadership Award from the NAB Leadership Foundation after all....
31/05/2024
Viant Technology said it formed an integration with Google Cloud's BigQuery data clean rooms...
31/05/2024
The CW said it will broadcast the 2024 Snoop Dogg Arizona Bowl presented by Gin & Juice by Dre and Snoop Last year, The CW aired the game when it was called Bar...
31/05/2024
Allen Media Group said it made a deal with Amazon's Fire TV Channel that will make content from several of Allen's outlet available via Fire TV and Echo...
31/05/2024
Robert De Niro, announced as the winner of the NAB's Leadership Foundation's Service to America Leadership Award on May 28, will not get the award. The ...
31/05/2024
Comcast Technology Solutions was selected by AccuWeather to create, manage and distribute linear and over-the-top video channels....
31/05/2024
Connected TV is becoming a bigger deal for consumers and advertisers, according to a new report from TVision....
31/05/2024
By Craig Anderton
Let's get right to what this sounds like. It's not quite feedback or tape reverse, it's well, listen to what it does in this blue...
31/05/2024
31 May 2024
VEON discloses April YTD trading update at its AGM Amsterdam, 31 May 2024 07:00 CEST -
VEON April YTD 2024 Highlights
Total revenue of USD 1,278 ...
31/05/2024
31 May 2024
VEON Announces Its New Board, Names Founder Augie Fabela as Chairman Amsterdam, 31 May 2024, 16.00 CET: VEON Ltd. (NASDAQ: VEON, Euronext Amsterdam...
31/05/2024
May 31st, 2024
TRIBECA FESTIVAL AND OPEN AI ANNOUNCE SORA SHORTS'
Five Commissioned Short Films Created Using Sora Debut at 2024 Tribeca Festival
New ...
31/05/2024
In-Venue Technology Wrap-Up: How 4K, IP, and More are Changing the Game Pres Gam...
31/05/2024
UEFA Champions League Final 2024: The host broadcast facts and figures for Real ...
31/05/2024
UEFA Europa Conference League Final 2024: Bringing the broadcast to global TV sc...
31/05/2024
Willow Offers Free 2024 ICC T20 Cricket World Cup Preview via DirecTV, Verizon, ...
31/05/2024
Women's College World Series: ESPN Brings Live Drone, Two-Point Cabled, and ...
31/05/2024
SVG College Summit Brings Video-Production Industry Together in Atlanta in the M...
31/05/2024
UEFA Champions League Final 2024: CBS Sports' UCL Today Studio Show Is Pitch...
31/05/2024
Back to All News
Master of the House' Unveils a Scandalous Tale of Power and Greed on July 18
Entertainment
31 May 2024
GlobalThailand
Link copied to ...
31/05/2024
Back to All News
JEEtu Bhaiya is Back: Kota Factory' Season 3 Arrives on Netflix June 20
Entertainment
31 May 2024
GlobalIndia
Link copied to clipboar...
31/05/2024
Facebook
Twitter
LinkedIn
With around 9,000 employees, Ground Transportation Systems is a global leader in Rail Signaling and Train Control Systems, Telec...
31/05/2024
RT 2FM is the home of music this summer, entertaining listeners across the day and night, with the best music, chat and interviews providing the soundtrack to ...
31/05/2024
After the 2008 financial crisis and increased risk-management regulations that f...
31/05/2024
RT Supporting the Arts: What's On this June
Watch our promo: RT Supporting the Arts | What's on YouTube
This June, RT is delighted to support Lim...
30/05/2024
When to Upgrade Software or Firmware This post is from our blog and news archive. The information may be out of date. Please contact us for further information ...
The gap between reality and simulation just became narrower.
