NVIDIA Omniverse Replicator for DRIVE Sim Accelerates AV Development, Improves Perception Results

The gap between reality and simulation just became narrower.

In his keynote at GTC, NVIDIA founder and CEO Jensen Huang announced NVIDIA Omniverse Replicator, an engine for generating synthetic data with ground truth for training AI networks. In a demo, Huang showed the power of Omniverse Replicator when applied to autonomous vehicle development using DRIVE Sim.

DRIVE Sim is a simulation tool built on Omniverse that takes advantage of the many capabilities of the platform. Data generated by DRIVE Sim is used to train deep neural networks that make up the perception systems in autonomous vehicles. For the NVIDIA DRIVE team, synthetic data has been an effective, critical component of its AV development workflow.

The deep neural networks powering an autonomous vehicle's perception are composed of two parts: an algorithmic model, and the data used to train that model. Engineers have dedicated significant time to refining algorithms. However, the data side of the equation is still underdeveloped due to the limitations of real-world data, which is incomplete and time consuming and costly to collect.

This imbalance often leads to a plateau in DNN development, hindering progress where the data cannot meet the demands of the model. With synthetic data generation, developers have more control over data development, tailoring it to the specific needs of the model.

While real-world data is a critical component for AV training, testing and validation, it presents significant challenges. The data used to train these networks is collected by sensors on a vehicle fleet during real-world drives. Once captured, the data must be labeled with the ground truth. Annotation is done by hand by thousands of labelers - a process that is time consuming, costly and can be inaccurate.

Augmenting real-world data collection with synthetic data removes these bottlenecks while allowing engineers to take a data-driven approach to DNN development, significantly accelerating AV development and improving real-world results.

The Sim-Real Domain Gap Problem Synthetic data generation is a well-known tool for AI training - researchers have been experimenting with video games such as Grand Theft Auto to create data as early as 2016.

Unlike video games, however, the quality of perception DNNs is heavily influenced by the fidelity of the data to the real world - training on datasets that don't translate to the physical world can actually degrade a network's performance.

This sim-to-real gap presents itself in two ways primarily. An appearance gap corresponds to pixel-level differences between the simulated image and the real image, which is caused by how the simulator generates the data. The renderer, sensor model, fidelity of 3D assets and material properties can all contribute to this.

A content gap can be caused by the lack of real-world content diversity and by differences between sim and real-world contexts. These inconsistencies occur when the context of a scene doesn't match reality. For example, the real world contains dirty roads, dented cars, and emergency vehicles on roadsides, which all must be reproduced in simulation. Another important factor is the behavior of actors, such as traffic and pedestrians - realistic interactions are key to realistic data output.

Applying Omniverse Replicator to Narrow the Sim-Real Domain Gap Omniverse Replicator is designed to narrow the appearance and content gaps.

To narrow the appearance gap, DRIVE Sim takes advantage of Omniverse's RTX path-tracing renderer to generate physically based sensor data for cameras, radars and lidars and ultrasonics sensors. Real-world effects are captured in the sensor data, including phenomena such as LED flicker, motion blur, rolling shutter, lidar beam divergence and doppler effect. These details even include high-fidelity vehicle dynamics, which are important because, for example, the movement of a vehicle during a lidar scan impacts the resulting point cloud.

The other half of this sensor equation is materials. Materials in DRIVE Sim are physically simulated for accurate beam reflections. DRIVE Sim includes a built-in lidar materials library and a forthcoming radar and ultrasonics materials library.

DRIVE Sim's sensor capabilities include path-traced camera, radar and lidar models that capture real-world effects such as motion blur, LED flicker, rolling shutter and doppler effect.

DRIVE Sim uses the RTX path-tracer to render these early morning and night time scenes with incredible fidelity. One of the primary ways to address the content gap is with more diverse assets at the highest levels of fidelity. DRIVE Sim leverages Omniverse's capabilities to connect to a wide range of content creation tools. However, generating proper scenes also requires that the context is correct.

Under the hood, Omniverse Replicator organizes data for rapid scene manipulation using a capability called domain randomization. DRIVE Sim includes tools for this and scene construction that create a large amount of diverse data while maintaining real-world context. Since Omniverse Replicator is also time accurate and deterministic, the dataset can be created in repeatable way.

DRIVE Sim provides tools for generating randomized scenes in a controlled and repeatable way that adds variety and diversity to the data generated. (Click image to play.) Real Results DRIVE Sim has already produced significant results in accelerating perception development with synthetic data at NVIDIA.

One example is the migration to the latest NVIDIA DRIVE Hyperion sensor set. The NVIDIA DRIVE Hyperion 8 platform includes sensors for complete production AV development. However, before these sensors were even available, the NVIDIA DRIVE team was able to bring up DNNs for the platform using synthetic data. DRIVE Sim generated millions of images and ground-trut

LINK:	https://blogs.nvidia.com/blog/2021/11/09/drive-sim-replicator-syntheti...
	See more stories from nvidia

Most recent headlines