Autonomous vehicle development and validation require the ability to replicate real-world scenarios in simulation.
At GTC, NVIDIA founder and CEO Jensen Huang showcased new AI-based tools for NVIDIA DRIVE Sim that accurately reconstruct and modify actual driving scenarios. These tools are enabled by breakthroughs from NVIDIA Research that leverage technologies such as NVIDIA Omniverse platform and NVIDIA DRIVE Map.
Huang demonstrated the methods side-by-side, showing how developers can easily test multiple scenarios in rapid iterations:
Once any scenario is reconstructed in simulation, it can act as the foundation for many different variations - from changing the trajectory of an oncoming vehicle, or adding an obstacle to the driving path - giving developers the ability to improve the AI driver.
However, reconstructing real-world driving scenarios and generating realistic data from it in simulation is a time- and labor-intensive process. It requires skilled engineers and artists, and even then, can be difficult to do.
NVIDIA has implemented two AI-based methods to seamlessly perform this process: virtual reconstruction and neural reconstruction. The first replicates the real-world scenario as a fully synthetic 3D scene, while the second uses neural simulation to augment real-world sensor data.
Both methods are able to expand well beyond recreating a single scenario to generating many new and challenging scenarios. This capability accelerates the continuous AV training, testing and validation pipeline.
Virtual Reconstruction In the keynote video above, an entire driving environment and set of scenarios around NVIDIA's headquarters are reconstructed in 3D using NVIDIA DRIVE Map, Omniverse and DRIVE Sim.
With DRIVE Map, developers have access to a digital twin of a road network in Omniverse. Using tools built on Omniverse, the detailed map is converted into a drivable simulation environment that can be used with NVIDIA DRIVE Sim.
With the reconstructed simulation environment, developers can recreate events, like a close call at an intersection or navigating a construction zone, using camera, lidar and vehicle data from real-world drives.
The platform's AI helps reconstruct the scenario. First, for each tracked object, an AI looks at camera images and finds the most similar 3D asset available from the DRIVE Sim catalog and color that most closely matches the color of the object from the video.
Finally, the actual path of the tracked object is recreated; however, there are often gaps because of occlusions. In such cases, an AI-based traffic model is applied to the tracked object to predict what it would have done and fill in the gaps in its trajectory.
Camera and lidar data from real drives are used with AI to reconstruct scenarios. Virtual reconstruction enables developers to find potentially challenging situations to train and validate the AV system with high-fidelity data generated by physically based sensors and AI behavior models that can create many new scenarios. Data from the scenario can also train the behavior model.
Neural Reconstruction The other approach relies on neural simulation rather than synthetically generating the scene, starting with real sensor data then modifying it.
Sensor replay - the process of playing back recorded sensor data to test the AV system's performance - is a staple of AV development. This process is open loop, meaning the AV stack's decisions don't affect the world since all of the data is prerecorded.
A preview of neural reconstruction methods by NVIDIA Research turn this recorded data into a fully reactive and modifiable world - as in the demo, when the originally recorded van driving past the car could be reenacted to swerve right instead. This revolutionary approach allows closed-loop testing and full interaction between the AV stack and the world it's driving in.
The process starts with recorded driving data. AI identifies the dynamic objects in the scene and removes them to create an exact replica of the 3D environment that can be rendered from new views. Dynamic objects are then reinserted into the 3D scene with realistic AI-based behaviors and physical appearance, accounting for illumination and shadows.
The AV system then drives in this virtual world and the scene reacts accordingly. The scene can be made more complex through augmented reality by inserting other virtual objects, vehicles and pedestrians which are rendered as if they were part of the real scene and can physically interact with the environment.
Every sensor on the vehicle, including camera and lidar, can be simulated in the scene using AI.
A Virtual World of Possibilities These new approaches are driven by NVIDIA's expertise in rendering, graphics and AI.
As a modular platform, DRIVE Sim supports these capabilities with a foundation of deterministic simulation. It provides the vehicle dynamics, AI-based traffic models, scenario tools and a comprehensive SDK to build any tool needed.
With these two powerful new AI methods, developers can easily move from the real world to the virtual one for faster AV development and deployment.
Most recent headlines
09/11/2025
Dalet today announced a transformative leap forward for media operations: Agentic Artificial Intelligence (AI) that unifies the Dalet ecosystem under one natura...
15/10/2025
Across the world, early-career reporters are interrogating systems that shape daily life, from mortuaries and classrooms to the corridors of legal education and...
15/10/2025
St. Louis Blues Pay Tribute to Ozzy Osbourne in New Pregame Open VideoThe NHL franchise worked with production firms 2WEI and 4th Floor CreativeBy Mark J Burns,...
15/10/2025
Cobalt Digital, G3DVu, Proxima Vision, and Telos Alliance Renew Premier Sponsors...
15/10/2025
By Paige Bethmann
Before my grandmother passed away in 2019, I went to visit her in the hospital where she'd been for a few weeks. When I arrived, the nurs...
15/10/2025
Two years ago, Spotify set out to grow the entire publishing industry by making ...
15/10/2025
People turn to DJ for a listening experience that feels personal, dynamic, and j...
15/10/2025
Las personas recurren a DJ para vivir una experiencia de escucha personal, din m...
15/10/2025
Rock legend Bruce Springsteen and luminary actor Jeremy Allen White came togethe...
15/10/2025
At Spotify, we're all about supporting rising talent and helping them connec...
15/10/2025
New SBS Documentary Series THE CANCER KILLERS Premieres 4 November on SBS & SBS ...
15/10/2025
SBS celebrates the Festivals of Lights with bold, illuminating stories across th...
15/10/2025
One of humanity's most profound questions continues: Is there any place in our galaxy suitable for life beyond Earth? NASA intends to find out, with the hel...
15/10/2025
eds3_5_jq(document).ready(function($) { $(#eds_sliderM519).chameleonSlider_2_1({ content_source:......
15/10/2025
Scality, a global leader in cyber-resilient storage software for the AI era, today announced the advancement of its comprehensive AI ecosystem certification pro...
15/10/2025
The Hollywood Professional Association (HPA) today unveiled key highlights of the 2026 HPA Tech Retreat, scheduled for Feb. 15 19 at the Westin Rancho Mirage Go...
15/10/2025
At NAB Show New York 2025 (Stand 244), Interra Systems will showcase the future of media QC, monitoring, and captioning, highlighting its award-winning, AI- and...
15/10/2025
Grass Valley today announced that RMC BFM, part of the CMA-CGM Group and France's third-largest private media group, has chosen Grass Valley's productio...
15/10/2025
Appear, the global leader in live production technology, will demonstrate how its powerful hardware and software innovations are shaping the future of hybrid, s...
15/10/2025
Rakim to Join Berklee's Hip-Hop Hall of Fame at Signature Series Concert The rap icon becomes the latest inductee, following luminaries such as Roxanne Sh...
15/10/2025
Berklee Brings Live Music to the Head of the Charles Student artists will perform on multiple riverfront stages during the 60th anniversary of the world's...
15/10/2025
CobbTV, the government access television channel for Cobb County, Ga. recently acquired an automated playout system from Pebble, a global provider of automation...
15/10/2025
OSLO, Norway Appear, the global leader in live production technology, will demonstrate how its powerful hardware and software innovations are shaping the future...
15/10/2025
NEW YORK The NBA is making major changes to the NBA App and NBA TV as it takes control of them from TNT Sports, which has long managed the league's digital ...
15/10/2025
SAN MATEO, Calif. In what promises to be a major expansion of interactive features and personalized content on the DirecTV platform, the operator and Glance hav...
15/10/2025
SAN JOSE, Calif. Roku has launched changes to its user interface (UI) that the streaming platform says will better showcase original programming on the platform...
15/10/2025
LOS ANGELES Software-defined data storage and data services provider OpenDrives has elevated Alex Dunfey to chief technology officer, responsible for driving th...
15/10/2025
Abu Dhabi, UAE October 15, 2025: Space42 (ADX: SPACE42), the UAE-based AI-powe...
15/10/2025
Sheldon Nichols and Will Trickett to find and prepare more classic cars for budget conscious would-be owners from Wiser Films
UKTV have recommissioned the clas...
15/10/2025
Wednesday 15 October 2025
Nick Cave
JPEG (275 KB)
Explore the creative univer...
15/10/2025
Wednesday 15 October 2025
From breaking up with your partner to quitting your job - new polling shows some messages are just too scary to send
Over a fifth (...
15/10/2025
Rohde & Schwarz unveils compact MXO 3 oscilloscopes with 4 and 8 channels: Advan...
15/10/2025
Back to All News
Revenge Series The Resurrected' Captivates Audiences Across Asia
Entertainment
15 October 2025
GlobalTaiwan
Link copied to clipboard
...
15/10/2025
Back to All News
Bringing the Best in VFX and Virtual Production Together as Eyeline
Jeffrey Shapiro
CEO, Eyeline
Business
15 October 2025
GlobalCanadaInd...
15/10/2025
Deployed by Astound Business Solutions, Harmonic's Primary Distribution Solution Ensures Outstanding Video Quality and Seamless Ad Insertion at the Edge
SA...
15/10/2025
Collaboration extends a trusted relationship as RMC BFM invests in scalable, fut...
15/10/2025
Series coming in 2026 stars Tom Vaughan-Lawlor, Justine Mitchell and Jason O'Mara released today
RT today released first look images of new comedy-drama ...
14/10/2025
SVG Europe Summit 2025: All Sessions Now Available to Watch on SVG PLAYNetworking event that preceded IBC2025 shone a light on elite live sports innovation acro...
14/10/2025
SVG Sit-Down: Author Rich Podolsky on Writing Madden & Summerall: How They Revo...
14/10/2025
SVG All-Stars: Michael Reiners, Coordinating Producer, FloRacingThe Illinois State grad steers a vast schedule of motorsports events at tracks across the countr...
14/10/2025
Content protection: Getting the right management for your DRM By Neal Romanek
Friday, October 10, 2025 - 10:11
Print This Story
Eluvio power the EPCR'...
14/10/2025
As League Takes Over Ops, NBA TV and NBA App Add 60 Games, Weekday Studio Show, ...
14/10/2025
Time and effort: World's largest student-led broadcast prepares to go On Air...
14/10/2025
(L-R) Guest, Kimberly Robinson Jones, Geeta Gandbhir, Pamela Dias, and Takema Ro...
14/10/2025
Lossless ist jetzt mit Spotify Premium verf gbar.
Verlustfreies Audio war eine...
14/10/2025
La qualit Lossless est disponible sur Spotify Premium.
Le format sans perte de...
14/10/2025
For the seventh edition of Spotify and FC Barcelona's artist jersey series, ...
14/10/2025
Spotify is committed to bringing the best listening experience to all our users, and that includes parents and families. That's why we're expanding mana...
14/10/2025
Since its debut, the Spotify Original podcast Caso 63 has been more than just a story; it's been a cultural sensation. The science fiction thriller captivat...
Autonomous vehicle development and validation require the ability to replicate real-world scenarios in simulation.
