Sony Pixel Power calrec Sony

Knight Rider Rides a GAN: Bringing KITT to Life with AI, NVIDIA Omniverse

16/04/2021

Fasten your seatbelts. NVIDIA Research is revving up a new deep learning engine that creates 3D object models from standard 2D images - and can bring iconic cars like the Knight Rider's AI-powered KITT to life - in NVIDIA Omniverse.

Developed by the NVIDIA AI Research Lab in Toronto, the GANverse3D application inflates flat images into realistic 3D models that can be visualized and controlled in virtual environments. This capability could help architects, creators, game developers and designers easily add new objects to their mockups without needing expertise in 3D modeling, or a large budget to spend on renderings.

A single photo of a car, for example, could be turned into a 3D model that can drive around a virtual scene, complete with realistic headlights, tail lights and blinkers.

To generate a dataset for training, the researchers harnessed a generative adversarial network, or GAN, to synthesize images depicting the same object from multiple viewpoints - like a photographer who walks around a parked vehicle, taking shots from different angles. These multi-view images were plugged into a rendering framework for inverse graphics, the process of inferring 3D mesh models from 2D images.

Once trained on multi-view images, GANverse3D needs only a single 2D image to predict a 3D mesh model. This model can be used with a 3D neural renderer that gives developers control to customize objects and swap out backgrounds.

When imported as an extension in the NVIDIA Omniverse platform and run on NVIDIA RTX GPUs, GANverse3D can be used to recreate any 2D image into 3D - like the beloved crime-fighting car KITT, from the popular 1980s Knight Rider TV show.

Previous models for inverse graphics have relied on 3D shapes as training data.

Instead, with no aid from 3D assets, We turned a GAN model into a very efficient data generator so we can create 3D objects from any 2D image on the web, said Wenzheng Chen, research scientist at NVIDIA and lead author on the project.

Because we trained on real images instead of the typical pipeline, which relies on synthetic data, the AI model generalizes better to real-world applications, said NVIDIA researcher Jun Gao, an author on the project.

The research behind GANverse3D will be presented at two upcoming conferences: the International Conference on Learning Representations in May, and the Conference on Computer Vision and Pattern Recognition, in June.

From Flat Tire to Racing KITT Creators in gaming, architecture and design rely on virtual environments like the NVIDIA Omniverse simulation and collaboration platform to test out new ideas and visualize prototypes before creating their final products. With Omniverse Connectors, developers can use their preferred 3D applications in Omniverse to simulate complex virtual worlds with real-time ray tracing.

But not every creator has the time and resources to create 3D models of every object they sketch. The cost of capturing the number of multi-view images necessary to render a showroom's worth of cars, or a street's worth of buildings, can be prohibitive.

That's where a trained GANverse3D application can be used to convert standard images of a car, a building or even a horse into a 3D figure that can be customized and animated in Omniverse.

To recreate KITT, the researchers simply fed the trained model an image of the car, letting GANverse3D predict a corresponding 3D textured mesh, as well as different parts of the vehicle such as wheels and headlights. They then used NVIDIA Omniverse Kit and NVIDIA PhysX tools to convert the predicted texture into high-quality materials that give KITT a more realistic look and feel, and placed it in a dynamic driving sequence.

Omniverse allows researchers to bring exciting, cutting-edge research directly to creators and end users, said Jean-Francois Lafleche, deep learning engineer at NVIDIA. Offering GANverse3D as an extension in Omniverse will help artists create richer virtual worlds for game development, city planning or even training new machine learning models.

GANs Power a Dimensional Shift Because real-world datasets that capture the same object from different angles are rare, most AI tools that convert images from 2D to 3D are trained using synthetic 3D datasets like ShapeNet.

To obtain multi-view images from real-world data - like images of cars available publicly on the web - the NVIDIA researchers instead turned to a GAN model, manipulating its neural network layers to turn it into a data generator.

The team found that opening the first four layers of the neural network and freezing the remaining 12 caused the GAN to render images of the same object from different viewpoints.

Keeping the first four layers frozen and the other 12 layers variable caused the neural network to generate different images from the same viewpoint. By manually assigning standard viewpoints, with vehicles pictured at a specific elevation and camera distance, the researchers could rapidly generate a multi-view dataset from individual 2D images.

The final model, trained on 55,000 car images generated by the GAN, outperformed an inverse graphics network trained on the popular Pascal3D dataset.

Read the full ICLR paper, authored by Wenzheng Chen, fellow NVIDIA researchers Jun Gao and Huan Ling, Sanja Fidler, director of NVIDIA's Toronto research lab, University of Waterloo student Yuxuan Zhang, Stanford student Yinan Zhang and MIT professor Antonio Torralba. Additional collaborators on the CVPR paper include Jean-Francois Lafleche, NVIDIA researcher Kangxue Yin and Adela Barriuso.

The NVIDIA Research team consists of more than 200 scientists around the globe, focusing on areas such as AI, computer vision, self-driving cars, robotics and graphics. Learn more about the company's latest research and industry breakthroughs in NVIDIA CEO Jensen Huang's k
LINK: https://blogs.nvidia.com/blog/2021/04/16/gan-research-knight-rider-ai-...
See more stories from nvidia

North America Stories

11/05/2026

Ross Production Services 48-Foot HyperMax-1 Truck Quickly in Demand

New production unit features not only wealth of Ross gear but also Sony cameras, Canon lenses, and a Calrec Argo S audio board...

11/05/2026

Solid State Logic Launches TCA Tour Portable Audio Production System

Solid State Logic (SSL) has announced TCA Tour, a portable fly-away audio production system built from System T components for broadcast, touring, and live prod...

11/05/2026

Riedel Communications Named Official Connectivity Integration Provider for Glasgow 2026 Commonwealth Games

Riedel Communications has announced it will serve as Official Connectivity Integ...

11/05/2026

G&D and NETGEAR AV Launch KVM-over-IP Plugin for Automated Network Configuration

Guntermann and Drunck (G&D) and NETGEAR AV have announced a plugin that automates network configuration for G&D KVM-over-IP deployments on NETGEAR AV Line infra...

11/05/2026

The Famous Group and Elite Edge Announce Unified Ownership

The Famous Group (TFG) and Elite Edge, a production studio specializing in live-action production, show opens, set design, and fabrication for sports and live e...

11/05/2026

SVG All-Stars: Odair Auger, Tech Project Manager, NBCUniversal Telemundo Enterprises

The native of S o Paulo, Brazil, will play a crucial role in the broadcaster'...

11/05/2026

Cobalt Digital to Showcase IPMX/ST 2110 Ecosystem at BroadcastAsia 2026

Cobalt Digital will exhibit its IPMX/ST 2110 product lineup at BroadcastAsia 2026 (Stand 5D2-4), anchored by the COBALT blueCORE family of standalone signal pro...

11/05/2026

Interra Systems to Demonstrate QC, Monitoring, and Captioning Solutions at BroadcastAsia 2026

Interra Systems will exhibit at BroadcastAsia 2026 (Stand 5C1-10), demonstrating...

11/05/2026

T-Mobile Deploys 5G and AI Tools Across 2026 PGA Championship

T-Mobile is serving as a technology partner for the 2026 PGA Championship at Aronimink Golf Club in Newtown Square, Pennsylvania, deploying 5G connectivity and ...

11/05/2026

Vizrt Named Official Technical Supplier of Eurovision Song Contest 2026

Vizrt has announced it has been named an Official Technical Supplier of the Eurovision Song Contest 2026, in partnership with host broadcaster ORF. The 70th edi...

11/05/2026

WNBA and AWS Announce Multi-Year Partnership

The WNBA and Amazon Web Services (AWS) have announced a multi-year partnership making AWS the Official Cloud and Cloud AI Partner of the WNBA. AWS is also joini...

11/05/2026

NAB 2026: kronehit Awards Lawo and SLG Broadcast Contract for Full Studio Modernization

Austrian private broadcaster kronehit publicly awarded SLG Broadcast AG and Lawo...

11/05/2026

Behind The Mic: PGA Championship Changing Weekday Announcer Lineup for 2026 Tournament

Behind The Mic provides a roundup of recent news regarding on-air talent, includ...

11/05/2026

L3Harris Reaches Over 50% in FAA Telecommunications Modernization

L3Harris has completed more than 50% of a nationwide effort to rebuild the invisible backbone of America's airspace....

11/05/2026

CBS Sacramento Debuts New AR/VR Studio

Share Copy link Facebook X Linkedin Bluesky Email...

11/05/2026

Gray Media Advances James Fitch to Senior VP, News Services

Share Copy link Facebook X Linkedin Bluesky Email...

11/05/2026

Smarter Monetization Stronger Retention and PurposeDriven...

As Asia Pacific's streaming market continues to evolve, operators across the region are rebalancing their strategies and shifting their focus from subscribe...

11/05/2026

QuickLink to Highlight Latest StudioEdge Models StudioPr...

LONDON, MAY 11, 2026 QuickLink, a leading provider of award-winning video production and remote guest contribution solutions, will showcase its latest innovat...

11/05/2026

Cobalt Digitals BroadcastAsia Line up Highlights Flexible...

Cobalt Digital's BroadcastAsia Line-up Highlights Flexible End-to-End IPMX/ST 2110 Ecosystem Award-winning blueCORE standalone signal processors at the he...

11/05/2026

Encompass and VideoMagic Launch Altitude Intelligence to...

Encompass Digital Media, in partnership with VideoMagic International, announced at NAB 2026 the launch of Altitude Intelligence, an AI-powered platform designe...

11/05/2026

Louis Libin Preps for the World Cup's Spectrum Crunch

Share Copy link Facebook X Linkedin Bluesky Email...

11/05/2026

Report: Broadcast Employment Hard Hit by AI

Share Copy link Facebook X Linkedin Bluesky Email...

11/05/2026

Alfalite installs 150 LED panels in Torre Iberdrola audit...

Alfalite, the only European manufacturer of LED screens, has completed a major new corporate installation in Spain with the upgrade of the auditorium at Torre I...

10/05/2026

Berklee Honors Jacob Collier, Jill Scott, and Vinnie Colaiuta at 2026 Commencement

Berklee Honors Jacob Collier, Jill Scott, and Vinnie Colaiuta at 2026 Commenceme...

09/05/2026

New SoundApp ARA Plugin Transforms Boris FX CrumplePop

New SoundApp ARA Plugin Transforms Boris FX CrumplePop Jessie Electa Petrov May 9, 2026 0 Comments CrumplePop 2026.5 streamlines music, voice, and cin...

09/05/2026

Celebrating a TV Legend: Sir David Attenborough

Share Copy link Facebook X Linkedin Bluesky Email...

09/05/2026

Playmetrics Acquires SportsEngine From Versant

Share Copy link Facebook X Linkedin Bluesky Email...

08/05/2026

NBC Sports, Peacock Scale Vertical-Video Production Across Sports - and Beyond - Ahead of FIFA World Cup 2026

NBC's 9:16 live-production workflows power Courtside Live, Rinkside Live, Vi...

08/05/2026

ESPN Pieces Together Production Strategy After WNBA's Unsettled Offseason

Native 1080p HDR productions will rely on a mix of remote workflows from Bristol and Los Angeles Despite another prosperous WNBA season in 2025, there was no g...

08/05/2026

SVG GameDay, Ep. 14: Buffalo Sabres' Autumn Bolton - Bringing Back Better Days at KeyBank Center

In-venue and creative video staffers at the professional and collegiate level ha...

08/05/2026

ESPN and Players Era Announce Multi-Year Broadcast Agreement for Mens College Basketball Championships

Players Era, an EverWonder Studio company, and ESPN have announced a multi-year ...

08/05/2026

BFBS Selects Synamedia to Build Next-Generation Digital Platform for UK Armed Forces

BFBS (British Forces Broadcasting Service) has announced the selection of Syname...

08/05/2026

UFC and Joe Hand Promotions Launch Championship Rounds Commercial Subscription Package

UFC and Joe Hand Promotions, UFC's exclusive commercial distributor in the U...

08/05/2026

Ease Live Granted US Patent for Interactive Streaming Overlay Technology

Ease Live, an Evertz company, has announced it has been granted US Patent No. 12,556,777 for its graphical overlay technology for interactive live streaming. Th...

08/05/2026

Ateme Deploys OTT Platform for VTVcabs VTVprime Service in Vietnam

Ateme has announced that VTVcab has deployed Ateme's OTT platform to power its VTVprime streaming service in Vietnam. The deployment covers live and on-dema...

08/05/2026

Amagi Launches In-Content Ads Marketplace for CTV Advertising

Amagi has announced the availability of its In-Content Ads offering via the Amagi ADS PLUS marketplace, providing programmatic access to ad formats inserted dir...

08/05/2026

USA Sports Dives Headfirst Into Its Inaugural WNBA Season With Onsite Postseason-Style' Productions

Young broadcaster will deploy 11-camera complement on large chunk of games, At ...

08/05/2026

Ratings Roundup: NBCs Derby Coverage Hits All-Time High; 1st Round of NBA, NHA Playoffs Up Big

Ratings Roundup is a rundown of recent rating news and is derived from press rel...

08/05/2026

A Mother's Day Toast to Joan Rubinsky in Theater Camp

Molly Gordon and Ben Platt appear in Theater Camp. (Courtesy of Searchlight Pictures) By Lucy Spicer Her character is the beloved founder of the AdirondACTS...

08/05/2026

G&D KVM-over-IP Plugin for Netgear AV Simplifies Deployments

Share Copy link Facebook X Linkedin Bluesky Email...

08/05/2026

Riedel to Support Glasgow 2026 as Official Connectivity I...

Riedel Communications today announced its return as a sponsor of the Glasgow 2026 Commonwealth Games, where the company will serve as the Official Connectivity ...

08/05/2026

Miri V410 Encoder-Decoder Wins 2026 NAB Show Product of t...

For the second straight year, Miri Technologies Inc. has been honored with a pair of coveted awards at the broadcast, media and entertainment industry's pre...

08/05/2026

Nine Alumni Nominated for Tony Awards

Nine Alumni Nominated for Tony Awards More than 20 alumni contributed to this year's Tony-nominated productions. May 7, 2026 By Tori Donahue Boston Co...

08/05/2026

Report: Rise of Video Podcasts Prompts Hybrid Approach

Share Copy link Facebook X Linkedin Bluesky Email...

08/05/2026

U.K. Military Broadcaster Taps Synamedia for Digital Platform

Share Copy link Facebook X Linkedin Bluesky Email...

08/05/2026

Gayle Benson To Receive NAB Leadership Foundation Leadership Award

Share Copy link Facebook X Linkedin Bluesky Email...

08/05/2026

Dems Blast FCC Disney Probe As 'Egregious Abuse of Power'

Share Copy link Facebook X Linkedin Bluesky Email...

08/05/2026

Liberty University on why broadcast technology isn't just a technical investment, it's a human one

Liberty University on why broadcast technology isn't just a technical invest...

08/05/2026

COW Jobs: UGC, On-Camera Video Content

COW Jobs: UGC, On-Camera Video Content Brie Clayton May 7, 2026 0 Comments UGC, On-Camera Video Content April 17, 2026Documentary Editor - US,......