Sony Pixel Power calrec Sony

Knight Rider Rides a GAN: Bringing KITT to Life with AI, NVIDIA Omniverse

16/04/2021

Fasten your seatbelts. NVIDIA Research is revving up a new deep learning engine that creates 3D object models from standard 2D images - and can bring iconic cars like the Knight Rider's AI-powered KITT to life - in NVIDIA Omniverse.

Developed by the NVIDIA AI Research Lab in Toronto, the GANverse3D application inflates flat images into realistic 3D models that can be visualized and controlled in virtual environments. This capability could help architects, creators, game developers and designers easily add new objects to their mockups without needing expertise in 3D modeling, or a large budget to spend on renderings.

A single photo of a car, for example, could be turned into a 3D model that can drive around a virtual scene, complete with realistic headlights, tail lights and blinkers.

To generate a dataset for training, the researchers harnessed a generative adversarial network, or GAN, to synthesize images depicting the same object from multiple viewpoints - like a photographer who walks around a parked vehicle, taking shots from different angles. These multi-view images were plugged into a rendering framework for inverse graphics, the process of inferring 3D mesh models from 2D images.

Once trained on multi-view images, GANverse3D needs only a single 2D image to predict a 3D mesh model. This model can be used with a 3D neural renderer that gives developers control to customize objects and swap out backgrounds.

When imported as an extension in the NVIDIA Omniverse platform and run on NVIDIA RTX GPUs, GANverse3D can be used to recreate any 2D image into 3D - like the beloved crime-fighting car KITT, from the popular 1980s Knight Rider TV show.

Previous models for inverse graphics have relied on 3D shapes as training data.

Instead, with no aid from 3D assets, We turned a GAN model into a very efficient data generator so we can create 3D objects from any 2D image on the web, said Wenzheng Chen, research scientist at NVIDIA and lead author on the project.

Because we trained on real images instead of the typical pipeline, which relies on synthetic data, the AI model generalizes better to real-world applications, said NVIDIA researcher Jun Gao, an author on the project.

The research behind GANverse3D will be presented at two upcoming conferences: the International Conference on Learning Representations in May, and the Conference on Computer Vision and Pattern Recognition, in June.

From Flat Tire to Racing KITT Creators in gaming, architecture and design rely on virtual environments like the NVIDIA Omniverse simulation and collaboration platform to test out new ideas and visualize prototypes before creating their final products. With Omniverse Connectors, developers can use their preferred 3D applications in Omniverse to simulate complex virtual worlds with real-time ray tracing.

But not every creator has the time and resources to create 3D models of every object they sketch. The cost of capturing the number of multi-view images necessary to render a showroom's worth of cars, or a street's worth of buildings, can be prohibitive.

That's where a trained GANverse3D application can be used to convert standard images of a car, a building or even a horse into a 3D figure that can be customized and animated in Omniverse.

To recreate KITT, the researchers simply fed the trained model an image of the car, letting GANverse3D predict a corresponding 3D textured mesh, as well as different parts of the vehicle such as wheels and headlights. They then used NVIDIA Omniverse Kit and NVIDIA PhysX tools to convert the predicted texture into high-quality materials that give KITT a more realistic look and feel, and placed it in a dynamic driving sequence.

Omniverse allows researchers to bring exciting, cutting-edge research directly to creators and end users, said Jean-Francois Lafleche, deep learning engineer at NVIDIA. Offering GANverse3D as an extension in Omniverse will help artists create richer virtual worlds for game development, city planning or even training new machine learning models.

GANs Power a Dimensional Shift Because real-world datasets that capture the same object from different angles are rare, most AI tools that convert images from 2D to 3D are trained using synthetic 3D datasets like ShapeNet.

To obtain multi-view images from real-world data - like images of cars available publicly on the web - the NVIDIA researchers instead turned to a GAN model, manipulating its neural network layers to turn it into a data generator.

The team found that opening the first four layers of the neural network and freezing the remaining 12 caused the GAN to render images of the same object from different viewpoints.

Keeping the first four layers frozen and the other 12 layers variable caused the neural network to generate different images from the same viewpoint. By manually assigning standard viewpoints, with vehicles pictured at a specific elevation and camera distance, the researchers could rapidly generate a multi-view dataset from individual 2D images.

The final model, trained on 55,000 car images generated by the GAN, outperformed an inverse graphics network trained on the popular Pascal3D dataset.

Read the full ICLR paper, authored by Wenzheng Chen, fellow NVIDIA researchers Jun Gao and Huan Ling, Sanja Fidler, director of NVIDIA's Toronto research lab, University of Waterloo student Yuxuan Zhang, Stanford student Yinan Zhang and MIT professor Antonio Torralba. Additional collaborators on the CVPR paper include Jean-Francois Lafleche, NVIDIA researcher Kangxue Yin and Adela Barriuso.

The NVIDIA Research team consists of more than 200 scientists around the globe, focusing on areas such as AI, computer vision, self-driving cars, robotics and graphics. Learn more about the company's latest research and industry breakthroughs in NVIDIA CEO Jensen Huang's k
LINK: https://blogs.nvidia.com/blog/2021/04/16/gan-research-knight-rider-ai-...
See more stories from nvidia

North America Stories

04/09/2026

EVS, Matrox Partner on EBUs Dynamic Media Facility Initiative

Share Copy link Facebook X Linkedin Bluesky Email...

04/09/2026

TAG Video Systems Unveils TAG Blue

Share Copy link Facebook X Linkedin Bluesky Email...

04/09/2026

Orlando Magic and Gray Media Ink Full Season Broadcast Deal

Share Copy link Facebook X Linkedin Bluesky Email...

03/09/2026

Creative Technology Group Announces Leadership Transition Effective January 2027

NEP Group has announced leadership changes at Creative Technology Group. Founder and CEO Graham Andrews will step away from day-to-day leadership at the end of ...

03/09/2026

NFL and TikTok Renew Multi-Year Partnership

The NFL and TikTok have announced a multi-year partnership renewal expanding the league's presence on the platform. NFL posts on TikTok increased more than ...

03/09/2026

Grass Valley Appoints Ronny Van Geel as Vice President, Cameras

Grass Valley has appointed Ronny Van Geel as Vice President, Cameras. Van Geel began his career in 1999 at Philips DVS and moved into product management in 2005...

03/09/2026

IBC 2026: Telestream to Preview Vantage Super Resolution

Telestream has announced Vantage Super Resolution, an AI-powered upscaling capability for the Vantage media workflow platform, developed in collaboration with N...

03/09/2026

IBC 2026: Solid State Logic TCA Tour

Solid State Logic (SSL) will debut the TCA Tour at IBC2026 (Hall 8, Stand 8.B45), a portable production system built from System T components in a desktop frame...

03/09/2026

IBC 2026: DPA Microphones Launches 6388 CORE+ Headset Microphone

DPA Microphones has announced the 6388 CORE Headset Microphone, making its debut at IBC2026 (Hall 8, Stand D60). The supercardioid headset features a physicall...

03/09/2026

IBC 2026: Qvest to Demonstrate Multi-Vendor Dynamic Media Facility Showcase

Qvest will present a multi-vendor Dynamic Media Facility (DMF) showcase at IBC2026 (Booth 10.C24), demonstrating how the EBU's DMF architecture can be imple...

03/09/2026

IBC 2026: Imagine Communications Debuts Magellan PanelFlex

Imagine Communications will showcase the Magellan PanelFlex control surface and other control portfolio updates at IBC2026 (Stand 1.B73, RAI Amsterdam, Septembe...

03/09/2026

IBC 2026: Bridge Technologies Launches Software-Based VB440 Deployment Option

Bridge Technologies will announce at IBC2026 (Booth 1.A71) that its VB440 production probe is now available as a software-based solution, deployable on commerci...

03/09/2026

Audio-Technica Launches BP350ST Mid-Side Stereo Microphone in Two Variants

Audio-Technica has unveiled the BP350ST MS (mid-side) microphone in two versions: the camera-mount BP350ST-UL and the boundary or gooseneck-mountable BP350ST-UB...

03/09/2026

IBC 2026: Jnger Audio flexAI Adds CastCompanion, 12G-SDI, NDI, and TVU MediaMesh Integration

Telos Alliance will introduce updates to the J nger Audio flexAI audio processin...

03/09/2026

DUNKMAN to Expand in 2027 with Team Competitions, Junior Division, and Global Qualifiers

DUNKMAN, the professional dunk league founded by Shaquille O'Neal in partner...

03/09/2026

Orlando Magic Tap DAZN as DTC Streaming Partner, Will Join OTA Distribution Plans for 2026-27 Season

Orlando's WRDQ TV 27, WFTV Channel 9 previously named over-the-air broadcast...

03/09/2026

EverPass Media Expands NFL Agreement with Netflix for Commercial Venues

EverPass Media has announced a multi-year commercial distribution agreement with Netflix, expanding on an existing partnership that has delivered NFL Christmas ...

03/09/2026

Yahoo Sports and Apple TV Partner on MLB and MLS Alternate Broadcasts

Yahoo Sports and Apple TV have announced a partnership to deliver alternate broadcasts of select Friday Night Baseball and MLS games on the Yahoo Sports website...

03/09/2026

BIG EAST's Scott Hecht on the Evolution of Campus Production and What Comes Next

With another academic year underway, college athletic departments across the cou...

03/09/2026

ESPN Debuts Revamped, Cutting-Edge Studio W in Bristol as New Home to NFL Live, College Football, and More

Facility becomes permanent home of NFL Live on Sept. 9; centerpiece studio for c...

03/09/2026

NAB Backs Most Proposed EAS Changes at FCC

Share Copy link Facebook X Linkedin Bluesky Email...

03/09/2026

TiVo Expands FAST Channels, AI Discovery with Latest OS Update

Share Copy link Facebook X Linkedin Bluesky Email...

03/09/2026

Eluvio Advances Content Fabric, AI at IBC 2026

Share Copy link Facebook X Linkedin Bluesky Email...

03/09/2026

Haivision Adds Bonded 5G Cellular Support to Makito ONE

Share Copy link Facebook X Linkedin Bluesky Email...

03/09/2026

Alex Pannell Joins Gravity Media as Chief Business Officer

Share Copy link Facebook X Linkedin Bluesky Email...

03/09/2026

AI-Media To Feature Multilingual AI Developments At IBC2026

Share Copy link Facebook X Linkedin Bluesky Email...

03/09/2026

Tuxera and ATTO collaborate to deliver blistering macOS n...

Recognizing the ever-increasing demands for media throughput in challenging environments like post-production, two leaders in network technology have worked tog...

03/09/2026

EVS and Matrox Video Join Forces to Accelerate Dynamic Me...

EVS, a leader in live video technology and production infrastructure solutions, and Matrox Video, a leader in software-defined media technology and IP-based vid...

03/09/2026

AJA Debuts OG-FiDO-4 openGear Four-Channel SDI/Fiber Conversion Cards

AJA Debuts OG-FiDO-4 openGear Four-Channel SDI/Fiber Conversion Cards Brie Clayton September 3, 2026 0 Comments New product family expands 12G-SDI and...

03/09/2026

Nice Touch launches V2 of its AI edit assistant for professional editors

Nice Touch launches V2 of its AI edit assistant for professional editors Brie Clayton September 3, 2026 0 Comments Built by two editors with twelve ye...

03/09/2026

Accedo and MediaKind Launch End-to-End Streaming Solution...

Accedo, a global provider of video streaming software and services, and MediaKind, a world-leading video technology company and the number one independent provi...

03/09/2026

14 Percent of live sports viewers have considered switchi...

International research from Big Blue Marble puts a consumer churn number against a technical failure, and sets out what streaming providers must fix before 2030...

03/09/2026

Broadpeak expands CDNaaS for streaming at scale with new...

High-performance points of presence power video streaming delivery to large audiences across Latin America with guaranteed quality of experience Broadpeak , a ...

03/09/2026

Dalia Delivers New Agentic Workflows Changing How Media W...

Dalet, a leading technology and service provider for media-rich organizations, will unveil the next evolution of Dalia at IBC2026 (7.A43), its media-aware agent...

03/09/2026

CGI showcases AI-assisted newsroom and radio workflows pa...

CGI (TSX: GIB.A) (NYSE: GIB), one of the world's largest independent IT and business consulting services firms, will showcase CGI OpenMedia Evolve and CGI d...

03/09/2026

LiveU Joins IBC Accelerator Smart Stories Consortium to D...

LiveU is participating in the IBC Accelerator Incubator 2026: Smart Stories The New Knowledge Architecture for Content Production, an industry-wide initiative...

03/09/2026

Christer Palsson brings global experience to Broadcast So...

Broadcast Solutions, a leading systems integrator and provider of innovative solutions for the broadcast and media industry, has taken another major step in its...

03/09/2026

Quortex brings AI and IP transition into focus at IBC 202...

Quortex today announced that it will showcase how it is accelerating innovation and the transition to IP at IBC 2026. The company will debut its newest intellig...

03/09/2026

Grass Valley Names Ronny Van Geel Vice President Of Cameras

Share Copy link Facebook X Linkedin Bluesky Email...

03/09/2026

VisualOn and EZDRM Enable Visualization of C2PA Provenanc...

The EZDRM approach to Revenue Security on streaming video can be visualized as a multi-dimensional set of solutions. A growing dimension of that approach is the...

03/09/2026

Kiloview Brings Hybrid AV-over-IP to Life at IBC 2026 wit...

From field capture to management, control and production, Kiloview will demonstrate one connected Hybrid AV-over-IP workflow including a complete 4RU producti...

03/09/2026

Mediagenix Curation Unlocks Greater Catalog Value Through...

Mediagenix, a global leader in smart content solutions to profitably connect the right content to the right audience, today announced Mediagenix Curation, a new...

03/09/2026

LucidLink brings distributed media into active workflows...

LucidLink, the Emmy-winning cloud-native file streaming platform for instant, secure access to large files, today announced the next step in its vision to make ...

03/09/2026

AJA Introduces FS8, Next-Gen 12G-SDI/SMPTE ST 2110 Media Processor

AJA Introduces FS8, Next-Gen 12G-SDI/SMPTE ST 2110 Media Processor Brie Clayton September 2, 2026 0 Comments AJA Video Systems unveiled FS8, a video a...

03/09/2026

Shooting Deadheading with Blackmagic URSA Cine 12K LF

Shooting Deadheading with Blackmagic URSA Cine 12K LF Brie Clayton September 2, 2026 0 Comments Bringing a small story to the big screen with dark com...

03/09/2026

OOONA Launches OOONA Go: Professional Localization Management for Small Companies

OOONA Launches OOONA Go: Professional Localization Management for Small Companie...

03/09/2026

East of Texas Opens at American Tobacco Campus, Bringing North Carolina's First Tex-Mex BBQ Concept to Durham

DURHAM, N.C. - East of Texas Durham has officially opened its doors at American ...

03/09/2026

Sparks Fly: NVIDIA Accelerates Local AI at IFA 2026

Frontier intelligence is going local. At IFA 2026, NVIDIA, Microsoft and its partners are teaming up to provide faster inference and new tools that make agents ...

03/09/2026

Telos Alliance Emphasizes Audio Observability Across Media Workflows

Telos Alliance Emphasizes Audio Observability Across Media Workflows Search IBC 2026 demonstrations highlight automated monitoring for increasingly complex...

03/09/2026

NBA 2K27' With NVIDIA DLSS 5 Leads 26 New Games Coming to GeForce NOW

September is here with 28 more games streaming on GeForce NOW this month, led by a slam dunk: NBA 2K27 with the NVIDIA DLSS 5 3D-Guided Neural Rendering feature...