Sony Pixel Power calrec Sony

Knight Rider Rides a GAN: Bringing KITT to Life with AI, NVIDIA Omniverse

16/04/2021

Fasten your seatbelts. NVIDIA Research is revving up a new deep learning engine that creates 3D object models from standard 2D images - and can bring iconic cars like the Knight Rider's AI-powered KITT to life - in NVIDIA Omniverse.

Developed by the NVIDIA AI Research Lab in Toronto, the GANverse3D application inflates flat images into realistic 3D models that can be visualized and controlled in virtual environments. This capability could help architects, creators, game developers and designers easily add new objects to their mockups without needing expertise in 3D modeling, or a large budget to spend on renderings.

A single photo of a car, for example, could be turned into a 3D model that can drive around a virtual scene, complete with realistic headlights, tail lights and blinkers.

To generate a dataset for training, the researchers harnessed a generative adversarial network, or GAN, to synthesize images depicting the same object from multiple viewpoints - like a photographer who walks around a parked vehicle, taking shots from different angles. These multi-view images were plugged into a rendering framework for inverse graphics, the process of inferring 3D mesh models from 2D images.

Once trained on multi-view images, GANverse3D needs only a single 2D image to predict a 3D mesh model. This model can be used with a 3D neural renderer that gives developers control to customize objects and swap out backgrounds.

When imported as an extension in the NVIDIA Omniverse platform and run on NVIDIA RTX GPUs, GANverse3D can be used to recreate any 2D image into 3D - like the beloved crime-fighting car KITT, from the popular 1980s Knight Rider TV show.

Previous models for inverse graphics have relied on 3D shapes as training data.

Instead, with no aid from 3D assets, We turned a GAN model into a very efficient data generator so we can create 3D objects from any 2D image on the web, said Wenzheng Chen, research scientist at NVIDIA and lead author on the project.

Because we trained on real images instead of the typical pipeline, which relies on synthetic data, the AI model generalizes better to real-world applications, said NVIDIA researcher Jun Gao, an author on the project.

The research behind GANverse3D will be presented at two upcoming conferences: the International Conference on Learning Representations in May, and the Conference on Computer Vision and Pattern Recognition, in June.

From Flat Tire to Racing KITT Creators in gaming, architecture and design rely on virtual environments like the NVIDIA Omniverse simulation and collaboration platform to test out new ideas and visualize prototypes before creating their final products. With Omniverse Connectors, developers can use their preferred 3D applications in Omniverse to simulate complex virtual worlds with real-time ray tracing.

But not every creator has the time and resources to create 3D models of every object they sketch. The cost of capturing the number of multi-view images necessary to render a showroom's worth of cars, or a street's worth of buildings, can be prohibitive.

That's where a trained GANverse3D application can be used to convert standard images of a car, a building or even a horse into a 3D figure that can be customized and animated in Omniverse.

To recreate KITT, the researchers simply fed the trained model an image of the car, letting GANverse3D predict a corresponding 3D textured mesh, as well as different parts of the vehicle such as wheels and headlights. They then used NVIDIA Omniverse Kit and NVIDIA PhysX tools to convert the predicted texture into high-quality materials that give KITT a more realistic look and feel, and placed it in a dynamic driving sequence.

Omniverse allows researchers to bring exciting, cutting-edge research directly to creators and end users, said Jean-Francois Lafleche, deep learning engineer at NVIDIA. Offering GANverse3D as an extension in Omniverse will help artists create richer virtual worlds for game development, city planning or even training new machine learning models.

GANs Power a Dimensional Shift Because real-world datasets that capture the same object from different angles are rare, most AI tools that convert images from 2D to 3D are trained using synthetic 3D datasets like ShapeNet.

To obtain multi-view images from real-world data - like images of cars available publicly on the web - the NVIDIA researchers instead turned to a GAN model, manipulating its neural network layers to turn it into a data generator.

The team found that opening the first four layers of the neural network and freezing the remaining 12 caused the GAN to render images of the same object from different viewpoints.

Keeping the first four layers frozen and the other 12 layers variable caused the neural network to generate different images from the same viewpoint. By manually assigning standard viewpoints, with vehicles pictured at a specific elevation and camera distance, the researchers could rapidly generate a multi-view dataset from individual 2D images.

The final model, trained on 55,000 car images generated by the GAN, outperformed an inverse graphics network trained on the popular Pascal3D dataset.

Read the full ICLR paper, authored by Wenzheng Chen, fellow NVIDIA researchers Jun Gao and Huan Ling, Sanja Fidler, director of NVIDIA's Toronto research lab, University of Waterloo student Yuxuan Zhang, Stanford student Yinan Zhang and MIT professor Antonio Torralba. Additional collaborators on the CVPR paper include Jean-Francois Lafleche, NVIDIA researcher Kangxue Yin and Adela Barriuso.

The NVIDIA Research team consists of more than 200 scientists around the globe, focusing on areas such as AI, computer vision, self-driving cars, robotics and graphics. Learn more about the company's latest research and industry breakthroughs in NVIDIA CEO Jensen Huang's k
LINK: https://blogs.nvidia.com/blog/2021/04/16/gan-research-knight-rider-ai-...
See more stories from nvidia

North America Stories

12/08/2026

ACLU Requests ABC Station Probe Documents from FCC

Share Copy link Facebook X Linkedin Bluesky Email...

12/08/2026

Roger Keating To Retire from Hearst Television

Share Copy link Facebook X Linkedin Bluesky Email...

12/08/2026

Netflix Closes U.S. Upfronts, Nearly Doubles Ad Commitments

Share Copy link Facebook X Linkedin Bluesky Email...

12/08/2026

CBS Pittsburgh Debuts New AR/VR News Studio

Share Copy link Facebook X Linkedin Bluesky Email...

12/08/2026

Matthews Intros T-Go Extreme Rigging Solutions

Matthews Studio Equipment introduces T-Go Mounting Solutions extreme camera rigging systems, engineered for the most demanding environments in motion picture pr...

12/08/2026

Anton Bauers EDEN Becomes Only Mobile Power Station in it...

Anton/Bauer, a Videndum brand and the industry leader in mobile power solutions for broadcast and cinematic production, announces that the EDEN Mobile Power Sta...

12/08/2026

TAG Video Systems Cuts Hardware Costs for Customers with...

TAG Video Systems, the global leader in software-based IP monitoring and multiviewer solutions, today announced a memory usage optimization that lowers the tota...

12/08/2026

Unified Streaming Launches Trusted Media - C2PA in Dynami...

Unified Streaming announces Trusted Media, a new solution that enables broadcasters, streaming platforms, and premium content owners to attach verifiable authen...

12/08/2026

Tedial Powers the AI-Driven Media Supply Chain at IBC 202...

Combining intelligent automation, composable business services and user-centric experiences The media industry is entering a new phase where AI intelligence is...

12/08/2026

DPA Microphones Appoints Raul Contreras as West Coast Reg...

DPA Microphones welcomes Raul Contreras as its new West Coast Regional Sales Manager. In this role, Contreras will oversee sales and customer relationships thro...

12/08/2026

Weather is Not a Factor for Marshall Electronics New CV57...

Marshall Electronics' first all-weather, all-IP 4K POV camera, the CV574-WP, makes its European debut at IBC 2026 (Stand 10.C11). Built for the elements, th...

12/08/2026

A double off-switch helps cells respond to stress

LA JOLLA, CA-Scientists at Scripps Research have discovered that two seemingly unrelated cellular enzymes become locked together during times of stress, simulta...

11/08/2026

Apple To Bring Friday Night Baseball Live to Apple Immersive on Vision Pro Beginning Aug. 28

Weekly immersive broadcasts will feature 8K 3D video, a 180-degree field of view...

11/08/2026

Looking for a New Cameraperson? ProductionHUB Guide Lays Out Considerations, Pitfalls

By Briana Brady, Social Media Manager, Production Hub Sports content is having ...

11/08/2026

SVG Sit-Down: Backlight CEO Kathleen Barrett on Building Media Infrastructure for a Global, AI-Driven Sports Landscape

She addresses why media organizations need more than asset storage, how sports t...

11/08/2026

FloSports Dennis LeMire on Implementing Cloud-Based Workflows on Lower Tier Sports

Remote commentary has improved significantly on these broadcasts At a clip of ...

11/08/2026

Miami HEAT's Revamped IP-Based Control Room Coordinates Workflows With Meinberg Intelligent Modular Synchronization System

A pair of LANTIME IMS M2000 PTP Grandmasters deliver resilient timing infrastruc...

11/08/2026

LTN Expands Live Event Master Control Capabilities for Sports Organizations

LTN has announced expanded capabilities for its Live Event Master Control offering, providing sports broadcasters, leagues, rights holders, and content owners w...

11/08/2026

WTA Ventures Appoints IMG as Managed Production Services Partner

WTA Ventures has appointed IMG as its managed production services partner in a multi-year agreement commencing with the 2027 season. The partnership is the late...

11/08/2026

New York Jets and CBS New York Extend Multi-Year Partnership

The New York Jets and CBS New York have agreed to a multi-year extension of their partnership, keeping CBS New York as the official broadcast home of the team. ...

11/08/2026

Nielsen Acquires DoubleVerify to Create New Combined Media Intelligence Platform

Nielsen Holdings and DoubleVerify have announced a definitive agreement under which Nielsen will acquire DoubleVerify in an all-cash transaction with an enterpr...

11/08/2026

AES Elects David v.R. Bowles as President-Elect for 2027, Unveils Other Election Results

The Audio Engineering Society has announced the results of its 2026 Board of Dir...

11/08/2026

SVG Sit-Down Creator Sports Network's Barrick Prince on Building the Next Layer of the Live-Sports Rights Stack

The CSN co-founder details its network and affiliate' model, how creator wi...

11/08/2026

Gravity Media Providing Host Broadcast Operations for Esports World Cup 2026

Gravity Media is providing host broadcast operations for the Esports World Cup (EWC) 2026, supporting ESL FACEIT Group (EFG) in delivering the tournament to glo...

11/08/2026

TAG Video Systems Reduces Memory Requirements for Compressed Workflows

TAG Video Systems has released a memory usage optimization for its MCM software that reduces the cost per channel by up to two-thirds for compressed workflows. ...

11/08/2026

SMT Expands LED Production and Scoring Services at 2026 Cincinnati Open

SMT is expanding its technology partnership with the Cincinnati Open this year, adding new LED displays, production capabilities, and fan-facing experiences at ...

11/08/2026

Pac-12 to Stream Fall Olympic Sports Home Events Free on YouTube

The Pac-12 has announced that home Olympic sports broadcasts throughout the fall season will be available free on the conference's YouTube channel at youtub...

11/08/2026

Case Study: Real Life Network Taps LiveU for Broadcast-Grade Streaming

Spreading a message of faith requires both widespread accessibility and uncompromising visual quality. Real Life Network (RLN), a free, faith-based digital stre...

11/08/2026

Tennessee Titans' New Nissan Stadium Will Live Up to Its Music City Location

Taking the synergy of sports and entertainment a step further, the NFL venue in Nashville will house iterations of two iconic music clubs The marriage of Taylo...

11/08/2026

Dolby Acquires VVC Patents From Florida Atlantic University

Share Copy link Facebook X Linkedin Bluesky Email...

11/08/2026

Alyson Stewart Joins Zixi as Senior Director, Sales

Share Copy link Facebook X Linkedin Bluesky Email...

11/08/2026

FCC Issues Alert Over Scammers Impersonating Its Commissioners

Share Copy link Facebook X Linkedin Bluesky Email...

11/08/2026

Comcast, Disney Ink Deal to End NFL Network Blackout

Share Copy link Facebook X Linkedin Bluesky Email...

11/08/2026

farmerswife and Cirkus are Exhibiting at IBC 2026

farmerswife and Cirkus will return to IBC 2026 to showcase new tools designed to help broadcast, media and post-production teams manage workflows with greater s...

11/08/2026

LiveU Powers Cross-Venue Coverage of Glasgow Commonwealth...

LiveUs LU900Q intelligent production unit played a central role in the host broadcast operation for the Games in Glasgow (July 23 to August 2, 2026). The host b...

11/08/2026

Jigsaw24 Appoints James Morse as Client Director to Drive...

Jigsaw24 has appointed James Morse as Client Director, Media Sales, as it deepens its focus on sports, live events and integrated production. His appointment co...

11/08/2026

Net Insight enables one of Europes largest broadcast grad...

Net Insight today announced that DMC Production Norway has selected its Open Live Media solution to centralize the production of Blinkfestivalen 2026. Combining...

11/08/2026

Vivaro Supports Record World Cup Audiences with the Appea...

Modular HEVC and JPEG XS workflows enabled resilient, ultra-low-latency delivery for leading linear and digital media platforms across all 104 matches Appear A...

11/08/2026

Homecoming Artist Spotlight: Fabiola Mndez

Homecoming Artist Spotlight: Fabiola M ndez The first cuatrista to graduate from Berklee talks about the community that shaped her career ahead of her Homecom...

11/08/2026

FOR-A To Highlight MixBoard, Viztrik AiDi Enhancement At IBC 2026

Share Copy link Facebook X Linkedin Bluesky Email...

11/08/2026

AMWA and EBU to Host Demos at IBC 2026

Share Copy link Facebook X Linkedin Bluesky Email...

11/08/2026

Dalet Solution Suites Enable Media Teams to Move Like a S...

Dalet, a leading technology and service provider for media-rich organizations, today announced Dalet Solution Suites, a new portfolio of cloud-native solutions ...

11/08/2026

Mediagenix Helps Broadcasters Prepare for Brazil DTV Futu...

Mediagenix, a global leader in smart content solutions to profitably connect the right content to the right audience, will showcase how media companies can prep...

11/08/2026

NVIDIA AI Factory Compute Is Becoming an Investable Asset Class

We announced partnerships with Apollo, BlackRock, Blackstone, Brookfield, Goldman Sachs and KKR to establish independent financing platforms designed to mobiliz...

11/08/2026

Why Scaling AI Compute Performance Requires a New Power Architecture

Every new generation of accelerated computing demands more from the infrastructure underneath it - more compute performance, higher rack density and more effici...

11/08/2026

NVIDIA Nemotron 3.5 Lightning and NeMo Switchyard Deliver Faster, Smarter, More Efficient Agentic AI

As AI shifts from chatbots to autonomous agents, open models are serving market ...

11/08/2026

NVIDIA and Local AI Community Fuel Open Source Models and Intelligent Agents

The open source ecosystem is making it easier for AI enthusiasts and developers to build, customize and run increasingly capable agents locally. Throughout Au...

10/08/2026

Apple to Bring Friday Night Baseball' Live to Apple Immersive on Vision Pro Beginning Aug. 28

Weekly immersive broadcasts will feature 8K 3D video, a 180-degree field of view...

10/08/2026

Former NAB Tech Exec Tom Keller Has Died

Share Copy link Facebook X Linkedin Bluesky Email...

10/08/2026

Matthews Bows Monitor Stand III

Share Copy link Facebook X Linkedin Bluesky Email...