Sony Pixel Power calrec Sony

Knight Rider Rides a GAN: Bringing KITT to Life with AI, NVIDIA Omniverse

16/04/2021

Fasten your seatbelts. NVIDIA Research is revving up a new deep learning engine that creates 3D object models from standard 2D images - and can bring iconic cars like the Knight Rider's AI-powered KITT to life - in NVIDIA Omniverse.

Developed by the NVIDIA AI Research Lab in Toronto, the GANverse3D application inflates flat images into realistic 3D models that can be visualized and controlled in virtual environments. This capability could help architects, creators, game developers and designers easily add new objects to their mockups without needing expertise in 3D modeling, or a large budget to spend on renderings.

A single photo of a car, for example, could be turned into a 3D model that can drive around a virtual scene, complete with realistic headlights, tail lights and blinkers.

To generate a dataset for training, the researchers harnessed a generative adversarial network, or GAN, to synthesize images depicting the same object from multiple viewpoints - like a photographer who walks around a parked vehicle, taking shots from different angles. These multi-view images were plugged into a rendering framework for inverse graphics, the process of inferring 3D mesh models from 2D images.

Once trained on multi-view images, GANverse3D needs only a single 2D image to predict a 3D mesh model. This model can be used with a 3D neural renderer that gives developers control to customize objects and swap out backgrounds.

When imported as an extension in the NVIDIA Omniverse platform and run on NVIDIA RTX GPUs, GANverse3D can be used to recreate any 2D image into 3D - like the beloved crime-fighting car KITT, from the popular 1980s Knight Rider TV show.

Previous models for inverse graphics have relied on 3D shapes as training data.

Instead, with no aid from 3D assets, We turned a GAN model into a very efficient data generator so we can create 3D objects from any 2D image on the web, said Wenzheng Chen, research scientist at NVIDIA and lead author on the project.

Because we trained on real images instead of the typical pipeline, which relies on synthetic data, the AI model generalizes better to real-world applications, said NVIDIA researcher Jun Gao, an author on the project.

The research behind GANverse3D will be presented at two upcoming conferences: the International Conference on Learning Representations in May, and the Conference on Computer Vision and Pattern Recognition, in June.

From Flat Tire to Racing KITT Creators in gaming, architecture and design rely on virtual environments like the NVIDIA Omniverse simulation and collaboration platform to test out new ideas and visualize prototypes before creating their final products. With Omniverse Connectors, developers can use their preferred 3D applications in Omniverse to simulate complex virtual worlds with real-time ray tracing.

But not every creator has the time and resources to create 3D models of every object they sketch. The cost of capturing the number of multi-view images necessary to render a showroom's worth of cars, or a street's worth of buildings, can be prohibitive.

That's where a trained GANverse3D application can be used to convert standard images of a car, a building or even a horse into a 3D figure that can be customized and animated in Omniverse.

To recreate KITT, the researchers simply fed the trained model an image of the car, letting GANverse3D predict a corresponding 3D textured mesh, as well as different parts of the vehicle such as wheels and headlights. They then used NVIDIA Omniverse Kit and NVIDIA PhysX tools to convert the predicted texture into high-quality materials that give KITT a more realistic look and feel, and placed it in a dynamic driving sequence.

Omniverse allows researchers to bring exciting, cutting-edge research directly to creators and end users, said Jean-Francois Lafleche, deep learning engineer at NVIDIA. Offering GANverse3D as an extension in Omniverse will help artists create richer virtual worlds for game development, city planning or even training new machine learning models.

GANs Power a Dimensional Shift Because real-world datasets that capture the same object from different angles are rare, most AI tools that convert images from 2D to 3D are trained using synthetic 3D datasets like ShapeNet.

To obtain multi-view images from real-world data - like images of cars available publicly on the web - the NVIDIA researchers instead turned to a GAN model, manipulating its neural network layers to turn it into a data generator.

The team found that opening the first four layers of the neural network and freezing the remaining 12 caused the GAN to render images of the same object from different viewpoints.

Keeping the first four layers frozen and the other 12 layers variable caused the neural network to generate different images from the same viewpoint. By manually assigning standard viewpoints, with vehicles pictured at a specific elevation and camera distance, the researchers could rapidly generate a multi-view dataset from individual 2D images.

The final model, trained on 55,000 car images generated by the GAN, outperformed an inverse graphics network trained on the popular Pascal3D dataset.

Read the full ICLR paper, authored by Wenzheng Chen, fellow NVIDIA researchers Jun Gao and Huan Ling, Sanja Fidler, director of NVIDIA's Toronto research lab, University of Waterloo student Yuxuan Zhang, Stanford student Yinan Zhang and MIT professor Antonio Torralba. Additional collaborators on the CVPR paper include Jean-Francois Lafleche, NVIDIA researcher Kangxue Yin and Adela Barriuso.

The NVIDIA Research team consists of more than 200 scientists around the globe, focusing on areas such as AI, computer vision, self-driving cars, robotics and graphics. Learn more about the company's latest research and industry breakthroughs in NVIDIA CEO Jensen Huang's k
LINK: https://blogs.nvidia.com/blog/2021/04/16/gan-research-knight-rider-ai-...
See more stories from nvidia

North America Stories

23/09/2026

University of Washington's New Sundodger Studios Leverages ST 2110 to Bring Broadcast, Videoboard, and Creative Under One Roof

Four control rooms support B1G+ streams, videoboard shows, creative content, and...

23/09/2026

Mountain West Conference Moves Replay Operations West, Cutting Late-Night Hours

By Andrew Cohen, SVG Coorespondant The Mountain West debuted its new Replay Operations Center at the start of this football season, bringing real-time collabor...

23/09/2026

SVG Students to Watch: Brian Short, Ball State University

The Taylorville, IL, native is pushing creative boundaries at Ball State, gaining reps across directing, technical directing, tape, and more than 100 ESPN+ broa...

23/09/2026

The Musical and the Power of Spiteful Thinking

Alexander Heller, Nevada Jose, Will Brill, and Giselle Bonilla pose with cast members at The Musical premiere during the 2026 Sundance Film Festival at Eccles...

23/09/2026

McDonald's takes top spot as New Zealand's fast-food tastes evolve, according to Nielsen

Nielsen CMI data reveals a changing fast-food landscape, with value, convenience...

23/09/2026

AFP, Dalet To Collaborate On News Video Provenance, Authenticity

Share Copy link Facebook X Linkedin Bluesky Email...

23/09/2026

Spectrum News NECN Launches Across New England

Share Copy link Facebook X Linkedin Bluesky Email...

23/09/2026

COW Jobs: Blender on Windows - Generalist - Remote

COW Jobs: Blender on Windows - Generalist - Remote Brie Clayton September 23, 2026 0 Comments Application Users - Blender on Windows - Generalist Se...

23/09/2026

Bitfocus wins IBC2026 Best of Show Award for latest Butto...

Version 1.8 recognised for new workflow tools, boosted control panel flexibility and ease of integration into IP networks...

23/09/2026

Hiltron Reports a Successful IBC2026

Hiltron Communications reports a successful 2026 International Broadcasting Convention, held at the RAI Amsterdam. Exhibiting on stand A37 in Hall 1, Hiltron ex...

23/09/2026

LiveU Showcases AI Driven Production Alongside Resilient...

Reinforcing its long-term commitment to the Middle East and North Africa (MENA) broadcast market, LiveU is presenting the Q Era at CABSAT 2026, on Stand H1-D35 ...

23/09/2026

AFP and Dalet Partner to Strengthen the Provenance and Au...

Agence France-Presse (AFP), a global news agency, and Dalet, a leading technology and service provider for media-rich organizations, today announced a partnersh...

23/09/2026

Berklee Connect Unlocks the Power of a Global Network

Berklee Connect Unlocks the Power of a Global Network Katie Vitolins BM '04 knows firsthand what Berklee's alumni community can make possible. Now she...

23/09/2026

Study: Renewals for Streaming Shows Continue to Decline

Share Copy link Facebook X Linkedin Bluesky Email...

23/09/2026

Study: More Than 4 in 10 U.S. Adults Own or Are Interested in Smart Glasses

Share Copy link Facebook X Linkedin Bluesky Email...

23/09/2026

Television Critics Association Revises Bylaws to Address AI

Share Copy link Facebook X Linkedin Bluesky Email...

23/09/2026

eX Celebrates Completion of Holland Mich. FTTP Network

Share Copy link Facebook X Linkedin Bluesky Email...

23/09/2026

Gray Media Renews Ohio Valley Conference Deal

Share Copy link Facebook X Linkedin Bluesky Email...

23/09/2026

QuickLink StudioPro Powers Next Generation Student Media...

Westminster Schools, a K through 12 school in Atlanta, Georgia with approximately 2,000 students, has transformed its student broadcast and digital storytelling...

23/09/2026

UK Pavilion Brings Leading British Technology to CABSAT a...

The 2026 edition of CABSAT and SATExpo may have a new home and dates in the diary, but it still retains a strong presence from UK companies in the GREAT Britain...

23/09/2026

Glensound marks 60 years with new audio developments at C...

Glensound will return to CABSAT with new developments in professional monitoring and Dante interfacing, alongside its established commentary, esports and parlia...

23/09/2026

Sakeena Fiza Helps NVIDIA Hardware Succeed at Scale

When Sakeena Fiza describes her work as a validation engineer at NVIDIA, she does so in terms more befitting a detective story than a world-class engineering la...

22/09/2026

ESPN, Precision Lasers Light Up NHL Player Media Tour in Las Vegas

The annual preseason content-gathering event brought 36 NHL players to Las Vegas, where ESPN used 2.5 kilowatts of laser power and a motion-control camera syste...

22/09/2026

Neumann Introduces KK 104 A and KK 105 A Wireless Capsule Heads

Neumann has introduced the KK 104 A (cardioid) and KK 105 A (supercardioid) wireless capsule heads, succeeding the KK 204 and KK 205. The new capsules are compa...

22/09/2026

NFL and TMRW Sports Announce Pro Flag Showcase During Super Bowl LXI Week

The NFL and TMRW Sports have announced a new annual Pro Flag Showcase to be held during Super Bowl LXI week in Los Angeles. The event will feature top flag foot...

22/09/2026

Washington State University Unveils New Daktronics Video Board and Audio System at Gesa Field

Washington State University has unveiled a new end zone video display and audio ...

22/09/2026

SVG Sit-Down: Disney's Dave Johnson on ESPN's New 4K HDR Distribution Pipeline and What It Means for Super Bowl LXI and Beyond

Disney's SVP of Live Operations and Engineering details the technology behin...

22/09/2026

Cosm Promotes Mazen Alawar to SVP of Marketing, Alexis Scalice to VP of Cosm Studios

Cosm has announced two senior promotions. Mazen Alawar has been promoted to SVP,...

22/09/2026

Mississippi State University Extends Camera Relationship with KOKUSAI DENKI

Mississippi State University's University Television Center has purchased four KOKUSAI DENKI SK-UHD7000-S2 cameras since 2024, with plans to add two more, a...

22/09/2026

Net Insight Appoints Christine Kjellvard as Interim CFO

Net Insight has announced that Christine Kjellvard will assume the role of interim Chief Financial Officer on November 1, 2026. Kjellvard has previously held in...

22/09/2026

Net Insight Launches Zyntai TimeNode Access PTP Grandmaster for Edge Sites

Net Insight has announced Zyntai TimeNode Access, a compact PTP Grandmaster designed for distributed edge sites. The product carries secure, precise time across...

22/09/2026

The Weight Explores the High Cost of Freedom in the Depression Era

Sam Hazeldine, Austin Amelio, Padraic McKinley, Julia Jones, and Ethan Hawke attend the premiere of The Weight at The Ray Theatre on January 26, 2026, in Park...

22/09/2026

Apogee Plugins Are Moving to Plugin Alliance

Apogee Plugins Are Moving to Plugin Alliance We're excited to share an important update about the future of Apogee plugins. Apogee plugins are moving to Pl...

22/09/2026

SipRadius honored with a Technology And Engineering Emmy...

Parkland, Florida, US, September 22, 2026: SipRadius, widely recognized for making content processing and connectivity secure and seamless, has been honored wit...

22/09/2026

VanityFilter 3.1.5 adds project files and adjustable protection around lips, brows and ears

VanityFilter 3.1.5 adds project files and adjustable protection around lips, bro...

22/09/2026

Chile's TV+ Modernizes Broadcast Operations with Blackmagic Design

Chile's TV Modernizes Broadcast Operations with Blackmagic Design Brie Clayton September 22, 2026 0 Comments URSA Broadcast G2 brings versatility...

22/09/2026

OWC Shares Sneak Peek at Amazon Prime Big Deal Days Pricing

OWC Shares Sneak Peek at Amazon Prime Big Deal Days Pricing Brie Clayton September 22, 2026 0 Comments Image courtesy of Deposit Photos Business and ...

22/09/2026

Berklee's First Homecoming Block Party Brings Alumni Home for Global Music Festival

Berklee's First Homecoming Block Party Brings Alumni Home for Global Music F...

22/09/2026

New TVU Pocket Offers Broadcast Features In Phone-Sized Unit

Share Copy link Facebook X Linkedin Bluesky Email...

22/09/2026

Operative Launches Unified Platform for Advertising and Broadcast Management

Share Copy link Facebook X Linkedin Bluesky Email...

22/09/2026

Element Launches New TiVo-Powered Smart TVs in the U.S.

Share Copy link Facebook X Linkedin Bluesky Email...

22/09/2026

NOAA Taps Tomorrow.io for Improved U.S. Weather Forecasts

Share Copy link Facebook X Linkedin Bluesky Email...

22/09/2026

Gray Media Promotes Tim Maloney to GM and Director of Sales at WBKO

Share Copy link Facebook X Linkedin Bluesky Email...

22/09/2026

Reports: Paramount Reaches Settlement with State AGs in Antitrust Suit

Share Copy link Facebook X Linkedin Bluesky Email...

22/09/2026

New Research Lays Out Streaming Services Best Defense Against Churn

Share Copy link Facebook X Linkedin Bluesky Email...

22/09/2026

Sinclair Expands The National News Desk

Share Copy link Facebook X Linkedin Bluesky Email...

22/09/2026

Paramount Settles Antitrust Lawsuit over WBD Merger

Share Copy link Facebook X Linkedin Bluesky Email...

22/09/2026

U.S. Broadband Industry Investment Hits $92.6 Billion

Share Copy link Facebook X Linkedin Bluesky Email...

22/09/2026

At AI Day Singapore, NVIDIA and Partners Showcase AI Advancements Across Southeast Asia

NVIDIA AI Day Singapore, which takes place Sept. 22-23 at the Raffles City Conve...

22/09/2026

NVIDIA Isaac ROS 5.0 Advances Agentic, Open Source Robotics Development

To build and deploy sophisticated robotics applications that can perceive, reason and act in dynamic environments, developers need new physical AI models and to...