Sony Pixel Power calrec Sony

Knight Rider Rides a GAN: Bringing KITT to Life with AI, NVIDIA Omniverse


Fasten your seatbelts. NVIDIA Research is revving up a new deep learning engine that creates 3D object models from standard 2D images - and can bring iconic cars like the Knight Rider's AI-powered KITT to life - in NVIDIA Omniverse.

Developed by the NVIDIA AI Research Lab in Toronto, the GANverse3D application inflates flat images into realistic 3D models that can be visualized and controlled in virtual environments. This capability could help architects, creators, game developers and designers easily add new objects to their mockups without needing expertise in 3D modeling, or a large budget to spend on renderings.

A single photo of a car, for example, could be turned into a 3D model that can drive around a virtual scene, complete with realistic headlights, tail lights and blinkers.

To generate a dataset for training, the researchers harnessed a generative adversarial network, or GAN, to synthesize images depicting the same object from multiple viewpoints - like a photographer who walks around a parked vehicle, taking shots from different angles. These multi-view images were plugged into a rendering framework for inverse graphics, the process of inferring 3D mesh models from 2D images.

Once trained on multi-view images, GANverse3D needs only a single 2D image to predict a 3D mesh model. This model can be used with a 3D neural renderer that gives developers control to customize objects and swap out backgrounds.

When imported as an extension in the NVIDIA Omniverse platform and run on NVIDIA RTX GPUs, GANverse3D can be used to recreate any 2D image into 3D - like the beloved crime-fighting car KITT, from the popular 1980s Knight Rider TV show.

Previous models for inverse graphics have relied on 3D shapes as training data.

Instead, with no aid from 3D assets, We turned a GAN model into a very efficient data generator so we can create 3D objects from any 2D image on the web, said Wenzheng Chen, research scientist at NVIDIA and lead author on the project.

Because we trained on real images instead of the typical pipeline, which relies on synthetic data, the AI model generalizes better to real-world applications, said NVIDIA researcher Jun Gao, an author on the project.

The research behind GANverse3D will be presented at two upcoming conferences: the International Conference on Learning Representations in May, and the Conference on Computer Vision and Pattern Recognition, in June.

From Flat Tire to Racing KITT Creators in gaming, architecture and design rely on virtual environments like the NVIDIA Omniverse simulation and collaboration platform to test out new ideas and visualize prototypes before creating their final products. With Omniverse Connectors, developers can use their preferred 3D applications in Omniverse to simulate complex virtual worlds with real-time ray tracing.

But not every creator has the time and resources to create 3D models of every object they sketch. The cost of capturing the number of multi-view images necessary to render a showroom's worth of cars, or a street's worth of buildings, can be prohibitive.

That's where a trained GANverse3D application can be used to convert standard images of a car, a building or even a horse into a 3D figure that can be customized and animated in Omniverse.

To recreate KITT, the researchers simply fed the trained model an image of the car, letting GANverse3D predict a corresponding 3D textured mesh, as well as different parts of the vehicle such as wheels and headlights. They then used NVIDIA Omniverse Kit and NVIDIA PhysX tools to convert the predicted texture into high-quality materials that give KITT a more realistic look and feel, and placed it in a dynamic driving sequence.

Omniverse allows researchers to bring exciting, cutting-edge research directly to creators and end users, said Jean-Francois Lafleche, deep learning engineer at NVIDIA. Offering GANverse3D as an extension in Omniverse will help artists create richer virtual worlds for game development, city planning or even training new machine learning models.

GANs Power a Dimensional Shift Because real-world datasets that capture the same object from different angles are rare, most AI tools that convert images from 2D to 3D are trained using synthetic 3D datasets like ShapeNet.

To obtain multi-view images from real-world data - like images of cars available publicly on the web - the NVIDIA researchers instead turned to a GAN model, manipulating its neural network layers to turn it into a data generator.

The team found that opening the first four layers of the neural network and freezing the remaining 12 caused the GAN to render images of the same object from different viewpoints.

Keeping the first four layers frozen and the other 12 layers variable caused the neural network to generate different images from the same viewpoint. By manually assigning standard viewpoints, with vehicles pictured at a specific elevation and camera distance, the researchers could rapidly generate a multi-view dataset from individual 2D images.

The final model, trained on 55,000 car images generated by the GAN, outperformed an inverse graphics network trained on the popular Pascal3D dataset.

Read the full ICLR paper, authored by Wenzheng Chen, fellow NVIDIA researchers Jun Gao and Huan Ling, Sanja Fidler, director of NVIDIA's Toronto research lab, University of Waterloo student Yuxuan Zhang, Stanford student Yinan Zhang and MIT professor Antonio Torralba. Additional collaborators on the CVPR paper include Jean-Francois Lafleche, NVIDIA researcher Kangxue Yin and Adela Barriuso.

The NVIDIA Research team consists of more than 200 scientists around the globe, focusing on areas such as AI, computer vision, self-driving cars, robotics and graphics. Learn more about the company's latest research and industry breakthroughs in NVIDIA CEO Jensen Huang's k
See more stories from nvidia

Most recent headlines


DiDi Chooses NVIDIA DRIVE for New Fleet of Self-Driving Robotaxis

Robotaxis are one major step closer to becoming reality. DiDi Autonomous Driving, the self-driving technology arm of mobility technology leader Didi Chuxing, a...


'Dream Home Makeover' Stars Reveal How to Throw a Relaxing But Memorable Memorial Day Cookout

Back to All News Dream Home Makeover Stars Reveal How to Throw a Relaxing But M...


'Ginny & Georgia' Costume Designer Bernadette Croft Reveals the Stories Behind the Looks

Back to All News Ginny & Georgia Costume Designer Bernadette Croft Reveals the ...


Documentary Film Pray Away Heads To Netflix

Back to All News Documentary Film Pray Away Heads To Netflix Business 17 May 2021 GlobalUnited States The Film Will Premiere at the Tribeca Film Festiva...


How Netflix, Bisha K Ali and Sky Are Supporting Underrepresented Screenwriting Talent

Back to All News How Netflix, Bisha K Ali and Sky Are Supporting Underrepresent...


Celebrities and chefs confirmed for ITV's Cooking With The Stars

Celebrities and chefs confirmed for ITV's Cooking With The Stars Produced by South Shore and co-funded with Marks & Spencer Hosted by Emma Willis and Tom ...


Sir Keir Starmer joins Piers for Piers Morgan's Life Stories

Sir Keir Starmer joins Piers for Piers Morgan's Life Stories ITV announces that Piers Morgan will be joined in the studio by Sir Keir Starmer for a rare an...



5G's economic potential at risk as countries diverge on critical 6 GHz spectrum 17 May 2021, London: The global future of 5G is at risk if governments fail...


Interview with Zurab Alasania, Director General of UA:PBC from 2014 to 2021

Interview with Zurab Alasania, Director General of UA:PBC from 2014 to 2021...


CODA Audio Launches HOPS12i

CODA Audio has announced the expansion of its HOPS (high output point source) range with the addition of the new HOPS12i. An extremely high output, three-way po...


Cisco CEO Addresses World's Largest Cybersecurity Conference

News Summary: Cisco unveils innovations that radically simplify security across devices, networks, applications and data with expanded Extended Detection & Res...



Cupertino, California Since the introduction of Apple Watch Pride Edition in 2016, Apple's unique Pride bands have been a visible illustration of the ways i...


LYNX Technik Launches greenMachine Callisto+ Hardware with Dual-Channel Up/Down/Cross Converter Configuration

LYNX Technik Launches greenMachine Callisto Hardware with Dual-Channel Up/Down/...


SMPTE Hollywood to Discuss Post-Production in a Post-Covid World

SMPTE Hollywood to Discuss Post-Production in a Post-Covid World data-src=


How Canon's latest PTZ camera innovations are helping the industry face new challenges

Latest camera and remote production system are helping companies zoom in on the ...


Pay-TV revenues to drop by a quarter over 10 years

Analysts believe revenues peaked at $201 billion in 2016 By Jenny Priestley Published: May 17, 2021 Analysts believe revenues peaked at $201 billion in 2016...


Sony sells off UK TV channels

All seven free-to-air channels have been acquired by Narrative Capital By Jenny Priestley Published: May 17, 2021 All seven free-to-air channels have been a...


Discovery, AT&T merge to create new super streamer'

Discovery CEO David Zaslav will lead the new company By Jenny Priestley Published: May 17, 2021 Discovery CEO David Zaslav will lead the new company targe...


Summer NAMM Returns

Nashville show will take place 15-16th July The NAMM Shows are usually the highlights of the musical instrument and technology industrys calendar, but for o...


Akai unveil MPC One Retro

Sampler/sequencer gets new retro look Akai have released a new special edition of their all-in-one music production workstation, the MPC One. Dubbed the MPC...


Q1 2021 Results

Strong start to 2021. Launching 100 million Share Buyback programme Luxembourg, 6 May 2021 -- SES S.A. announces financial results for the three months ended...


Ghostframe- Game-Changing Technology For Virtual Production

Ghostframe - Game-Changing Technology For Virtual Production posted: 17/05/2021 Industry experts AGS, Megapixel VR, and ROE Visual collaborate to form Gho...


Screen Australia announces $5.6 million of production funding for 5 projects

17 05 2021 - Media release Screen Australia announces $5.6 million of production funding for 5 projects Adam Elliot Screen Australia has announced $5.6 milli...


North Carolina Courage Announces Expanded Ownership Group

New Members Committed to Galvanizing Their Unique Experiences to Build on Courage Success The North Carolina Courage is pleased to announce an additional group...


New SMI Metric Allows Buyers To Compare TV, Digital Video

At a time when more ad dollars are shifting from traditional TV to digital video, research company Standard Media Index has developed a new metric that enables ...


Another Nielsen Critic: iSpot Says Ad Ratings Rose During COVID

Jumping into the controversy over Nielsen undercounting TV usage during the pandemic, measurement company iSpot says its system shows ad ratings were up during ...


NBC Shares 2021-2022 Schedule

NBC has shared its 2021-2022 schedule, which sees a second season pickup of Law & Order: Organized Crime and the sixth and final season of This Is Us starting i...


Coop, Sweetie, Little Boo Take Flight When The Chicken Squad' Premieres May 14

The Chicken Squad, an animated series blending comedy and adventure, premieres o...


Critics Choice Awards' on The CW Jan. 9

The 27th annual Critics Choice Awards airs live on The CW Jan. 9. The three-hour event, celebrating the best in television and film, comes from the Critics Choi...


ABC Shares Renewals

ABC has renewed dramas A Million Little Things for season four and The Rookie for season four as well. ABC also renewed comedies Black-ish (season eight), The C...


Bill McCreary, Pioneering New York Anchor, Has Died

New York anchor Bill McCreary died last month in Brooklyn. He was 87 and suffered from a neurological disease....


Friends: The Reunion on HBO Max May 27

HBO Max will premiere the unscripted special Friends: The Reunion May 27, marking the one-year anniversary of the streaming network's launch. Stars Jennifer...


Primetime Ratings Thursday: Law & Order Duo Paces NBC

NBC had the top score in Thursday's prime, with the Law & Order double bill garnering a healthy audience. NBC earned a 0.6 in viewers 18-49, per the Nielsen...


The Wonder Years Gets Green Light on ABC

ABC has picked up The Wonder Years to series, with the show to run in 2021-2022. Inspired by the series of the same name that was on ABC from 1988 to 1993, The ...



BACKENTERTAINMENT ONES THE ROOKIE RENEWED BY ABC IN THE U.S. FOR SEASON 4Sold to more than 180 territories worldwide, the hit series is co-produced by Entertain...


Leading German Broadcast and Professional AV Merger

Leading German Broadcast and Professional AV Merger posted: 15/05/2021 Two leading German providers of the broadcast industry, Teltec AG and BPM Broadcast &...


Adobe: Video the Most Effective Marketing Tool Small Businesses Are Underutilizing

Adobe: Video the Most Effective Marketing Tool Small Businesses Are Underutilizi...


Golden Rooster Award Winner Chaogtu with Sarula graded by Luo Mengzhou with DaVinci Resolve Studio

Golden Rooster Award Winner Chaogtu with Sarula graded by Luo Mengzhou with DaVi...


Creative COW with Erik Messerschmidt: A Labor of Light

Creative COW with Erik Messerschmidt: A Labor of Light data-src=


Gold returns to visit The Cockfields

14 May 2021 Praise for series one: Daily Mail Weekend - One of the most well-observed comedies of 2019 The Times - Meet The Parents on the Isle of Wight Radi...


VUit Adds New Programming from Gray TV, Lilly Broadcasting and Heritage Broadcasting

NEW YORK-VUit, the free, ad-supported streaming service that focuses on local pr...


Cinematographer Powers Remote Collaborative Workflow With Dejero EnGo

WATERLOO, Ontario-Cinematographer Vinit Borrison powered what's dubbed a virtual video village that connected the geographically dispersed creative team i...


Verizon, Canon, RED Among Founding Members of Volumetric Format Association

LOS ANGELES-Seven companies have come together to form the Volumetric Format Association, dedicated to ensuring interoperability across the volumetric video eco...


Disney+ Subs Fall Short of Latest Expectations

BURBANK, Calif.-Is Disney+ starting to hit a wall? In one of the first instances since the streaming service launched in November 2019, Disney+ failed to meet m...


Gary Herman Named PESA's Director of Channel Partner Programs

HUNTSVILLE, Ala.-PESA Switching Systems has named Gary Herman the director of Channel Partner Management for the company. In his new role, Herman will lead PESA...


Evoca Partners With Unmudl on Adult Education TV Channel

AUSTIN, Texas & BOISE, Idaho-Evoca, a NextGen TV-based OTA-OTT hybrid pay-TV service, is partnering with Unmudl, billed as a skills-to-job marketplace, to devel...


AWS, NHL Debuting New Advanced Stats for Stanley Cup Playoff Broadcasts

SEATTLE-Fans will be able to dive deeper into critical moments of the upcoming NHL Stanley Cup Playoffs with new advanced statistics that are set to debut via t...


HPA ALL Announces League Honorees

BURBANK, Calif.-Hollywood Professional Association (HPA) has announced the honorees for its inaugural HPA League Honors, which recognize diverse members of the ...


Riedel Bolero Intercom Keeps Crew Connected on Amazon's The Expanse'

WUPPERTAL, Germany-To ensure that the crew of Amazon Prime Video's The Expanse has reliable multichannel communications, the show's director of photog...


ESPN Scores New Streaming Rights in MLB Broadcast Extension

BRISTOL, Conn.-ESPN and Major League Baseball are continuing their broadcast relationship with a new deal, which will include the new right to simulcast games o...