Sony Pixel Power calrec Sony

Knight Rider Rides a GAN: Bringing KITT to Life with AI, NVIDIA Omniverse

16/04/2021

Fasten your seatbelts. NVIDIA Research is revving up a new deep learning engine that creates 3D object models from standard 2D images - and can bring iconic cars like the Knight Rider's AI-powered KITT to life - in NVIDIA Omniverse.

Developed by the NVIDIA AI Research Lab in Toronto, the GANverse3D application inflates flat images into realistic 3D models that can be visualized and controlled in virtual environments. This capability could help architects, creators, game developers and designers easily add new objects to their mockups without needing expertise in 3D modeling, or a large budget to spend on renderings.

A single photo of a car, for example, could be turned into a 3D model that can drive around a virtual scene, complete with realistic headlights, tail lights and blinkers.

To generate a dataset for training, the researchers harnessed a generative adversarial network, or GAN, to synthesize images depicting the same object from multiple viewpoints - like a photographer who walks around a parked vehicle, taking shots from different angles. These multi-view images were plugged into a rendering framework for inverse graphics, the process of inferring 3D mesh models from 2D images.

Once trained on multi-view images, GANverse3D needs only a single 2D image to predict a 3D mesh model. This model can be used with a 3D neural renderer that gives developers control to customize objects and swap out backgrounds.

When imported as an extension in the NVIDIA Omniverse platform and run on NVIDIA RTX GPUs, GANverse3D can be used to recreate any 2D image into 3D - like the beloved crime-fighting car KITT, from the popular 1980s Knight Rider TV show.

Previous models for inverse graphics have relied on 3D shapes as training data.

Instead, with no aid from 3D assets, We turned a GAN model into a very efficient data generator so we can create 3D objects from any 2D image on the web, said Wenzheng Chen, research scientist at NVIDIA and lead author on the project.

Because we trained on real images instead of the typical pipeline, which relies on synthetic data, the AI model generalizes better to real-world applications, said NVIDIA researcher Jun Gao, an author on the project.

The research behind GANverse3D will be presented at two upcoming conferences: the International Conference on Learning Representations in May, and the Conference on Computer Vision and Pattern Recognition, in June.

From Flat Tire to Racing KITT Creators in gaming, architecture and design rely on virtual environments like the NVIDIA Omniverse simulation and collaboration platform to test out new ideas and visualize prototypes before creating their final products. With Omniverse Connectors, developers can use their preferred 3D applications in Omniverse to simulate complex virtual worlds with real-time ray tracing.

But not every creator has the time and resources to create 3D models of every object they sketch. The cost of capturing the number of multi-view images necessary to render a showroom's worth of cars, or a street's worth of buildings, can be prohibitive.

That's where a trained GANverse3D application can be used to convert standard images of a car, a building or even a horse into a 3D figure that can be customized and animated in Omniverse.

To recreate KITT, the researchers simply fed the trained model an image of the car, letting GANverse3D predict a corresponding 3D textured mesh, as well as different parts of the vehicle such as wheels and headlights. They then used NVIDIA Omniverse Kit and NVIDIA PhysX tools to convert the predicted texture into high-quality materials that give KITT a more realistic look and feel, and placed it in a dynamic driving sequence.

Omniverse allows researchers to bring exciting, cutting-edge research directly to creators and end users, said Jean-Francois Lafleche, deep learning engineer at NVIDIA. Offering GANverse3D as an extension in Omniverse will help artists create richer virtual worlds for game development, city planning or even training new machine learning models.

GANs Power a Dimensional Shift Because real-world datasets that capture the same object from different angles are rare, most AI tools that convert images from 2D to 3D are trained using synthetic 3D datasets like ShapeNet.

To obtain multi-view images from real-world data - like images of cars available publicly on the web - the NVIDIA researchers instead turned to a GAN model, manipulating its neural network layers to turn it into a data generator.

The team found that opening the first four layers of the neural network and freezing the remaining 12 caused the GAN to render images of the same object from different viewpoints.

Keeping the first four layers frozen and the other 12 layers variable caused the neural network to generate different images from the same viewpoint. By manually assigning standard viewpoints, with vehicles pictured at a specific elevation and camera distance, the researchers could rapidly generate a multi-view dataset from individual 2D images.

The final model, trained on 55,000 car images generated by the GAN, outperformed an inverse graphics network trained on the popular Pascal3D dataset.

Read the full ICLR paper, authored by Wenzheng Chen, fellow NVIDIA researchers Jun Gao and Huan Ling, Sanja Fidler, director of NVIDIA's Toronto research lab, University of Waterloo student Yuxuan Zhang, Stanford student Yinan Zhang and MIT professor Antonio Torralba. Additional collaborators on the CVPR paper include Jean-Francois Lafleche, NVIDIA researcher Kangxue Yin and Adela Barriuso.

The NVIDIA Research team consists of more than 200 scientists around the globe, focusing on areas such as AI, computer vision, self-driving cars, robotics and graphics. Learn more about the company's latest research and industry breakthroughs in NVIDIA CEO Jensen Huang's k
LINK: https://blogs.nvidia.com/blog/2021/04/16/gan-research-knight-rider-ai-...
See more stories from nvidia

Most recent headlines

03/06/2024

Dalet and Veritone Reach Agreement to Distribute, Transact and Monetize Media Archives

Dalet, a leading technology and service provider for media-rich organizations, a...

28/04/2024

Mediahaus delivers the first SRT live-streaming sports production over 5G with URSA Broadcast G2

Mediahaus delivers the first SRT live-streaming sports production over 5G with U...

29/03/2024

Dalet to Demo AI-Enabled Production, Automation and Monetization at NAB Show

PARIS Dalet has announced that during the 2024 NAB Show (booth #W1713), attendees can experience firsthand Dalet's collaborative, AI-enabled workflows that ...

29/03/2024

DirecTV Inks Deal to Carry 2024 Colorado Rockies Games

DENVER AND EL SEGUNDO, Calif. DirecTV has announced a new agreement with Major League Baseball that will allow Colorado Rockies fans to see this season's ga...

29/03/2024

S&P: Media, Telecom M&A Plunges to 13-Month Low

S&P Global Market Intelligence has released new data showing that dealmaking in the North America for media and telecom companies remains in the doldrums....

29/03/2024

NAB Show: Pebble Launches PRIMA Software Platform

EPSOM, Surrey, UK The automation, content management, and integrated channel specialist Pebble, has announced that it will be launching a new software platform ...

29/03/2024

Gray Television to Air Arena Football League Local Games

ATLANTA Gray Television Inc. has inked a multi-year agreement with the Arena Football League to bring Saturday arena football games over-the-air to local audien...

29/03/2024

Muvhango: Thursday's episode, 28 March 2024 [VIDEO]

Muvhango: Thursday's episode, 28 March 2024 [VIDEO]Stay up to date with the latest episodes of your favourite South African soapie on The South African webs...

29/03/2024

Gqeberha: The Empire - Thursday's episode, 28 March 2024 [VIDEO]

Gqeberha: The Empire - Thursday's episode, 28 March 2024 [VIDEO]Stay up to date with the latest episodes of your favourite South African soapie on The South...

29/03/2024

My Brother's Keeper: Thursday's episode, 28 March 2024 [VIDEO]

My Brother's Keeper: Thursday's episode, 28 March 2024 [VIDEO]Stay up to date with the latest episodes of your favourite South African soapie on The Sou...

29/03/2024

Umkhokha: The Curse - Thursday's episode, 28 March 2024 [VIDEO]

Umkhokha: The Curse - Thursday's episode, 28 March 2024 [VIDEO]Stay up to date with the latest episodes of your favourite South African soapie on The South ...

29/03/2024

Skeem Saam: Thursday's episode, 28 March 2024 [VIDEO]

Skeem Saam: Thursday's episode, 28 March 2024 [VIDEO]Missed an episode of Skeem Saam? No problem! Watch the latest episode of your favourite South African s...

29/03/2024

House of Zwide: Thursday's episode, 28 March 2024 [VIDEO]

House of Zwide: Thursday's episode, 28 March 2024 [VIDEO]Stay up to date with the latest episode of South African soapie House of Zwide on The South African...

28/03/2024

No votes for all

Out of an estimated 20,000 journalists in Pakistan, less than one thousand are women according to a recent Gender Disparity Report published by the Internationa...

28/03/2024

Give Me the Backstory: Get to Know 8 Women Filmmakers With Shorts at the 2024 Sundance Film Festival

A still from Jasmin Baumgartner's Bye Bye, Bowser, which screened as part ...

28/03/2024

The Best Shared Storage For Video Production

alt= class=wp-image-12214 data-lazy-src=/wp-content/uploads/2024/03/Best-Shared-Storage-EVO-logo-960x540-1.jpg/> Embarking on a cinematic journey or crafting a ...

28/03/2024

Future-Proofing the B-52

L3Harris is upgrading critical electronic warfare systems to ensure the iconic aircrafts spectrum superiority for decades to come....

28/03/2024

Supporting the RCAF for more than 25 Years with L3Harris' WESCAM MX-Series

L3Harris' WESCAM MX-Series EO/IR systems have supported the RCAF for more than 25 years....

28/03/2024

ENCO to Add AI News Anchor to Qimera Virtual Production Set

NOVI, Mich. During the 2024 NAB Show, ENCO will be showcasing an AI-driven virtual anchor delivering live news from a realistic 3D virtual news set....

28/03/2024

Johan de Nysschen Named President and CEO of ARRI Americas

MUNICH ARRI has announced that Johan de Nysschen has been put in charge of the ARRI Americas team as president and CEO ARRI Americas Inc....

28/03/2024

Fubo Inks Carriage Deal with YES Network

NEW YORK The YES Network and FuboTV have reached a multi-year carriage agreement that will allow Fubo to launch YES in its Pro base channel plan in time for YES...

28/03/2024

Dalet to Demo AI-Enabled Production, Automation and Monetization at NAB 2024

PARIS Dalet has announced that during its 2024 NAB showcase (booth #W1713), attendees can experience firsthand Dalet's collaborative, AI-enabled workflows t...

28/03/2024

Blackmagic Design Used on 2024 Oscar Nominated Films

Blackmagic Design Used on 2024 Oscar Nominated Films Brie Clayton March 28, 2024 0 Comments Blackmagic Design congratulates the many film professional...

28/03/2024

Arkona to Introduce BLADE//master Control and Orchestration Service at NAB 2024

Arkona to Introduce BLADE//master Control and Orchestration Service at NAB 2024 Brie Clayton March 28, 2024 0 Comments New option for BLADE//runner ke...

28/03/2024

Join Maxon for NAB 2024

Join Maxon for NAB 2024 Brie Clayton March 28, 2024 0 Comments Maxon returns to NAB with presentations by artists from around the world and demos of t...

28/03/2024

Boston Conservatory to Present Center Stage Production of A Midsummer Night's Dream at The Huntington

Boston Conservatory to Present Center Stage Production of A Midsummer Night'...

28/03/2024

The 4K cameras giving audiences an astronaut's view of Earth

TVBEurope meets Charles Black, founder and CEO of Sen, which has sent a payload of 4K cameras to the ISS to livestream images of Earth to audiences around the w...

28/03/2024

2024 Big Night In Raises $145k to Support Triangle Arts Councils

Capitol Broadcasting and WRAL-TV once again proudly supported our local arts community with the broadcast of Big Night in for the Arts Thursday, March 7, 2024. ...

28/03/2024

JIC, MRC Announce Accord On Roles In Judging Measurement Companies

With more companies offering different approaches to measuring television viewing and campaign effectiveness, two of the groups that evaluate potential media bu...

28/03/2024

Hulu Added to Disney Plus App for Bundle Subscribers

The Walt Disney Co. said that it launched Hulu on Disney Plus, which makes Hulu programming available via the Disney Plus app....

28/03/2024

GroupM To Double Ad Spending On Women's Sports Programming

Giant media buyer GroupM said it plans to double its media spending on women's sports....

28/03/2024

Scripps Renews State Farm As Sponsor of Friday Night WNBA on Ion

Scripps Sports said that State Farm has signed a new multi-year agreement to continue as the title sponsor of Ion's Friday night WNBA telecasts....

28/03/2024

Elisabeth Moss, Kerry Washington Sign On for Imperfect Women' on Apple TV Plus

Elisabeth Moss and Kerry Washington will star in, and executive produce, Imperfe...

28/03/2024

Telemundo Station Group Addresses Climate Change With Month-Long Initiative

Telemundo Station Group launches the month-long climate change initiative Our Planet Thursday, March 28. The multiplatform campaign ends on Earth Day, April 22,...

28/03/2024

Circle City Broadcasting Renews Nielsen Ratings Deal

Nielsen said that Circle City Broadcasting, owner of WISH-TV in Indianapolis, has renewed its local TV measurement deal....

28/03/2024

Lionsgate Extends Deal With Seth Rogen's Point Grey Pictures

Lionsgate Television said it extended its deal with Point Grey Pictures, the production company led by Seth Rogen, Evan Goldberg and James Weaver....

28/03/2024

Ariana DeBose Back as Tonys Host

Ariana DeBose returns as host of the Tony Awards Sunday, June 16, on CBS and Paramount Plus. It is her third consecutive time hosting, and it will be the 77th T...

28/03/2024

The Steve Wilkos Show' Renewed for Season 18

The Steve Wilkos Show has been renewed for its 18th season in national syndication, Tracie Wilson, executive VP, NBCUniversal Syndication Studios & E! News, sai...

28/03/2024

Sinclair Selects Embrace as Its Media Orchestration Engine

PARIS Embrace, a developer of advanced automation and orchestration software solutions, has announced that Sinclair Broadcast Group has selected Embrace's P...

28/03/2024

Nielsen and Circle City Broadcasting Renew Local TV Measurement Deal

NEW YORK Nielsen has a multi-year renewal with Circle City Broadcasting, Indianas only locally owned and operated broadcast and digital news media network, for ...

28/03/2024

JVC to Demo Its First Ever 40x Zoom PTZ Camera Series at 2024 NAB Show

LAS VEGAS JVC Professional Video has announced that it will debut its new KY-PZ540 Series PTZ Cameras with 40x zoom at 2024 NAB Show (Booth C4720)....

28/03/2024

Brightcove Launches Cloud Playout 2.0

BOSTON Brightcove has announced the launch of Cloud Playout 2.0, a platform that the company said will offer customers new ways to create linear channels, monet...

28/03/2024

NBC Sports Chicago to Air First-Ever Animated Real-Time Local Sports Telecast

The Chicago Blackhawks and NBC Sports Chicago have announced a collaboration with the NHL to bring fans the first-ever local animated real-time sports telecast....

28/03/2024

The Best News Coverage is Real, Live, Balanced, Professional

The Best News Coverage is Real, Live, Balanced, Professional Andy Marken March 28, 2024 0 Comments Hero image source News of the World, Perfect Wo...

28/03/2024

DJ Sbu makes shock return to SABC radio after almost a decade

DJ Sbu makes shock return to SABC radio after almost a decadeRevered radio host, media personality, philanthropist, and businessman DJ Sbu will finally be back ...

28/03/2024

Smoke and Mirrors: Thursday's episode, 28 March 2024 [VIDEO]

Smoke and Mirrors: Thursday's episode, 28 March 2024 [VIDEO]Stay up to date with the latest episodes of your favourite South African soapie on The South Afr...

28/03/2024

VEON Pays Back Outstanding Balance and Cancels its Revolving Credit Facility

28 Mar 2024 VEON Pays Back Outstanding Balance and Cancels its Revolving Credit Facility Amsterdam, 28 March 2024, 09.30 CET - VEON Ltd. (Nasdaq: VEON, Euronex...

28/03/2024

TNT Sports Gets Its College Game On for March Madness Audio

TNT Sports Gets Its College Game On for March Madness Audio Marching bands, crowds make up for lack of court FX and wired players By Dan Daley, Audio Editor ...

28/03/2024

Cosm/ESPN Deal Will Bring NBA, NHL, Tennis, College Football, and More to Shared Reality' Venues

Cosm/ESPN Deal Will Bring NBA, NHL, Tennis, College Football, and More to Share...

28/03/2024

MLB 2024: The Famous Group, Miami Marlins Team Up for AI-Powered Intro for All 2024 Home Games

MLB 2024: The Famous Group, Miami Marlins Team Up for AI-Powered Intro for All 2...