Sony Pixel Power calrec Sony

Knight Rider Rides a GAN: Bringing KITT to Life with AI, NVIDIA Omniverse

16/04/2021

Fasten your seatbelts. NVIDIA Research is revving up a new deep learning engine that creates 3D object models from standard 2D images - and can bring iconic cars like the Knight Rider's AI-powered KITT to life - in NVIDIA Omniverse.

Developed by the NVIDIA AI Research Lab in Toronto, the GANverse3D application inflates flat images into realistic 3D models that can be visualized and controlled in virtual environments. This capability could help architects, creators, game developers and designers easily add new objects to their mockups without needing expertise in 3D modeling, or a large budget to spend on renderings.

A single photo of a car, for example, could be turned into a 3D model that can drive around a virtual scene, complete with realistic headlights, tail lights and blinkers.

To generate a dataset for training, the researchers harnessed a generative adversarial network, or GAN, to synthesize images depicting the same object from multiple viewpoints - like a photographer who walks around a parked vehicle, taking shots from different angles. These multi-view images were plugged into a rendering framework for inverse graphics, the process of inferring 3D mesh models from 2D images.

Once trained on multi-view images, GANverse3D needs only a single 2D image to predict a 3D mesh model. This model can be used with a 3D neural renderer that gives developers control to customize objects and swap out backgrounds.

When imported as an extension in the NVIDIA Omniverse platform and run on NVIDIA RTX GPUs, GANverse3D can be used to recreate any 2D image into 3D - like the beloved crime-fighting car KITT, from the popular 1980s Knight Rider TV show.

Previous models for inverse graphics have relied on 3D shapes as training data.

Instead, with no aid from 3D assets, We turned a GAN model into a very efficient data generator so we can create 3D objects from any 2D image on the web, said Wenzheng Chen, research scientist at NVIDIA and lead author on the project.

Because we trained on real images instead of the typical pipeline, which relies on synthetic data, the AI model generalizes better to real-world applications, said NVIDIA researcher Jun Gao, an author on the project.

The research behind GANverse3D will be presented at two upcoming conferences: the International Conference on Learning Representations in May, and the Conference on Computer Vision and Pattern Recognition, in June.

From Flat Tire to Racing KITT Creators in gaming, architecture and design rely on virtual environments like the NVIDIA Omniverse simulation and collaboration platform to test out new ideas and visualize prototypes before creating their final products. With Omniverse Connectors, developers can use their preferred 3D applications in Omniverse to simulate complex virtual worlds with real-time ray tracing.

But not every creator has the time and resources to create 3D models of every object they sketch. The cost of capturing the number of multi-view images necessary to render a showroom's worth of cars, or a street's worth of buildings, can be prohibitive.

That's where a trained GANverse3D application can be used to convert standard images of a car, a building or even a horse into a 3D figure that can be customized and animated in Omniverse.

To recreate KITT, the researchers simply fed the trained model an image of the car, letting GANverse3D predict a corresponding 3D textured mesh, as well as different parts of the vehicle such as wheels and headlights. They then used NVIDIA Omniverse Kit and NVIDIA PhysX tools to convert the predicted texture into high-quality materials that give KITT a more realistic look and feel, and placed it in a dynamic driving sequence.

Omniverse allows researchers to bring exciting, cutting-edge research directly to creators and end users, said Jean-Francois Lafleche, deep learning engineer at NVIDIA. Offering GANverse3D as an extension in Omniverse will help artists create richer virtual worlds for game development, city planning or even training new machine learning models.

GANs Power a Dimensional Shift Because real-world datasets that capture the same object from different angles are rare, most AI tools that convert images from 2D to 3D are trained using synthetic 3D datasets like ShapeNet.

To obtain multi-view images from real-world data - like images of cars available publicly on the web - the NVIDIA researchers instead turned to a GAN model, manipulating its neural network layers to turn it into a data generator.

The team found that opening the first four layers of the neural network and freezing the remaining 12 caused the GAN to render images of the same object from different viewpoints.

Keeping the first four layers frozen and the other 12 layers variable caused the neural network to generate different images from the same viewpoint. By manually assigning standard viewpoints, with vehicles pictured at a specific elevation and camera distance, the researchers could rapidly generate a multi-view dataset from individual 2D images.

The final model, trained on 55,000 car images generated by the GAN, outperformed an inverse graphics network trained on the popular Pascal3D dataset.

Read the full ICLR paper, authored by Wenzheng Chen, fellow NVIDIA researchers Jun Gao and Huan Ling, Sanja Fidler, director of NVIDIA's Toronto research lab, University of Waterloo student Yuxuan Zhang, Stanford student Yinan Zhang and MIT professor Antonio Torralba. Additional collaborators on the CVPR paper include Jean-Francois Lafleche, NVIDIA researcher Kangxue Yin and Adela Barriuso.

The NVIDIA Research team consists of more than 200 scientists around the globe, focusing on areas such as AI, computer vision, self-driving cars, robotics and graphics. Learn more about the company's latest research and industry breakthroughs in NVIDIA CEO Jensen Huang's k
LINK: https://blogs.nvidia.com/blog/2021/04/16/gan-research-knight-rider-ai-...
See more stories from nvidia

North America Stories

15/05/2021

Adobe: Video the Most Effective Marketing Tool Small Businesses Are Underutilizing

Adobe: Video the Most Effective Marketing Tool Small Businesses Are Underutilizi...

15/05/2021

Golden Rooster Award Winner Chaogtu with Sarula graded by Luo Mengzhou with DaVinci Resolve Studio

Golden Rooster Award Winner Chaogtu with Sarula graded by Luo Mengzhou with DaVi...

15/05/2021

Creative COW with Erik Messerschmidt: A Labor of Light

Creative COW with Erik Messerschmidt: A Labor of Light data-src=https://creativecow.net/wp-content/uploads/avatars/263018/5fa08c562aaf4-bpthumb.jpg......

14/05/2021

VUit Adds New Programming from Gray TV, Lilly Broadcasting and Heritage Broadcasting

NEW YORK-VUit, the free, ad-supported streaming service that focuses on local pr...

14/05/2021

Cinematographer Powers Remote Collaborative Workflow With Dejero EnGo

WATERLOO, Ontario-Cinematographer Vinit Borrison powered what's dubbed a virtual video village that connected the geographically dispersed creative team i...

14/05/2021

Verizon, Canon, RED Among Founding Members of Volumetric Format Association

LOS ANGELES-Seven companies have come together to form the Volumetric Format Association, dedicated to ensuring interoperability across the volumetric video eco...

14/05/2021

Disney+ Subs Fall Short of Latest Expectations

BURBANK, Calif.-Is Disney+ starting to hit a wall? In one of the first instances since the streaming service launched in November 2019, Disney+ failed to meet m...

14/05/2021

Gary Herman Named PESA's Director of Channel Partner Programs

HUNTSVILLE, Ala.-PESA Switching Systems has named Gary Herman the director of Channel Partner Management for the company. In his new role, Herman will lead PESA...

14/05/2021

Evoca Partners With Unmudl on Adult Education TV Channel

AUSTIN, Texas & BOISE, Idaho-Evoca, a NextGen TV-based OTA-OTT hybrid pay-TV service, is partnering with Unmudl, billed as a skills-to-job marketplace, to devel...

14/05/2021

AWS, NHL Debuting New Advanced Stats for Stanley Cup Playoff Broadcasts

SEATTLE-Fans will be able to dive deeper into critical moments of the upcoming NHL Stanley Cup Playoffs with new advanced statistics that are set to debut via t...

14/05/2021

HPA ALL Announces League Honorees

BURBANK, Calif.-Hollywood Professional Association (HPA) has announced the honorees for its inaugural HPA League Honors, which recognize diverse members of the ...

14/05/2021

Riedel Bolero Intercom Keeps Crew Connected on Amazon's The Expanse'

WUPPERTAL, Germany-To ensure that the crew of Amazon Prime Video's The Expanse has reliable multichannel communications, the show's director of photog...

14/05/2021

ESPN Scores New Streaming Rights in MLB Broadcast Extension

BRISTOL, Conn.-ESPN and Major League Baseball are continuing their broadcast relationship with a new deal, which will include the new right to simulcast games o...

14/05/2021

Morgan Murphy Media Held Vaccine Day' Across Stations

MADISON, Wis.-As the U.S. attempts to get enough of its citizens to get Covid-19 vaccines to reach herd immunity, a point of issue are those who remain hesitant...

14/05/2021

Behind the Mic: Kenny Albert Gets Nod as Lead Broadcaster for NHL on TNT; Sports Legend Chris Berman Signs Multi-year Deal With ESPN

Behind the Mic: Kenny Albert Gets Nod as Lead Broadcaster for NHL on TNT; Sports...

14/05/2021

SVG SportsTech On Demand: Panasonic's Jim Jensen Spotlights the Massive Growth of PTZ and Remote Cameras

SVG SportsTech On Demand: Panasonic's Jim Jensen Spotlights the Massive Grow...

14/05/2021

2021 SVG College Sports Media Awards: Today is the Deadline to Submit Your Entries!

2021 SVG College Sports Media Awards: Today is the Deadline to Submit Your Entri...

14/05/2021

In Final NHL Stanley Cup Playoffs Outing, NBC Sports' Ops Team Aims To Go Out on a High Note

In Final NHL Stanley Cup Playoffs Outing, NBC Sports' Ops Team Aims To Go Ou...

14/05/2021

ESPN's Sara Gaiero: Broadcasting the WNBA In Its 25th Season Is A Great Responsibility

ESPN's Sara Gaiero: Broadcasting the WNBA In Its 25th Season Is A Great Res...

14/05/2021

146th Preakness Stakes: NBC's Pimlico Setup Builds on Onsite Derby Production

146th Preakness Stakes: NBC's Pimlico Setup Builds on Onsite Derby Productio...

14/05/2021

2021 Sundance Episodic Fellows

Los Angeles - Sundance Institute announced today the 21 Fellows across 18 projects completing its first-ever multi-track Episodic Lab, designed as an immersive ...

14/05/2021

Starship SN15

Back to updates MAY 05, 2021 Starship SN15 On Wednesday, May 5, Starship serial number 15 (SN15) successfully completed SpaceX's fifth high-altitude f...

14/05/2021

AI Slam Dunk: Startup's Checkout-Free Stores Provide Stadiums Fast Refreshments

With live sports making a comeback, one thing remains a constant: Nobody likes t...

14/05/2021

New Cg-animated Feature Film Following The Iconic Japanese Superhero Ultraman In Development At Netflix

Back to All News New Cg-animated Feature Film Following The Iconic Japanese Sup...

14/05/2021

NETFLIX ANNOUNCES THE START OF PRODUCTION FOR 365 DAYS PART 2 & PART 3 FILMS

Back to All News NETFLIX ANNOUNCES THE START OF PRODUCTION FOR 365 DAYS PART 2 & PART 3 FILMS Entertainment 13 May 2021 GlobalPoland Netflix announces the ...

14/05/2021

Catching Up With Kathleen Turner of 'The Kominsky Method'

Back to All News Catching Up With Kathleen Turner of The Kominsky Method Netflix Staff Entertainment 14 May 2021 Global Oscar and Tony-nominated actress K...

14/05/2021

Netflix and Shondaland Announce Exciting 'Bridgerton' Spinoff & Welcome Jess Brownell as Showrunner for Seasons 3 & 4

Back to All News Netflix and Shondaland Announce Exciting Bridgerton Spinoff & ...

14/05/2021

Advancing gender diversity in traditionally male roles

News | May 12 2021 Advancing gender diversity in traditionally male roles Offshore Brazil, a women-only team racked up a first for Subsea Services during o...

14/05/2021

Cobalt Adds Newly Designed One and Two Channel 3G-SDI Models to Portfolio of Reliable Distribution Amplifiers

Cobalt Adds Newly Designed One and Two Channel 3G-SDI Models to Portfolio of Rel...

14/05/2021

Cisco Announces Intent to Acquire Kenna Security

News Summary: Intent to acquire Kenna Security a market leader in risk-based vulnerability management, to advance Cisco's commitment to radically simplify ...

14/05/2021

The Flanger Lab

The Flanger Lab FX Chain provides a wide variety of effects, from traditional flanging to psycho-acoustic panning, and can even incorporate some mid-side mojo-i...

14/05/2021

Rattling Stick Signs Director Vanessa Beletic

Rattling Stick Signs Director Vanessa Beletic data-src=https://creativecow.net/wp-content/uploads/avatars/263018/5fa08c562aaf4-bpthumb.jpg data-srcset=https...

14/05/2021

Boris FX Announces Silhouette 2021

Boris FX Announces Silhouette 2021 data-src=https://creativecow.net/wp-content/uploads/avatars/263018/5fa08c562aaf4-bpthumb.jpg data-srcset=https://creative...

14/05/2021

Waves eMotion LV1 Mixer Chosen to Mix and Broadcast the Heat Latin Music Awards

Waves eMotion LV1 Mixer Chosen to Mix and Broadcast the Heat Latin Music Awards data-src=https://creativecow.net/wp-content/uploads/avatars/263018/5fa08c562...

14/05/2021

Milk Goes Intergalactic on VFX for SkyOne

Milk Goes Intergalactic on VFX for SkyOne data-src=https://creativecow.net/wp-content/uploads/avatars/263018/5fa08c562aaf4-bpthumb.jpg data-srcset=https://c...

14/05/2021

MTV Gives the 2021 MTV Movie & TV Awards TV's Biggest Promo Push

B&C has partnered with always-on TV ad measurement and attribution company iSpot.tv to bring you a weekly chart we call Promo Mojo: exclusive data showing the t...

14/05/2021

Nickelodeon's Baby Shark' Shows Sharp Ratings Teeth

Nickelodeon said its new series, Baby Shark's Big Show! is generating big ratings two months since its TV premiere on March 26....

14/05/2021

Telemundo Upfront: 1,000 Hours of Original Content in 2021-22

NBCUniversal Telemundo said it plans to offer more than 1,000 hours of original content next broadcast year....

14/05/2021

CatapultX Launches On-Screen AI Ad Message Platform

Ad tech company CatapultX launched its On-Stream platform, which uses artificial intelligence to place non-interruptive messages on screen within relevant video...

14/05/2021

A Little Late With Lilly Singh' Signs Off June 3

A Little Late With Lilly Singh, airing in the 1:30 a.m. slot on NBC, ends June 3. The 30-minute A Little Late premiered in September 2019....

14/05/2021

Study: Broadcasters Deliver Record Amount of Local News

For the second year in a row, local TV stations produced a record amount of news in 2020 despite pandemic-related budget cuts and revenue shortfalls. Likely hel...

14/05/2021

NBC Orders Two More Seasons of Days of Our Lives'

Days of Our Lives has been picked up for two more years. The renewal from NBC takes the daytime show through seasons 57 and 58....

14/05/2021

Primetime Ratings Tuesday: CBS, NBC on Top

CBS and NBC shared the Tuesday ratings crown. Both had a 0.6 in viewers 18-49, per the Nielsen overnights, and a 4 share. CBS was paced by NCIS and NBC had This...

14/05/2021

Aqua Teen Hunger Force Leads Movies Coming From Adult Swim

AT&T's Adult Swim is in the movie business, giving a green light to original films based on three of its iconic titles: Aqua Teen Hunger Force, Metalocalyps...

14/05/2021

Who Is the Next Jeopardy!' Host?' The Answer Awaits

The procession of guest hosts fronting Jeopardy! has rejuvenated interest in the beloved game show, and turned the notion of predicting Alex Trebek's succes...

13/05/2021

SVG SportsTech On Demand: WorldStage's Michael May on Elevating Virtual Events Through Digital Oasis, xR Technology

SVG SportsTech On Demand: WorldStage's Michael May on Elevating Virtual Even...

13/05/2021

SVG SportsTech On Demand: M2A Media's CEO Marina Kalkanis on the Growth of Live Video Delivery Over the Public Cloud

SVG SportsTech On Demand: M2A Media's CEO Marina Kalkanis on the Growth of L...

13/05/2021

SVG SportsTech On Demand: Imagine Communications' Cassidy Phillips on the Move from SDI to IP

SVG SportsTech On Demand: Imagine Communications' Cassidy Phillips on the Mo...

13/05/2021

Never.no Connects Global Sports Fans With Bee-On Cloud-Based Engagement Platform

Never.no Connects Global Sports Fans With Bee-On Cloud-Based Engagement Platform By Kristian Hernandez, Associate Editor Thursday, May 13, 2021 - 9:15 am P...