Sony Pixel Power calrec Sony

How Omniverse Wove a Real CEO - and His Toy Counterpart - Together With Stunning Demos at GTC

21/12/2021

It could only happen in NVIDIA Omniverse - the company's virtual world simulation and collaboration platform for 3D workflows.

And it happened during an interview with a virtual toy model of NVIDIA's CEO, Jensen Huang.

What are the greatest one of Toy Jensen's creators asked, stumbling, then stopping before completing his scripted question.

Unfazed, the tiny Toy Jensen paused for a moment, considering the answer carefully.

The greatest are those, Toy Jensen replied, who are kind to others.

Leading-edge computer graphics, physics simulation, a live CEO, and a supporting cast of AI-powered avatars came together to make NVIDIA's GTC keynote - delivered using Omniverse - possible.

Along the way, a little soul got into the mix, too.

The AI-driven comments, added to the keynote as a stinger, provided an unexpected peek at the depth of Omniverse's technology.

Omniverse is the hub in which all the various research domains converge and align and work in unison, says Kevin Margo, a member of NVIDIA's creative team who put the presentation together. Omniverse facilitates the convergence of all of them.

Toy Jensen's ad-lib capped a presentation that seamlessly mixed a real CEO with virtual and real environments as Huang took viewers on a tour of how NVIDIA technologies are weaving AI, graphics and robotics together with humans in real and virtual worlds.

Real CEO, Digital Kitchen While the CEO viewers saw was all real, the environment around him morphed as he spoke to support the story he was telling.

Viewers saw Huang deliver a keynote that seemed to begin, like so many during the global COVID pandemic, in Huang's kitchen.

Then, with a flourish, Huang's kitchen - modeled down to the screws holding its cabinets together - slid away from sight as Huang strolled toward a virtual recreation of Endeavor's gleaming lobby.

One of our goals is to find a way to elevate our keynote events, Margo says. We're always looking for those special moments when we can do something novel and fantastical, and that showcase NVIDIA's latest technological innovations.

It was the start of a visual journey that would take Huang from that lobby to Shannon's, a gathering spot inside Endeavor, through a holodeck, and a data center with stops inside a real robotics lab and the exterior of Endeavor.

Virtual environments such as Huang's kitchen were created by a team using familiar tools supported by Omniverse such as Autodesk Maya and 3ds Max, and Adobe Substance Painter.

Omniverse served to connect them all in real-time - so each team member could see changes made by colleagues using different tools simultaneously, accelerating their work.

That was critical, Margo says.

The virtual and the real came together quickly once live filming began.

A small on-site video team recorded Huang's speech in just four days, starting October 30, in a spare pair of conference rooms at NVIDIA's Silicon Valley headquarters.

Omniverse allowed NVIDIA's team to project the dynamic virtual environments their colleagues had created on a screen behind Huang.

As a result, the light spill onto Huang changed as the scene around him changed, better integrating him into the virtual environment.

And as Huang moved through the scene, or as the camera shifted, the environment changed around Huang.

As the camera moves, the perspective and parallax of the world on the video wall responds accordingly, Mago says.

And because Huang could see the environment projected on the screens around him, he was better able to navigate each scene.

At the Speed of Omniverse All of this accelerated the work of NVIDIA's production team, which had most of what they needed in-camera after each shot rather than adding elaborate digital sets in post-production.

As a result, the video team quickly created a presentation seamlessly blending a real CEO with virtual and real-world settings.

However, Omniverse was more than just a way to speed collaboration between creatives working with real and digital elements hustling to hit a deadline. It also served as the platform that knit the string of demos featured in the keynote together.

To help developers create intelligent, interactive agents with Omniverse that can see, speak, converse on a wide range of subjects and understand naturally spoken intent, Huang announced Omniverse Avatar.

Omniverse brings together a deep stack of technologies - from ray-tracing to recommender systems - that were mixed and matched throughout the keynote to drive a series of stunning demos.

In a demo that swiftly made headlines, Huang showed how Project Tokkio for Omniverse Avatar connects Metropolis computer vision, Riva speech AI, avatar animation and graphics into a real-time conversational AI robot - the Toy Jensen Omniverse Avatar.

The conversation between three of NVIDIA's engineers and a tiny toy model of Huang was more than just a technological tour de force, demonstrating expert, natural Q&A.

It showed how photorealistic modeling of Toy Jensen and his environment - right down to the glint on Toy Jensen's glasses as he moved his head - and NVIDIA's Riva speech synthesis technology powered by the Megatron 530B large language model could support natural, fluid conversations.

To create the demo, NVIDIA's creative team created the digital model in Maya Substance, and Omniverse did the rest.

None of it was manual, you just load up the animation assets and talk to it, he said.

Huang also showed a second demo of Project Tokkio, a customer-service avatar in a restaurant kiosk that was able to see, converse with and understand two customers.

Rather than relying on Megatron, however, this model relied on a model that integrated the restaurant's menu, allowing the avatar to smoothly guide customers through their op
LINK: https://blogs.nvidia.com/blog/2021/12/21/how-omniverse-keynote/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

04/08/2026

Dalet Announces Commercial Availability of Dalia, Bringing Media-Aware Agentic AI to Enterprise Productions

Dalet, a leading technology and service provider for media-rich organizations, t...

04/07/2026

Detective Conan: Fallen Angel of the Highway Opens in Dolby Cinemas Across Japan, Presented in Dolby Atmos and Dolby ...

April 7 2026, 19:00 (PDT) Detective Conan: Fallen Angel of the Highway Opens in...

01/06/2026

Dolby Sets the New Standard for Premium Entertainment at CES 2026

January 6 2026, 05:30 (PST) Dolby Sets the New Standard for Premium Entertainment at CES 2026 Throughout the week, Dolby brings to life the latest innovatio...

02/05/2026

Dalet Flex LTS Delivers Smarter Search, Faster Editing, and an AI-Ready Foundation for Modern Media

Dalet, a leading technology and service provider for media-rich organizations, t...

01/05/2026

NBCUniversal's Peacock to Be First Streamer to Integrate Dolby's Full Suite of Premium Picture and Sound Innovations

January 5 2026, 18:30 (PST) NBCUniversal's Peacock to Be First Streamer to ...

30/04/2026

Study: Rokus Low-Cost, Ad-Free Howdy Streamer Hits 1 Million Subs

Share Copy link Facebook X Linkedin Bluesky Email...

30/04/2026

MLS Innovation Lab Selects AI Partners

Share Copy link Facebook X Linkedin Bluesky Email...

30/04/2026

NRB Files FCC Complaint Over Jimmy Kimmel Live!' Monologue

Share Copy link Facebook X Linkedin Bluesky Email...

30/04/2026

D.C. Court Denies Emergency Stay of Nexstar/Tegna Merger

Share Copy link Facebook X Linkedin Bluesky Email...

30/04/2026

NAB Criticizes FCC for Ordering Early Renewal of ABC-Owned Stations

Share Copy link Facebook X Linkedin Bluesky Email...

30/04/2026

FCC Approves Station Swaps Between Scripps and Gray Media

Share Copy link Facebook X Linkedin Bluesky Email...

30/04/2026

Telestream Introduces Pulse - a Software-Defined Test an...

A flexible monitoring platform designed to simplify ST 2110 operations, consolidate vendor tools, and support modern live production environments. See it at NAB...

30/04/2026

Langlev Takes Run Amok to Sundance with Zeiss Supreme Pri...

Sundance-premiering Run Amok is an expressive, unconventional take on today's teen experience, replete with musical numbers. When cinematographer Shachar ...

30/04/2026

Studio Technologies Elevates Jacksonville State Universit...

JACKSONVILLE, AL, APRIL 29, 2026 Jacksonville State University, known as Jax State and a proud NCAA Division I member of Conference USA, has transformed its a...

30/04/2026

Knowledge Network selects ThinkAnalytics to launch AI-pow...

Transforming viewer and content data into real-time intelligence to deliver relevant streaming experiences at scale ThinkAnalytics, the global leader in AI-pow...

30/04/2026

Sports Production, Delivery is Big Biz at NAB

Sports Production, Delivery is Big Biz at NAB Andy Marken April 29, 2026 0 Comments Hero image source: NAB One of the neat things about trade shows i...

30/04/2026

April 29, 2026

Scripps Research ranks third in 2026 Cure Innovation Index April 29, 2026 LA JOLLA, CA Scripps Research ranked third in the inaugural 2026 Cure Innovation In...

29/04/2026

Churchill Downs Racetrack Lifts Curtain on New Big Board Ahead of Kentucky Derby Week

It was a delicate job in a 150-year-old venue laden with traditions. Begun at th...

29/04/2026

How the MLS Innovation Lab Enhances Everything From Athlete Performance to Production to Ticket Sales

In annual event, the league gives startup companies the opportunity to prove the...

29/04/2026

Save the Date: SVG Venues & Teams Summit Travels South to Miamis Kaseya Center on Aug. 12

Panel discussions, networking, and a facility tour will take place in the renova...

29/04/2026

The Cast and Crew of Conbody VS Everybody Have Each Other's Backs for Life

(L-R) Derek Drescher, Coss Marte, and Syretta Wright have each other's backs. (Micheal Hurcomb/Shutterstock for Sundance Film Festival) By Veronika Lee Cla...

29/04/2026

Free Tilt EQ plug-in from Techivation

Combines EQ and harmonic distortion Techivation's latest release is a simple EQ designed to offer quick control over a source's overall tonal balanc...

29/04/2026

Expressive E launch the Osmose CE

Two new MPE controllers announced Expressive E caused quite a stir when they released the Osmose, making the sort of expression that was once reserved for p...

29/04/2026

iZotope RX 12 is here

New modules & enhanced machine-learning The latest version of iZotope's flagship restoration suite is now available, and now offers over 50 tools design...

29/04/2026

Surgeon Dr Jasmina Kevric wins 2026 Les Murray Award

Surgeon Dr Jasmina Kevric wins 2026 Les Murray Award 29 April, 2026 Media releases Australia for UNHCR and SBS are proud to announce that Dr Jasmina Kevric...

29/04/2026

YEP Spotlight: Julissa Padilla

Some people stumble into their passion. Julissa Padilla walked straight into a film vault. For her, entertainment was never just about the movies themselves. It...

29/04/2026

Clear-Com Promotes Brian Grahn, Ben Turnwell to Expanded Roles

Share Copy link Facebook X Linkedin Bluesky Email...

29/04/2026

The CW Unveils Major Streaming Deals with ESPN and Roku

Share Copy link Facebook X Linkedin Bluesky Email...

29/04/2026

Clear-Com Announces New Roles for Brian Grahn and Ben Tur...

Clear-Com has appointed Brian Grahn as Market Outreach Manager of the Americas and Ben Turnwell as Business Development Manager for EMEA live, expanding their ...

29/04/2026

Reduce Workflow Complexity with nxtedition at MPTS 2026

nxtedition is bringing its range of consolidated production tools to MPTS 2026, with new developments spanning transcription, editing, graphics and AI-assisted ...

29/04/2026

Telxius taps Synamedia to unlock seamless global multi-CD...

Quortex Switch to boost the streaming experience for Telxius customers, reaching millions of viewers worldwide Synamedia and Telxius, the leading global connec...

29/04/2026

freispace and Projective Announce Joint Showcase of Integ...

freispace, the leading ERP-as-a-Service platform for media and entertainment production, and Projective, a leading provider of post-production collaboration tec...

29/04/2026

DHD Celebrates 30th Anniversary at 2026 NAB Show

DHD reports strong interest in its broadcast audio product range, exhibited at the April 19th-22nd NAB Show in Las Vegas. The event attracted a claimed 58,000 a...

29/04/2026

Student Spotlight: Alan Catz

Student Spotlight: Alan Catz The Argentine film and game composer talks about working on League of Legends, receiving Berklee's BMI Award, and the lifelon...

29/04/2026

Jay Jennings Builds the Worlds You Hear on Screen

Jay Jennings Builds the Worlds You Hear on Screen The supervising sound designer behind A Minecraft Movie, The Meg, Letters from Iwo Jima, and dozens of other...

29/04/2026

Paramount Will be 49.5% Foreign Owned After WBD Merger

Share Copy link Facebook X Linkedin Bluesky Email...

29/04/2026

NAB Show 2026: AI, Vertical and BPS Dominate Broadcasters' Discussions

Share Copy link Facebook X Linkedin Bluesky Email...

29/04/2026

IAB Tech Lab Announces New OpenRTB Attributes

Share Copy link Facebook X Linkedin Bluesky Email...

29/04/2026

Meet the eight finalists for RT Today's TV Home Cook competition

Voting opens at 4pm today RT 's Today show have announced the eight finalists for their TV Home Cook competition. Amateur cooks from Cork, Dublin, Galway ...

29/04/2026

VEON and Kyivstar Fulfill Commitment to Invest USD 1 Billion in Ukraine over 2023-2027 Ahead of Schedule

29 Apr 2026 VEON and Kyivstar Fulfill Commitment to Invest USD 1 Billion in Ukr...

29/04/2026

U challenges stars and their dogs to compete for Wagging Rights

Rhod Gilbert, Harriet Kemsley, Kae Kurd, Sara Pascoe and Vicki Pattison to take part in brand new series on free streaming service U London, 29th April 2026: F...

29/04/2026

Katie Price: Nothing to Hide, a Sky Original documentary series from BAFTA-winning Mindhouse, coming this summer

Wednesday 29 April 2026 Katie Price: Nothing to Hide, a Sky Original documentar...

29/04/2026

Sky announces upcoming documentary series The Lie That Exposed The Truth

Re-examining the case of Ellie Williams and the wider story of grooming in the town of BarrowWednesday 29 April 2026 Sky announces upcoming documentary series ...

29/04/2026

Jennifer Garner to lead an all-star cast in new Sky Exclusive summer escape The Five Star Weekend, coming to screens this July

Wednesday 29 April 2026 Jennifer Garner to lead an all-star cast in new Sky Exc...

29/04/2026

'SUPERNOVA: GENESIS' Reached a Peak Audience of More Than 6.5 Million Viewers on Netflix

Back to All News SUPERNOVA: GENESIS Reached a Peak Audience of More Than 6.5 Mi...

29/04/2026

dB Sponsors College Sustainability Week

Students and staff from Hills Road Sixth Form College in Cambridge ran a 4.5km course around the roads of Cambridge as part of their annual programme of sustain...

29/04/2026

Listen to the birds sing! Experience mother nature in a seven-hour radio marathon

The Dawn Chorus airs Sunday 3 May from midnight to 7am on RT Radio 1 and RT ly...

29/04/2026

April 28, 2026

Jin-Quan Yu elected to the National Academy of Sciences Yu is recognized for his pioneering work in synthetic organic chemistry. April 28, 2026 LA JOLLA, CA S...

28/04/2026

Sound for the NFL Draft Show Balances Often Conflicting Goals

The audio team for the entertainment event must blend speech intelligibility with full-range music reproduction while considering the broadcast Last week's...