Sony Pixel Power calrec Sony

How Omniverse Wove a Real CEO - and His Toy Counterpart - Together With Stunning Demos at GTC

21/12/2021

It could only happen in NVIDIA Omniverse - the company's virtual world simulation and collaboration platform for 3D workflows.

And it happened during an interview with a virtual toy model of NVIDIA's CEO, Jensen Huang.

What are the greatest one of Toy Jensen's creators asked, stumbling, then stopping before completing his scripted question.

Unfazed, the tiny Toy Jensen paused for a moment, considering the answer carefully.

The greatest are those, Toy Jensen replied, who are kind to others.

Leading-edge computer graphics, physics simulation, a live CEO, and a supporting cast of AI-powered avatars came together to make NVIDIA's GTC keynote - delivered using Omniverse - possible.

Along the way, a little soul got into the mix, too.

The AI-driven comments, added to the keynote as a stinger, provided an unexpected peek at the depth of Omniverse's technology.

Omniverse is the hub in which all the various research domains converge and align and work in unison, says Kevin Margo, a member of NVIDIA's creative team who put the presentation together. Omniverse facilitates the convergence of all of them.

Toy Jensen's ad-lib capped a presentation that seamlessly mixed a real CEO with virtual and real environments as Huang took viewers on a tour of how NVIDIA technologies are weaving AI, graphics and robotics together with humans in real and virtual worlds.

Real CEO, Digital Kitchen While the CEO viewers saw was all real, the environment around him morphed as he spoke to support the story he was telling.

Viewers saw Huang deliver a keynote that seemed to begin, like so many during the global COVID pandemic, in Huang's kitchen.

Then, with a flourish, Huang's kitchen - modeled down to the screws holding its cabinets together - slid away from sight as Huang strolled toward a virtual recreation of Endeavor's gleaming lobby.

One of our goals is to find a way to elevate our keynote events, Margo says. We're always looking for those special moments when we can do something novel and fantastical, and that showcase NVIDIA's latest technological innovations.

It was the start of a visual journey that would take Huang from that lobby to Shannon's, a gathering spot inside Endeavor, through a holodeck, and a data center with stops inside a real robotics lab and the exterior of Endeavor.

Virtual environments such as Huang's kitchen were created by a team using familiar tools supported by Omniverse such as Autodesk Maya and 3ds Max, and Adobe Substance Painter.

Omniverse served to connect them all in real-time - so each team member could see changes made by colleagues using different tools simultaneously, accelerating their work.

That was critical, Margo says.

The virtual and the real came together quickly once live filming began.

A small on-site video team recorded Huang's speech in just four days, starting October 30, in a spare pair of conference rooms at NVIDIA's Silicon Valley headquarters.

Omniverse allowed NVIDIA's team to project the dynamic virtual environments their colleagues had created on a screen behind Huang.

As a result, the light spill onto Huang changed as the scene around him changed, better integrating him into the virtual environment.

And as Huang moved through the scene, or as the camera shifted, the environment changed around Huang.

As the camera moves, the perspective and parallax of the world on the video wall responds accordingly, Mago says.

And because Huang could see the environment projected on the screens around him, he was better able to navigate each scene.

At the Speed of Omniverse All of this accelerated the work of NVIDIA's production team, which had most of what they needed in-camera after each shot rather than adding elaborate digital sets in post-production.

As a result, the video team quickly created a presentation seamlessly blending a real CEO with virtual and real-world settings.

However, Omniverse was more than just a way to speed collaboration between creatives working with real and digital elements hustling to hit a deadline. It also served as the platform that knit the string of demos featured in the keynote together.

To help developers create intelligent, interactive agents with Omniverse that can see, speak, converse on a wide range of subjects and understand naturally spoken intent, Huang announced Omniverse Avatar.

Omniverse brings together a deep stack of technologies - from ray-tracing to recommender systems - that were mixed and matched throughout the keynote to drive a series of stunning demos.

In a demo that swiftly made headlines, Huang showed how Project Tokkio for Omniverse Avatar connects Metropolis computer vision, Riva speech AI, avatar animation and graphics into a real-time conversational AI robot - the Toy Jensen Omniverse Avatar.

The conversation between three of NVIDIA's engineers and a tiny toy model of Huang was more than just a technological tour de force, demonstrating expert, natural Q&A.

It showed how photorealistic modeling of Toy Jensen and his environment - right down to the glint on Toy Jensen's glasses as he moved his head - and NVIDIA's Riva speech synthesis technology powered by the Megatron 530B large language model could support natural, fluid conversations.

To create the demo, NVIDIA's creative team created the digital model in Maya Substance, and Omniverse did the rest.

None of it was manual, you just load up the animation assets and talk to it, he said.

Huang also showed a second demo of Project Tokkio, a customer-service avatar in a restaurant kiosk that was able to see, converse with and understand two customers.

Rather than relying on Megatron, however, this model relied on a model that integrated the restaurant's menu, allowing the avatar to smoothly guide customers through their op
LINK: https://blogs.nvidia.com/blog/2021/12/21/how-omniverse-keynote/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

01/06/2026

Dolby Sets the New Standard for Premium Entertainment at CES 2026

January 6 2026, 05:30 (PST) Dolby Sets the New Standard for Premium Entertainment at CES 2026 Throughout the week, Dolby brings to life the latest innovatio...

02/05/2026

Dalet Flex LTS Delivers Smarter Search, Faster Editing, and an AI-Ready Foundation for Modern Media

Dalet, a leading technology and service provider for media-rich organizations, t...

01/05/2026

NBCUniversal's Peacock to Be First Streamer to Integrate Dolby's Full Suite of Premium Picture and Sound Innovations

January 5 2026, 18:30 (PST) NBCUniversal's Peacock to Be First Streamer to ...

01/04/2026

DOLBY AND DOUYIN EMPOWER THE NEXT GENERATON OF CREATORS WITH DOLBY VISION

January 4 2026, 18:00 (PST) DOLBY AND DOUYIN EMPOWER THE NEXT GENERATON OF CREATORS WITH DOLBY VISION Douyin Users Can Now Create And Share Videos With Stun...

27/03/2026

NAB Pans Netflix's Coverage of MLB Opening Day

Share Copy link Facebook X Linkedin Bluesky Email...

27/03/2026

2026 NAB Show Exhibitor Insight: Netgear

Share Copy link Facebook X Linkedin Bluesky Email...

27/03/2026

Marshall Electronics Showcases New Feature-Rich CV320 and CV520 IP and 3G-SDI POV Cameras at NAB 2026

Marshall Electronics Showcases New Feature-Rich CV320 and CV520 IP and 3G-SDI PO...

27/03/2026

Sony Electronics Inc. Elevates Professional Video Workflows with Powerful Updates to Monitor & Control and Catalyst Tools

Sony Electronics Inc. Elevates Professional Video Workflows with Powerful Update...

27/03/2026

GatesAir Extends AirWatch365 Managed Service with Edge Gateway Site Appliance

GatesAir Extends AirWatch365 Managed Service with Edge Gateway Site Appliance Brie Clayton March 27, 2026 0 Comments NAB marks global launch of servic...

27/03/2026

GatesAir Expands AirWatch365 Managed Service

Share Copy link Facebook X Linkedin Bluesky Email...

27/03/2026

Dejero, Eutelsat, Others to Offer 'Field to Air' Demo at NAB Show

Share Copy link Facebook X Linkedin Bluesky Email...

27/03/2026

Global NextGen TV Takes Center Stage at 2026 NAB Show

Share Copy link Facebook X Linkedin Bluesky Email...

27/03/2026

USSI Global Opens Media Transport Solutions Lab

Share Copy link Facebook X Linkedin Bluesky Email...

27/03/2026

Net Insight Elevates Nimbra Edge to Govern Live Media Sys...

At NAB Show 2026, Net Insight will showcase the next evolution of Nimbra Edge, its orchestration and control layer designed to manage live media services across...

27/03/2026

Harmonic Delivers Winning Live Sports Streaming Innovatio...

Harmonic (NASDAQ: HLIT) today announced powerful new innovations that further elevate the company's sports streaming solution. The advanced capabilities enh...

27/03/2026

Bitmovin s Player Web X Expands with Performance First Ad...

Bitmovin, the leading provider of video streaming solutions, today announced significant new capabilities for Player Web X, its next-generation web video player...

27/03/2026

CS live Builds Future Ready OB Van With Riedel MediorNet...

Riedel Communications today announced that Czech-based production company CS live has equipped its newest outside broadcast (OB) van with an integrated Riedel i...

27/03/2026

MPTS 2026 Unveils First Wave of Speakers Featuring BBC Sk...

130 Industry Experts Confirmed as Show Celebrates its 10th Anniversary MPTS, the UK's largest and most influential event for the media, production and tech...

27/03/2026

Grass Valley and NVP Elevate LALIGA Broadcast Production...

Grass Valley today announced the successful completion of a major live production deployment with NVP, a leading European media company specializing in live spo...

27/03/2026

Appear Business Momentum Drives Leadership in Live Produc...

Showcases resilient solutions for satellite-to-IP migration, REMI and hybrid live production Appear ASA (Appear, OSE:APR), a global leader in live production t...

27/03/2026

Lightware MX2 delivers rock-solid 4K60 HDMI 10-bit signal...

San Francisco, California, March 2026 - Microsoft Ignite, a major annual conference hosted by Microsoft for developers, IT professionals, partners and business ...

27/03/2026

Clear-Coms EQUIP Wireless System Keeps Youth With A Missi...

Clear-Com is proud to highlight its support for worship teams through professional communication solutions, with the deployment of its EQUIP wireless system a...

27/03/2026

Orban Collaborates with Sage on Virtualized EAS Technology Demo

Orban Collaborates with Sage on Virtualized EAS Technology Demo Brie Clayton March 26, 2026 0 Comments Sage Alerting Systems (SAS), in cooperation wit...

27/03/2026

Berklee Alums Bring GIANT to Broadway

Berklee Alums Bring GIANT to Broadway Brian and Dayna Lee BM '13 met and fell in love at Berklee, and now their production companys first Broadway debut i...

27/03/2026

RT is Supporting 20 Arts and Cultural Events all over Ireland this April

RT Supporting the Arts RT is Supporting 20 Arts and Cultural Events all over Ireland this April RT is proud to support an exciting month of arts and cul...

27/03/2026

Average of 1.37 million viewers watched Irish World Cup heartbreak on RT2

Average of 1.37 million viewers watched Irish World Cup heartbreak on RT 2 Over one million streams on RT Player Highest number of streams for any single eve...

27/03/2026

Dolby Unveils Dolby House Shanghai, its First Global Flagship Experience Center

March 27 2026, 01:30 (PDT) Dolby Unveils Dolby House Shanghai, its First Global Flagship Experience Center A New Global Platform Showcasing Dolbys Innovatio...

27/03/2026

Dolby and QQ Music Advance Streaming Music with Enhancements to Dolby AC-4

March 27 2026, 03:30 (PDT) Dolby and QQ Music Advance Streaming Music with Enhancements to Dolby AC-4 Delivering greater fidelity, clarity and detail to Dol...

26/03/2026

Audio-Technica and Extron Announce Microsoft Teams-Certified Enterprise Solutions

Extron and Audio-Technica have announced that their Design Solutions devices, Ce...

26/03/2026

SPORTEL Asia Returns to Singapore with 370 Participants from 35 Countries

SPORTEL Asia took place March 24-25 at the Orchard Hotel Singapore, drawing 370 participants representing more than 220 companies from 35 countries. Approximate...

26/03/2026

NAB 2026: LTN to Showcase Satellite-to-IP Migration Solutions

LTN will demonstrate satellite-to-IP migration solutions for broadcasters, cable networks, MVPDs, and station groups at NAB Show 2026 (Booth W2100). The C-Ban...

26/03/2026

Seattle Mariners Upgrade Center Field Video Display at T-Mobile Park with Daktronics

The Seattle Mariners have partnered with Daktronics to install an upgraded outfi...

26/03/2026

Esports Nations Cup 2026 Names National Team Partners Across 100+ Countries

The Esports Foundation (EF) has awarded National Team Partner status to organizations and individuals across more than 100 countries and territories for the Esp...

26/03/2026

NAB 2026: Telestream Expands Vantage Integration with Adobe Premiere, Media Encoder, and Frame.io

Telestream has announced expanded integration between its Vantage workflow platf...

26/03/2026

NAB 2026: FOR-A Brings Nippon TV's viztrick AiDi AI Production Tool to Market with NVIDIA

FOR-A Corporation has announced a partnership with Nippon TV and NVIDIA to distr...

26/03/2026

CBS Sports Reaches New Long-Term Multiplatform Rights Agreement with the WNBA

CBS Sports and the WNBA have expanded their partnership through a new long-term, multiplatform rights agreement. CBS Sports will feature up to 20 regular-season...

26/03/2026

Zac Brown Band to Perform Sunday Night Baseball Open on NBC and Peacock

Zac Brown Band will star in the Sunday Night Baseball show open on NBC and Peacock, NBC Sports announced. The open was filmed at American Family Field, home of ...

26/03/2026

2026 MLB Season Opens With Full Slate of National Games

Major League Baseball opens the 2026 season with nationally televised games across multiple media partners. Fans can visit MLB.com/Watch for a complete schedule...

26/03/2026

DIRECTV Adds Nine MLB Team Channels for 2026 Season

DIRECTV has announced an agreement to carry nine new MLB-operated team channels, covering the Cincinnati Reds, Detroit Tigers (Detroit SportsNet), Kansas City R...

26/03/2026

Nationals.TV Now Available on Charter, Comcast, Cox, DIRECTV, Verizon Fios and More

Nationals.TV, the Washington Nationals' local broadcast channel, is now avai...

26/03/2026

MLB.TV Launches on the ESPN App Starting Opening Day

Starting Thursday, March 26, MLB.TV will be available on the ESPN App, giving fans access to more than 2,000 out-of-market regular season games. Fans can also r...

26/03/2026

What's Next for Spotify and NAVER in Korea After a Stockholm Summit

Spotify's partnership with NAVER, Korea's leading digital platform, is built around one simple goal: making it easier for listeners in Korea to discover...

26/03/2026

EastWest DrumX: Early access

Exclusive to ComposerCloud for first month Sample library and virtual instrument gurus EastWest have announced the upcoming release of a new virtual drum i...

26/03/2026

Dynamic Range Day 2026

Friday 27 March Friday 27 March will mark Dynamic Range Day 2026, the annual day of online activity that was started in 2010 by mastering engineer Ian Sheph...

26/03/2026

ASMR Choir from SRM Sounds

Includes four separate instruments Max Richter's SRM Sounds venture have just announced their fifth release, ASMR Choir. Captured at the main live room ...

26/03/2026

IMPORTANT FEEDBACK: PESP 6

The National Film and Video Foundation (NFVF), an agency of the Department of Sport, Arts and Culture (DSAC), has been tasked with administering the Sixth Presi...

26/03/2026

Why YEP Changed My Career (And Why It Might Change Yours)

By Matt Klein Hello to everyone reading this - whether you're a current YEP, HPA member, or just YEP-curious. I'm Matt Klein, one of the co-chairs of t...

26/03/2026

CBS Sports Inks New WNBA Deal

Share Copy link Facebook X Linkedin Bluesky Email...

26/03/2026

Apogee Insight Acquires PMA Research

Share Copy link Facebook X Linkedin Bluesky Email...