In a turducken of a demo, NVIDIA researchers stuffed four AI models into a serving of digital avatar technology for SIGGRAPH 2021's Real-Time Live showcase - winning the Best in Show award.
The showcase, one of the most anticipated events at the world's largest computer graphics conference, held virtually this year, celebrates cutting-edge real-time projects spanning game technology, augmented reality and scientific visualization. It featured a lineup of jury-reviewed interactive projects, with presenters hailing from Unity Technologies, Rensselaer Polytechnic Institute, the NYU Future Reality Lab and more.
Broadcasting live from our Silicon Valley headquarters, the NVIDIA Research team presented a collection of AI models that can create lifelike virtual characters for projects such as bandwidth-efficient video conferencing and storytelling.
The demo featured tools to generate digital avatars from a single photo, animate avatars with natural 3D facial motion and convert text to speech.
Making digital avatars is a notoriously difficult, tedious and expensive process, said Bryan Catanzaro, vice president of applied deep learning research at NVIDIA, in the presentation. But with AI tools, there is an easy way to create digital avatars for real people as well as cartoon characters. It can be used for video conferencing, storytelling, virtual assistants and many other applications.
AI Aces the Interview In the demo, two NVIDIA research scientists played the part of an interviewer and a prospective hire speaking over video conference. Over the course of the call, the interviewee showed off the capabilities of AI-driven digital avatar technology to communicate with the interviewer.
The researcher playing the part of interviewee relied on an NVIDIA RTX laptop throughout, while the other used a desktop workstation powered by RTX A6000 GPUs. The entire pipeline can also be run on GPUs in the cloud.
While sitting in a campus coffee shop, wearing a baseball cap and a face mask, the interviewee used the Vid2Vid Cameo model to appear clean-shaven in a collared shirt on the video call (seen in the image above). The AI model creates realistic digital avatars from a single photo of the subject - no 3D scan or specialized training images required.
The digital avatar creation is instantaneous, so I can quickly create a different avatar by using a different photo, he said, demonstrating the capability with another two images of himself.
Instead of transmitting a video stream, the researcher's system sent only his voice - which was then fed into the NVIDIA Omniverse Audio2Face app. Audio2Face generates natural motion of the head, eyes and lips to match audio input in real time on a 3D head model. This facial animation went into Vid2Vid Cameo to synthesize natural-looking motion with the presenter's digital avatar.
Not just for photorealistic digital avatars, the researcher fed his speech through Audio2Face and Vid2Vid Cameo to voice an animated character, too. Using NVIDIA StyleGAN, he explained, developers can create infinite digital avatars modeled after cartoon characters or paintings.
The models, optimized to run on NVIDIA RTX GPUs, easily deliver video at 30 frames per second. It's also highly bandwidth efficient, since the presenter is sending only audio data over the network instead of transmitting a high-resolution video feed.
Taking it a step further, the researcher showed that when his coffee shop surroundings got too loud, the RAD-TTS model could convert typed messages into his voice - replacing the audio fed into Audio2Face. The breakthrough text-to-speech, deep learning-based tool can synthesize lifelike speech from arbitrary text inputs in milliseconds.
RAD-TTS can synthesize a variety of voices, helping developers bring book characters to life or even rap The Real Slim Shady by Eminem, as the research team showed in the demo's finale.
SIGGRAPH continues through Aug. 13. Check out the full lineup of NVIDIA events at the conference and catch the premiere of our documentary, Connecting in the Metaverse: The Making of the GTC Keynote, on Aug. 11.
Most recent headlines
04/08/2024
Dalet, a leading technology and service provider for media-rich organizations, is excited to announce Santiago Solanas as its new Chief Executive Officer (CEO)....
03/06/2024
Dalet, a leading technology and service provider for media-rich organizations, a...
30/04/2024
European telecom satellite company SES announced today that it is acquiring rival Intelsat for $3.1 billion. The deal, unanimously approved by both companies...
30/04/2024
Bob Bakish, chief executive of Paramount stepped down on Monday, effective immediately. Bakish, who was a staunch ally of Shari Redstone, controlling shareholde...
30/04/2024
Hotspring said the breakthrough had been achieved through the acceleration of machine learning technology
By Matthew Corrigan
Published: April 30, 2024
Ho...
30/04/2024
Speaking about the deal, Adel Al-Saleh, CEO of SES, said: it will bring value-added, efficient, and reliable offerings for both companies media customers
By Je...
30/04/2024
Paramount said the Office of the CEO is working with its board to develop a comp...
30/04/2024
LOUISVILLE, Ky. In another sign that broadcasters are finally using NextGen TV broadcasts to offer viewers new features instead of simply simulcasting existing ...
30/04/2024
OKLAHOMA CITY The Oklahoma Journalism Hall of Fame will induct Blaise Labbe, a group news director for Sinclair Broadcast Group, May 2 at the 54th Anniversary I...
30/04/2024
ATLANTA and ENGLEWOOD, Colo. Cox Media Group and Dish have ended a dispute over retransmission fees that lasted nearly a year and a half with a new multi-year ...
30/04/2024
Video transport solution provider LTN has announced that it is working with TelevisaUnivision to help the Spanish-language media giant migrate to IP-based video...
30/04/2024
ALAMEDA, Calif. Clear-Com has released details about how it played a pivotal role in the seamless coverage of the recent solar eclipse on April 8, 2024 by NBC K...
30/04/2024
CINCINNATI The E.W. Scripps Company is adding Seth Walters to its sales leadership team as head of CTV sales, effective April 29....
30/04/2024
The force was strong at the Durham Bulls Athletic Park on Saturday night, April 27, 2024, as the night netted the ballpark's first official sellout crowd of...
30/04/2024
Revry, a network celebrating queer culture. said its second annual NewFront presentation will be a cabaret-type spectacle showcasing the spirit of the LGBTQ+ co...
30/04/2024
Family Entertainment Television (FETV) said that the Western series The Life and Legend of Wyatt Earp will be joining the network's afternoon lineup startin...
30/04/2024
Bob Bakish, president and CEO of Paramount Global, will depart the company as Paramount explores a merger, according to numerous published reports. The reports ...
30/04/2024
Value accretive transaction underpinned by 2.4 billion (NPV) of readily executable synergies.
Creating a stronger multi-orbit operator with 60% of revenue in...
30/04/2024
Facebook
Twitter
LinkedIn
Order intake: 5.0 billion, up 47% ( 46% on an organic basis1 )
Sales: 4.4 billion, up 9.8% ( 7.9% on an organic basis)
All f...
29/04/2024
(L-R) Mark Gatiss and Polly Walker on set
Monday 29 April 2024 Further casting is now confirmed for Bookish (6x70), a brand-new drama created by Emmy Award-wi...
29/04/2024
A groundbreaking initiative offering training and mentoring to 30 women journali...
29/04/2024
Even though we've been doing this for over 40 years, a spark of excitement r...
29/04/2024
L3Harris is delivering manpack radios to U.S. Army CH-47 Chinooks as part of the Air-to-Ground Networking Radio program, providing seamless, resilient communica...
29/04/2024
He writes in Defense One: Despite the near-universal acknowledgement throughout the U.S. government and defense industrial base of the criticality of resilient ...
29/04/2024
eds3_5_jq(document).ready(function($) { $(#eds_sliderM519).chameleonSlider_2_1({ content_source:......
29/04/2024
Codemill aims to revolutionize media workflow efficiency at this years NAB Show by introducing Just-In-Time (JIT) playback technology in Accurate.Video Validate...
29/04/2024
April 29, 2024
-- TF1 Chooses Broadpeak to Power Targeted Advertising for New...
29/04/2024
LOS ANGELES Allen Media Group (AMG) has partnered with LG Electronics to bring 223 Local Now FAST channels to LG's free streaming service, LG Channels, avai...
29/04/2024
OTTAWA Ross Video has announced the introduction of Raiden, a data-driven weather graphics software that combines data gathering, processing, and visualization ...
29/04/2024
Panalux, a leading rental provider of lighting and power solutions for the motion-picture industry and part of Panavision's end-to-end service offerings for...
29/04/2024
Cobalt Iron Inc., a leading provider of SaaS-based enterprise data protection, today announced that it has received a patent on its technology for dynamic autho...
29/04/2024
SDVI, the leading platform provider for cloud-native media supply chains, today announced that Rally Access Workstation, a fully managed solution for editing in...
29/04/2024
Premier Sports, a premium sports broadcaster, has selected QuickLink's Remote Commentary solution for introducing professional, high-quality remote commenta...
29/04/2024
Glookast has chosen Jigsaw24 Media as the only UK channel partner to represent their portfolio of ingest and workflow optimisation products. The agreement, sign...
29/04/2024
Clear-Com played a pivotal role in the seamless coverage of the recent solar eclipse on April 8, 2024. Leveraging its cutting-edge Eclipse HX Digital Matrix i...
29/04/2024
PlayBox Neo will promote its complete range of television channel management, graphic branding and playout solutions to EMEA region media content owners and bro...
29/04/2024
Lingo, RuPaul's word-twisting game show, returns for season two on CBS Friday, May 24. Two episodes air that night, and stream on Paramount Plus, too....
29/04/2024
ESPN and Amazon Prime Video are reportedly close to scoring television rights to the National Basketball Association, according to published reports....
29/04/2024
Judge Judy and Hot Bench, CBS Media Ventures' genre-leading court shows, have been renewed through the 2025-26 TV season in more than 95% of the country, Gr...
29/04/2024
Premier Sports, an Irish-based premium sports broadcaster, has selected QuickLink's Remote Commentary solution for introducing professional, high-quality re...
29/04/2024
The awards celebrate the craft of behind-the-scenes TV talent and the best programmes of 2023
By Matthew Corrigan
Published: April 29, 2024
The awards cel...
29/04/2024
The team at Milk had to create and deliver the VFX and environment work for the royal residences featured in the drama from scratch
By Jenny Priestley
Publis...
29/04/2024
CEO Bob Bakish is expected to leave the company as early as today, with a new leadership committee likely to run the company on an interim basis
By Jenny Pries...
29/04/2024
Albena Ivanova, director, media and entertainment and strategic products at CHAOS talks to TVBEurope about her route into the industry
By TVBEurope Staff
Pub...
29/04/2024
The acquisition gives Grand Technix the opportunity to expand its footprint in the audio visual and broadcast technology sectors
By Jenny Priestley
Published...
29/04/2024
29 04 2024 - Media release Screen Australia announces James J. Robinson's debut feature First Light
First Light
Principal Photography is underway on Firs...
29/04/2024
Capitol Broadcasting recently became the inaugural company honored with inductio...
29/04/2024
Tonight on House of Zwide: Zanele tells a surprised Zola that Faith fully suppor...
29/04/2024
Tonight on Scandal: A line is crossed and a mother starts to lose focusDon't miss Friday, 26 April's riveting episode of South African soapie Scandal! o...
In a turducken of a demo, NVIDIA researchers stuffed four AI models into a serving of digital avatar technology for SIGGRAPH 2021's Real-Time Live showcase - winning the Best in Show award.
