Sony Pixel Power calrec Sony

Crowning Achievement: NVIDIA Research Model Enables Fast, Efficient Dynamic Scene Reconstruction

09/12/2024

Content streaming and engagement are entering a new dimension with QUEEN, an AI model by NVIDIA Research and the University of Maryland that makes it possible to stream free-viewpoint video, which lets viewers experience a 3D scene from any angle.

QUEEN could be used to build immersive streaming applications that teach skills like cooking, put sports fans on the field to watch their favorite teams play from any angle, or bring an extra level of depth to video conferencing in the workplace. It could also be used in industrial environments to help teleoperate robots in a warehouse or a manufacturing plant.

The model will be presented at NeurIPS, the annual conference for AI research that begins Tuesday, Dec. 10, in Vancouver.

To stream free-viewpoint videos in near real time, we must simultaneously reconstruct and compress the 3D scene, said Shalini De Mello, director of research and a distinguished research scientist at NVIDIA. QUEEN balances factors including compression rate, visual quality, encoding time and rendering time to create an optimized pipeline that sets a new standard for visual quality and streamability.

Reduce, Reuse and Recycle for Efficient Streaming Free-viewpoint videos are typically created using video footage captured from different camera angles, like a multicamera film studio setup, a set of security cameras in a warehouse or a system of videoconferencing cameras in an office.

Prior AI methods for generating free-viewpoint videos either took too much memory for livestreaming or sacrificed visual quality for smaller file sizes. QUEEN balances both to deliver high-quality visuals - even in dynamic scenes featuring sparks, flames or furry animals - that can be easily transmitted from a host server to a client's device. It also renders visuals faster than previous methods, supporting streaming use cases.

In most real-world environments, many elements of a scene stay static. In a video, that means a large share of pixels don't change from one frame to another. To save computation time, QUEEN tracks and reuses renders of these static regions - focusing instead on reconstructing the content that changes over time.

Using an NVIDIA Tensor Core GPU, the researchers evaluated QUEEN's performance on several benchmarks and found the model outperformed state-of-the-art methods for online free-viewpoint video on a range of metrics. Given 2D videos of the same scene captured from different angles, it typically takes under five seconds of training time to render free-viewpoint videos at around 350 frames per second.

This combination of speed and visual quality can support media broadcasts of concerts and sports games by offering immersive virtual reality experiences or instant replays of key moments in a competition.

In warehouse settings, robot operators could use QUEEN to better gauge depth when maneuvering physical objects. And in a videoconferencing application - such as the 3D videoconferencing demo shown at SIGGRAPH and NVIDIA GTC - it could help presenters demonstrate tasks like cooking or origami while letting viewers pick the visual angle that best supports their learning.

The code for QUEEN will soon be released as open source and shared on the project page.

QUEEN is one of over 50 NVIDIA-authored NeurIPS posters and papers that feature groundbreaking AI research with potential applications in fields including simulation, robotics and healthcare. Among them is a best paper runner-up, titled Guiding a Diffusion Model With a Bad Version of Itself - one of just five papers recognized among the over 4,500 accepted this year.

Generative Adversarial Nets, the paper that first introduced GAN models, won the NeurIPS 2024 Test of Time Award. Cited more than 85,000 times, the paper was coauthored by Bing Xu, distinguished engineer at NVIDIA. Hear more from its lead author, Ian Goodfellow, research scientist at DeepMind, on the AI Podcast:

The AI Podcast Ep. 25: Google's Ian Goodfellow on How an Argument in a Bar Led to Generative Adversarial Networks

Learn more about NVIDIA Research at NeurIPS.

See the latest work from NVIDIA Research, which has hundreds of scientists and engineers worldwide, with teams focused on topics including AI, computer graphics, computer vision, self-driving cars and robotics.

Academic researchers working on large language models, simulation and modeling, edge AI and more can apply to the NVIDIA Academic Grant Program.

See notice regarding software product information.
LINK: https://blogs.nvidia.com/blog/neurips-2024-research/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

06/09/2026

Dolby and MagentaTV Bring Fans Closer to the FIFA World Cup 2026 in Germany with Dolby Vision and Dolby Atmos

June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...

04/08/2026

Dalet Announces Commercial Availability of Dalia, Bringing Media-Aware Agentic AI to Enterprise Productions

Dalet, a leading technology and service provider for media-rich organizations, t...

04/07/2026

Detective Conan: Fallen Angel of the Highway Opens in Dolby Cinemas Across Japan, Presented in Dolby Atmos and Dolby ...

April 7 2026, 19:00 (PDT) Detective Conan: Fallen Angel of the Highway Opens in...

24/06/2026

NoiseWorks Audio add Mouth De-Click to VoiceAssist

Plus: VoiceAssist Basic now available to UA LUNA users NoiseWorks Audio have just released an update that adds a new Mouth De-Click module to the Advanced t...

24/06/2026

Gator introduce the Frameworks Studio Mic Boom 2000

New heavy-duty mic stand joins range The latest arrival to Gator's Frameworks family introduces a new heavy-duty boom stand that's been designed for...

24/06/2026

Waves V17 now available

Latest major plug-in update goes live Waves have just announced that the latest major update for their hugely popular plug-in range is now officially availa...

24/06/2026

Why Four Bars of Signal Doesn't Always Mean Good Performance

When assessing cellular coverage, many people look at the signal bars displayed on a smartphone, router or modem. More bars are often assumed to mean better per...

24/06/2026

Rohde & Schwarz THORIS sets new standard for counterUAS defense

Rohde & Schwarz THORIS sets new standard for counter UAS defense At Eurosatory 2026, Rohde & Schwarz is unveiling THORIS, a German engineered, sovereign count...

24/06/2026

Rohde & Schwarz expands voice communications modernization program for Egyptian air traffic control

Rohde & Schwarz expands voice communications modernization program for Egyptian ...

24/06/2026

Clear-Com FreeSpeak Cell Successfully Tested by RTL Deutschland in 5G Network at...

eds3_5_jq(document).ready(function($) { $(#eds_sliderM519).chameleonSlider_2_1({...

24/06/2026

Nielsen's Q1 2026 Ad Supported Gauge

Streaming sets record high of 46.6% of ad supported TV viewing, driven by Super Bowl and Winter Olympics; overall share of ad supported TV remains steady NEW Y...

24/06/2026

FCC Flooded with Nearly 28K Comments Regarding Its Probe of 'The View'

Share Copy link Facebook X Linkedin Bluesky Email...

24/06/2026

Hearst Television Brings Ad Addressability to Local Broadcast TV

Share Copy link Facebook X Linkedin Bluesky Email...

24/06/2026

FCC Raises $3.5 Billion in AWS-3 Wireless Auction

Share Copy link Facebook X Linkedin Bluesky Email...

24/06/2026

RE:Vision Effects Announces Twixtor Standalone v 8.1 and a Sale!

RE:Vision Effects Announces Twixtor Standalone v 8.1 and a Sale! Brie Clayton June 24, 2026 0 Comments Twixtor v8.1 Standalone adds support for variab...

24/06/2026

Dreamtek Uses Full Blackmagic Workflow for Vercel Next JS Event

Dreamtek Uses Full Blackmagic Workflow for Vercel Next JS Event Brie Clayton June 24, 2026 0 Comments Blackmagic cameras, switchers, routers, recorder...

24/06/2026

Chyron LIVE Unveils New Features: Haivision StreamHub Integration, SCTE-35 Ad Insertion, and Refined Switching Tools

Chyron LIVE Unveils New Features: Haivision StreamHub Integration, SCTE-35 Ad In...

24/06/2026

Mapping an Education

Mapping an Education How composer Chloe Clarke Smith navigated her Boston Conservatory experience and brought new meaning to her work June 24, 2026 By Sara...

24/06/2026

The Next Act

The Next Act Dean Krisha Marcano's vision for a connected Theater Division, and the fund making it possible June 24, 2026 Photo by Eric Antoniou The Or...

24/06/2026

Announcing STAGES Magazine 2026

Announcing STAGES Magazine 2026 Marking a decade since Boston Conservatory and Berklee College of Music joined forces, this issue spotlights some of the groun...

24/06/2026

Rede Legislativa Chooses Appear to Support Brazil TV Ver...

In Brazil's TV 3.0 Trials, Appear's X5 is transporting live signals from Bras lia to S o Paulo over the public internet using secure, reliable next-gene...

24/06/2026

Mediaproxy partners with HVS for US broadcast market

Melbourne, Australia - 24 June 2026: Mediaproxy, the global standard for software-based IP compliance monitoring and multiviewing solutions, has named Heartland...

24/06/2026

Gray Media Launches Political 360 Digital Advertising Solution

Share Copy link Facebook X Linkedin Bluesky Email...

24/06/2026

Walmart to Pay $1.4 Billion to Acquire Ad Tech Firm Vibe.co

Share Copy link Facebook X Linkedin Bluesky Email...

24/06/2026

FCC Flooded with Nearly 28K Comments on 'The View'

Share Copy link Facebook X Linkedin Bluesky Email...

24/06/2026

First Rush Brings SDI Multicam ProRes Recording to Apple Silicon Macs

First Rush Brings SDI Multicam ProRes Recording to Apple Silicon Macs Brie Clayton June 23, 2026 0 Comments First Rush is a native macOS application d...

24/06/2026

Vertical Drama Beneath Crimson Sails Created with Blackmagic Design

Vertical Drama Beneath Crimson Sails Created with Blackmagic Design Brie Clayton June 23, 2026 0 Comments Thunder Child Productions relies on cameras&...

24/06/2026

Sky reveals wild new trailer and images for Shaun the Sheep: The Beast of Mossy Bottom

Wednesday 24 June 2026 Sky reveals wild new trailer and images for Shaun the Sh...

24/06/2026

Seven paradoxes shaping the next era of media production - Episode 1

Why Dynamic Media Facilities Matter In this series, we explore the technologies, architectures and operational realities shaping modern media operations. Along ...

24/06/2026

Comscore Announces Partnership with Amazon DSP to Expand Content Addressability to Drive Campaign Performance

Comscore Announces Partnership with Amazon DSP to Expand Content Addressability ...

24/06/2026

RT is supporting 21 Arts and Cultural Events all over Ireland this July

From Cork to Limerick to Earagail: RT is supporting 21 Arts and Cultural Events all over Ireland this July RT Supporting the Arts is delighted to spotlight...

23/06/2026

Case Study: YES Networks IP Transition Expands Production Possibilities and Redefines Workflows

When we began planning our transition from an SDI-based infrastructure to a new ...

23/06/2026

Imagine Communications Appoints Greg Garmon as SVP, Americas Video Sales

Imagine Communications has announced the appointment of Greg Garmon as Senior Vice President, Americas Video Sales. Garmon will oversee account growth and busin...

23/06/2026

Snap Promotes Emma Wakely to Head of Sports and Media Partnerships, Americas

Snap has promoted Emma Wakely to Head of Sports and Media Partnerships, Americas, succeeding Anmol Malhotra, who has been elevated to Global Head of Content and...

23/06/2026

YES Network and Gotham Sports App to Air MI New York Major League Cricket Matches

YES Network and The Gotham Sports App will air MI New York's Major League Cr...

23/06/2026

HAND Issues Persistent Digital IDs to 2026 NBA Draft Class

The Universal Talent Identifier (HAND) has issued HAND IDs to 34 top projected prospects in the 2026 NBA Draft class, including AJ Dybantsa, Cameron Boozer, and...

23/06/2026

World Boxing Launches World Boxing TV Streaming Platform

World Boxing has announced the launch of World Boxing TV, a subscription-based streaming platform built on the Joymo platform, offering live events, on-demand c...

23/06/2026

FloRacing to Stream 32 Off-Road Motorcycle Racing Events Including AMA Amateur National Motocross Championship

FloSports will stream 32 off-road motorcycle racing events on FloRacing, includi...

23/06/2026

SES Adds 14 Regional Channels and New Set-Top Boxes to ASTRA TV in Spain

SES has announced the expansion of its ASTRA TV platform in Spain with the addition of 14 regional channels in HD and UHD quality and the launch of new hybrid s...

23/06/2026

Appear Supports Rede Legislativas Contribution Workflow for Brazils TV 3.0 Trials

Appear ASA has announced its role in Rede Legislativa de R dio e TV's contri...

23/06/2026

PBS Selects LTN for Nationwide IP Video Network Across 330 Member Stations

LTN has announced that PBS has selected it as its IP video partner to modernize content distribution and contribution across more than 330 public television sta...

23/06/2026

Ease Live Powers Interactive Experience on Rally.TV for WRC

Ease Live has announced that its graphics overlay platform is powering an interactive fan experience on Rally.TV, the official streaming platform of the FIA Wor...

23/06/2026

Chyron LIVE Adds Haivision StreamHub Integration, SCTE-35 Ad Insertion, and Switcher Updates

Chyron has announced updates to Chyron LIVE, its cloud-native live production pl...

23/06/2026

ESPN Announces ESPN Fan House, Fan Engagement Hub Powered by Flowcode

ESPN has announced ESPN Fan House, a fan engagement hub powered by Flowcode, launching in August ahead of the 2026 college football season. Publicis Sports will...

23/06/2026

Sennheiser Relocates Americas Regional Hub to Nashville

The city's solid position in broadcast, entertainment, and sports attracted the major microphone manufacturer Sennheiser Group is moving its Americas Regio...

23/06/2026

Violet Audio's dMix 128 now shipping

128 channels of signal routing & DSP Announced just before the NAMM Show 2026, Violet Audio's latest digital audio matrix offers 128 channels of signal ...

23/06/2026

Minimal Audio launch Memory Rites

Latest Current expansion created by EPROM Minimal Audio have just launched the latest Current Expansion, Memory Rites. Designed in collaboration with renown...

23/06/2026

Undertone Audio's MPEQ-1 goes virtual

Popular hardware EQ gets official plug-in emulation Undertone Audio have just launched a new plug-in that brings one of their most popular hardware designs ...

23/06/2026

Colorfront Support for AWS CDI Unlocks New Potential for Cloud-Based Visual Effects and Content Creation

December 7, 2022 Colorfront (colorfront.com) - the multi-award-winning develope...