Sony Pixel Power calrec Sony

Crowning Achievement: NVIDIA Research Model Enables Fast, Efficient Dynamic Scene Reconstruction

09/12/2024

Content streaming and engagement are entering a new dimension with QUEEN, an AI model by NVIDIA Research and the University of Maryland that makes it possible to stream free-viewpoint video, which lets viewers experience a 3D scene from any angle.

QUEEN could be used to build immersive streaming applications that teach skills like cooking, put sports fans on the field to watch their favorite teams play from any angle, or bring an extra level of depth to video conferencing in the workplace. It could also be used in industrial environments to help teleoperate robots in a warehouse or a manufacturing plant.

The model will be presented at NeurIPS, the annual conference for AI research that begins Tuesday, Dec. 10, in Vancouver.

To stream free-viewpoint videos in near real time, we must simultaneously reconstruct and compress the 3D scene, said Shalini De Mello, director of research and a distinguished research scientist at NVIDIA. QUEEN balances factors including compression rate, visual quality, encoding time and rendering time to create an optimized pipeline that sets a new standard for visual quality and streamability.

Reduce, Reuse and Recycle for Efficient Streaming Free-viewpoint videos are typically created using video footage captured from different camera angles, like a multicamera film studio setup, a set of security cameras in a warehouse or a system of videoconferencing cameras in an office.

Prior AI methods for generating free-viewpoint videos either took too much memory for livestreaming or sacrificed visual quality for smaller file sizes. QUEEN balances both to deliver high-quality visuals - even in dynamic scenes featuring sparks, flames or furry animals - that can be easily transmitted from a host server to a client's device. It also renders visuals faster than previous methods, supporting streaming use cases.

In most real-world environments, many elements of a scene stay static. In a video, that means a large share of pixels don't change from one frame to another. To save computation time, QUEEN tracks and reuses renders of these static regions - focusing instead on reconstructing the content that changes over time.

Using an NVIDIA Tensor Core GPU, the researchers evaluated QUEEN's performance on several benchmarks and found the model outperformed state-of-the-art methods for online free-viewpoint video on a range of metrics. Given 2D videos of the same scene captured from different angles, it typically takes under five seconds of training time to render free-viewpoint videos at around 350 frames per second.

This combination of speed and visual quality can support media broadcasts of concerts and sports games by offering immersive virtual reality experiences or instant replays of key moments in a competition.

In warehouse settings, robot operators could use QUEEN to better gauge depth when maneuvering physical objects. And in a videoconferencing application - such as the 3D videoconferencing demo shown at SIGGRAPH and NVIDIA GTC - it could help presenters demonstrate tasks like cooking or origami while letting viewers pick the visual angle that best supports their learning.

The code for QUEEN will soon be released as open source and shared on the project page.

QUEEN is one of over 50 NVIDIA-authored NeurIPS posters and papers that feature groundbreaking AI research with potential applications in fields including simulation, robotics and healthcare. Among them is a best paper runner-up, titled Guiding a Diffusion Model With a Bad Version of Itself - one of just five papers recognized among the over 4,500 accepted this year.

Generative Adversarial Nets, the paper that first introduced GAN models, won the NeurIPS 2024 Test of Time Award. Cited more than 85,000 times, the paper was coauthored by Bing Xu, distinguished engineer at NVIDIA. Hear more from its lead author, Ian Goodfellow, research scientist at DeepMind, on the AI Podcast:

The AI Podcast Ep. 25: Google's Ian Goodfellow on How an Argument in a Bar Led to Generative Adversarial Networks

Learn more about NVIDIA Research at NeurIPS.

See the latest work from NVIDIA Research, which has hundreds of scientists and engineers worldwide, with teams focused on topics including AI, computer graphics, computer vision, self-driving cars and robotics.

Academic researchers working on large language models, simulation and modeling, edge AI and more can apply to the NVIDIA Academic Grant Program.

See notice regarding software product information.
LINK: https://blogs.nvidia.com/blog/neurips-2024-research/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

07/10/2026

Dalet Flex LTS Delivers Smarter Media Operations from Ingest to Distribution

Dalet, a leading technology and service provider for media-rich organizations, today announced the latest Long-Term Supported (LTS) release of Dalet Flex. Build...

06/09/2026

Dolby and MagentaTV Bring Fans Closer to the FIFA World Cup 2026 in Germany with Dolby Vision and Dolby Atmos

June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...

04/08/2026

Dalet Announces Commercial Availability of Dalia, Bringing Media-Aware Agentic AI to Enterprise Productions

Dalet, a leading technology and service provider for media-rich organizations, t...

31/07/2026

SVG Sit-Down: NHL's Steve Mayer on Building a Centralized Production Operation From Scratch

With an emphasis on local, the league is looking beyond the initial four teams t...

31/07/2026

Reflecting on the 2026 Summer Season of Our Sundance Institute Labs

The 2026 Sundance Institute Directors Lab fellows in Estes Park, Colorado (Photo by Gabe Rovick) Dear Friends, This has been an extraordinary year for our ar...

31/07/2026

Audiocube launch Audiocube Space

Offers 360-degree sound placement in any DAW Audiocube have just announced the launch of a new plug-in that embeds a complete 3D acoustic environment inside...

31/07/2026

Sonarworks expand SoundID VoiceAI library

10 new Chinese C-Pop voices introduced While Sonarworks are best known to many for their room-correction software, their product range also includes an inno...

31/07/2026

PSPaudioware release PSP BBDelay

Reimagines the vintage BBD sound character PSPaudioware have now officially launched the new delay plug-in that they were previewing at GearExpo UK. Said to...

31/07/2026

Savannah Bananas Hits a Broadcast Home Run with Calrec

The Savannah Bananas has built its reputation on refusing to play baseball by traditional rules, and the team's production arm Banana Ball TV (BTV) has draf...

31/07/2026

How Valuable Is TV to Smaller, Independent Cable Operators?

Share Copy link Facebook X Linkedin Bluesky Email...

31/07/2026

Early Registration Opens for the 2026 SMPTE Media Techno...

SMPTE, the home for media professionals, technologists and engineers, today opened early registration for the 2026 Media Technology Summit (MTS) and announced i...

31/07/2026

Central Christian Church Connects Remote Production Teams...

Clear-Com announced that Central Christian Church has transformed communication across its live production operations by implementing Clear-Com's LQ Seri...

31/07/2026

Haivision To Feature Makito ONE, Falkon X4 At IBC 2026

Share Copy link Facebook X Linkedin Bluesky Email...

31/07/2026

ABC Stations Tout Community Support for License Renewals

Share Copy link Facebook X Linkedin Bluesky Email...

31/07/2026

Finalists Announced for IBC2026 Innovation Awards

Share Copy link Facebook X Linkedin Bluesky Email...

31/07/2026

IBC2026 Innovation Awards finalists unveiled honouring co...

IBC has unveiled the finalists for the IBC2026 Innovation Awards, recognising collaborative projects from around the world that solve real-world challenges and ...

31/07/2026

From focus puller to colourist

From focus puller to colourist Caroline Shawley July 30, 2026 0 Comments Ana Mar a Ormaza shares her story of instinct, storytelling and finding a cre...

31/07/2026

Amazon Prime Documentary Andata e Ritorno Relies on Blackmagic Design

Amazon Prime Documentary Andata e Ritorno Relies on Blackmagic Design Brie Clayton July 30, 2026 0 Comments DaVinci Resolve Studio and Blackmagic Clou...

31/07/2026

Krotos Launches Video to Sound Plugin for DaVinci Resolve

Krotos Launches Video to Sound Plugin for DaVinci Resolve Brie Clayton July 30, 2026 0 Comments AI-assisted workflow helps editors add synchronized, p...

31/07/2026

Half-Year Report on SES's Liquidity Contract

Luxembourg, July 31, 2026 - Pursuant to the liquidity contract entered into by SES with BNP Paribas as of 7 April 2026, please see the below update on the progr...

31/07/2026

SES Publishes FY2025 EU Taxonomy Restatement

Luxembourg, July 31, 2026 - SES today published a restatement of its FY2025 EU Taxonomy (Article 8) disclosure. The restatement updates the FY2025 EU Taxonomy d...

31/07/2026

Mercedes-Benz Ireland returning as the official car partner of The Traitors Ireland

RT Commercial has today announced Mercedes-Benz Ireland will return as the offi...

31/07/2026

RT Supporting the Arts is supporting 15 cultural events nationwide this August

RT Supporting the Arts is delighted to spotlight a diverse range of arts, culture and heritage events taking place across Ireland this August. From major festi...

30/07/2026

CP Communications Provides Broadcast Support for 2026 MLB All-Star Week in Philadelphia

CP Communications provided RF audio, RF video, communications, RF coordination, ...

30/07/2026

OpenDrives Named to 2026 CRN Storage 100 List

OpenDrives has been included in CRN's 2026 Storage 100 list in the Software-Defined Storage category. The annual list, selected by the CRN editorial team, r...

30/07/2026

SES Selected by LATAM Airlines for Multi-Orbit Satellite Connectivity

LATAM Airlines has selected SES to provide multi-orbit inflight connectivity to its fleet of Airbus and Embraer aircraft. More than 60 aircraft - including Airb...

30/07/2026

Tagboard Launches Producer API with Elgato Stream Deck Integration

Tagboard has launched the Producer API, an open interface that allows key commands in Tagboard's live production environment to be triggered from external d...

30/07/2026

FOX Sports Adds Del Mar to Horse Racing Coverage in Multi-Year Deal

FOX Sports and the New York Racing Association (NYRA) have announced a multi-year agreement with Del Mar Thoroughbred Club that will make FOX Sports the exclusi...

30/07/2026

ABC Commercial Launches Four FAST Channels on LG Smart TVs

ABC Commercial has launched four free ad-supported streaming television (FAST) channels on LG Smart TVs across North America, Great Britain, and select countrie...

30/07/2026

Six Kings Slam Returns to Netflix in October with Sinner, Alcaraz, Djokovic

The Six Kings Slam exhibition tennis tournament will return to Riyadh on October 21, 22, and 24, streaming live on Netflix at no additional cost to subscribers....

30/07/2026

Central Christian Church Uses Clear-Coms Gen-IC to Connect Remote Production Teams

Central Christian Church in Mt. Vernon, Illinois has deployed Clear-Com's LQ...

30/07/2026

Blackmagic Design Releases UltraStudio Express 3G Capture and Playback Devices

Blackmagic Design has released the UltraStudio Express 3G family, a pair of USB4 capture and playback devices compatible with Mac, Windows, and Linux computers ...

30/07/2026

ESPN Secures Media Rights to Womens Pro Baseball League

ESPN has reached a media rights agreement with the Women's Pro Baseball League (WPBL), making ESPN the national streaming home of the league's 2026 seas...

30/07/2026

Most Valuable Promotions and Professional Fighters League Merge

Most Valuable Promotions (MVP) and the Professional Fighters League (PFL) have announced a merger that will operate under the MVP banner. PFL CEO John Martin wi...

30/07/2026

Columbus Blue Jackets To Simulcast TV and Radio Broadcasts Beginning 2026-27

The Columbus Blue Jackets will simulcast all game broadcasts across television and radio beginning with the 2026-27 NHL season. Under the new format, the televi...

30/07/2026

TMRW Sports Selects Populous To Design Professional Flag Football Stadium

TMRW Sports has selected Populous as architect for a purpose-built stadium for its professional flag football league, being developed in partnership with the NF...

30/07/2026

SiriusXM Launches Sports Pass Subscription at $5 per Month

SiriusXM has announced SiriusXM Sports Pass, a new subscription plan launching September 1 that bundles the company's sports audio programming into a single...

30/07/2026

SVG Cloud & Content Workflows Summit Focuses on Live Production, Content Management, AI

Leagues, broadcasters, and technologists gather to tackle cloud economics, distr...

30/07/2026

AWSs Jason Dvorkin on the Power of AWS Elemental Inference, Agentic AI in Live Sports Production

A major player in the cloud-based solutions, Amazon Web Services (AWS) has conti...

30/07/2026

Minnesota Timberwolves Ink Local-Media-Rights Deal With DAZN

The NBA will provide production for all local game broadcasts, including pregame and postgame coverage...

30/07/2026

Cosm Detroit To Open September 10

Cosm will open its fourth immersive sports and entertainment venue in downtown Detroit on September 10. Located at 25 Cadillac Square, adjacent to Campus Martiu...

30/07/2026

NFL Network Added to ESPN App Beginning July 30

NFL Network will be available on the ESPN App beginning July 30 for Unlimited plan subscribers. The addition brings NFL Network's live games, studio program...

30/07/2026

NEP Platform Adds Riedels SimplyLive Production Suite

NEP Group has added Riedel's SimplyLive Production Suite to NEP Platform, giving customers access to the live production software through NEP's orchestr...

30/07/2026

SUNDANCE INSTITUTE NAMES TRISTA SCHROEDER GENERAL COUNSEL & CHIEF OPERATING OFFICER

LOS ANGELES, CA, July 30, 2026 - The nonprofit Sundance Institute today announce...

30/07/2026

Waves release Atlas Reverb

A new take on classic algorithmic reverbs Waves have just introduced a new flagship algorithmic reverb plug-in that offers the company's take on the ico...

30/07/2026

Arturia's Summer Sale now live

Save up to 30% on virtual instruments & plug-ins Arturia have recently launched their annual Summer Sale, which sees discounts of up to 30% applied across a...

30/07/2026

SSL unveil the Odyssey console

ActiveAnalogue technology powers new flagship modular console A year on from the launch of the Oracle, SSL have revealed a new flagship console which they s...

30/07/2026

DT 275 Pro from beyerdynamic

Up to 22dBA of passive attenuation The latest arrival to the beyerdynamic range introduces a new compact monitoring headphone that's been designed to ta...

30/07/2026

Apprenticeship success at SGL Carbon in Meitingen

As part of its traditional graduation ceremony, SGL Carbon honored the achievements of a total of twelve young professionals who successfully completed their ap...