Sony Pixel Power calrec Sony

Crowning Achievement: NVIDIA Research Model Enables Fast, Efficient Dynamic Scene Reconstruction

09/12/2024

Content streaming and engagement are entering a new dimension with QUEEN, an AI model by NVIDIA Research and the University of Maryland that makes it possible to stream free-viewpoint video, which lets viewers experience a 3D scene from any angle.

QUEEN could be used to build immersive streaming applications that teach skills like cooking, put sports fans on the field to watch their favorite teams play from any angle, or bring an extra level of depth to video conferencing in the workplace. It could also be used in industrial environments to help teleoperate robots in a warehouse or a manufacturing plant.

The model will be presented at NeurIPS, the annual conference for AI research that begins Tuesday, Dec. 10, in Vancouver.

To stream free-viewpoint videos in near real time, we must simultaneously reconstruct and compress the 3D scene, said Shalini De Mello, director of research and a distinguished research scientist at NVIDIA. QUEEN balances factors including compression rate, visual quality, encoding time and rendering time to create an optimized pipeline that sets a new standard for visual quality and streamability.

Reduce, Reuse and Recycle for Efficient Streaming Free-viewpoint videos are typically created using video footage captured from different camera angles, like a multicamera film studio setup, a set of security cameras in a warehouse or a system of videoconferencing cameras in an office.

Prior AI methods for generating free-viewpoint videos either took too much memory for livestreaming or sacrificed visual quality for smaller file sizes. QUEEN balances both to deliver high-quality visuals - even in dynamic scenes featuring sparks, flames or furry animals - that can be easily transmitted from a host server to a client's device. It also renders visuals faster than previous methods, supporting streaming use cases.

In most real-world environments, many elements of a scene stay static. In a video, that means a large share of pixels don't change from one frame to another. To save computation time, QUEEN tracks and reuses renders of these static regions - focusing instead on reconstructing the content that changes over time.

Using an NVIDIA Tensor Core GPU, the researchers evaluated QUEEN's performance on several benchmarks and found the model outperformed state-of-the-art methods for online free-viewpoint video on a range of metrics. Given 2D videos of the same scene captured from different angles, it typically takes under five seconds of training time to render free-viewpoint videos at around 350 frames per second.

This combination of speed and visual quality can support media broadcasts of concerts and sports games by offering immersive virtual reality experiences or instant replays of key moments in a competition.

In warehouse settings, robot operators could use QUEEN to better gauge depth when maneuvering physical objects. And in a videoconferencing application - such as the 3D videoconferencing demo shown at SIGGRAPH and NVIDIA GTC - it could help presenters demonstrate tasks like cooking or origami while letting viewers pick the visual angle that best supports their learning.

The code for QUEEN will soon be released as open source and shared on the project page.

QUEEN is one of over 50 NVIDIA-authored NeurIPS posters and papers that feature groundbreaking AI research with potential applications in fields including simulation, robotics and healthcare. Among them is a best paper runner-up, titled Guiding a Diffusion Model With a Bad Version of Itself - one of just five papers recognized among the over 4,500 accepted this year.

Generative Adversarial Nets, the paper that first introduced GAN models, won the NeurIPS 2024 Test of Time Award. Cited more than 85,000 times, the paper was coauthored by Bing Xu, distinguished engineer at NVIDIA. Hear more from its lead author, Ian Goodfellow, research scientist at DeepMind, on the AI Podcast:

The AI Podcast Ep. 25: Google's Ian Goodfellow on How an Argument in a Bar Led to Generative Adversarial Networks

Learn more about NVIDIA Research at NeurIPS.

See the latest work from NVIDIA Research, which has hundreds of scientists and engineers worldwide, with teams focused on topics including AI, computer graphics, computer vision, self-driving cars and robotics.

Academic researchers working on large language models, simulation and modeling, edge AI and more can apply to the NVIDIA Academic Grant Program.

See notice regarding software product information.
LINK: https://blogs.nvidia.com/blog/neurips-2024-research/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

07/10/2026

Dalet Flex LTS Delivers Smarter Media Operations from Ingest to Distribution

Dalet, a leading technology and service provider for media-rich organizations, today announced the latest Long-Term Supported (LTS) release of Dalet Flex. Build...

06/09/2026

Dolby and MagentaTV Bring Fans Closer to the FIFA World Cup 2026 in Germany with Dolby Vision and Dolby Atmos

June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...

05/08/2026

Thomson toolkit added to virtual hub built to strengthen climate storytelling

Thomson's Climate Crisis Toolkit for Media in Tanzania is being included in a new virtual library designed to ensure accurate and engaging storytelling arou...

05/08/2026

SVG Regional Sports Production Summit 2026: All Sessions Now Available to Watch on SVG PLAY

RSNs, teams, leagues, and streamers explore the present and future of local spor...

05/08/2026

SVG New Sponsor Spotlight: Hitachi Vantara's Lenny Khaitov on Building Resilient Data Infrastructure for Sports Production

As sports-production workflows generate larger volumes of unstructured data and ...

05/08/2026

NESN's Women's Celebration Night Returns With Expanded Focus on the Next Generation

Three aspiring broadcasters will contribute on-camera and inside the production ...

05/08/2026

Union County Paints an Authentic Picture of Recovery and Community

Adam Meeks attends the premiere of his film Union County, an official selection of the 2026 Sundance Film Festival. (Photo by Jemal Countess/Sundance Institut...

05/08/2026

Steinberg celebrate VST & HALion anniversaries

Major discounts, updates & remakes of early VST plug-ins Steinberg are currently celebrating two significant anniversaries, with this week marking the 30th ...

05/08/2026

KRK introduce the V Series Five

Flagship monitor range gains wireless remote control KRK's flagship studio monitor range has just been upgraded, with three new fifth-generation models ...

05/08/2026

Rohde & Schwarz launches next phase of Airways New Zealand Air Traffic Control system upgrade

Rohde & Schwarz launches next phase of Airways New Zealand Air Traffic Control s...

05/08/2026

Rohde & Schwarz presents the R&SMP007, a man-portable tactical direction finding system for seamless signal detection from 9 MHz to 8 GHz

Rohde & Schwarz presents the R&S MP007, a man-portable tactical direction findin...

05/08/2026

Australia's retirement sector advertising rises more than 10% as millions plan for life after work according to Nielsen

Advertisers invested $53.6 million in the retirement sector in the last 12 month...

05/08/2026

New Nielsen data shows IKEA's New Zealand launch attracted more than 243,000 shoppers in the last month

Nielsen CMI and Ad Intel data reveals who's visiting IKEA, where else they s...

05/08/2026

Nielsen expands Four-Screen Ad Deduplication across CTV in Japan

Media buyers and sellers can now compare deduplicated campaign reach across computer, mobile, connected TV and linear TV across all measurable CTV publishers in...

05/08/2026

Stranger Things,' Bluey,' The Pitt' Top Nielsen's Streaming Charts Through First Half of 2026

Stranger Things' Leads All Streaming Titles with 23.3 Billion Minutes in Fir...

05/08/2026

Live Media Group, Gravity Media Strike Deal on U.S. OB Fleets

Share Copy link Facebook X Linkedin Bluesky Email...

05/08/2026

Beyond VVC: Call for Proposals for Next-Gen Video Coding

Share Copy link Facebook X Linkedin Bluesky Email...

05/08/2026

QuickLink Expands North American Team With Appointment of...

QuickLink, the leading provider of award-winning video production and remote guest integration solutions, announces the appointment of Kane Peterson to its Nort...

05/08/2026

Seven in 10 UK advertisers exceed production budgets desp...

XR research reveals marketers believe they have control over production costs, yet budget overruns, campaign delays and creative cutbacks remain commonplace Se...

05/08/2026

Game On Live Studio Uses Blackmagic Design to Stream Tropical 7s

Game On Live Studio Uses Blackmagic Design to Stream Tropical 7s Brie Clayton August 4, 2026 0 Comments Every match of world's largest youth rugby...

05/08/2026

Chris Fitzgerald to Head TDC's Experiential Growth

Chris Fitzgerald to Head TDC's Experiential Growth Brie Clayton August 4, 2026 0 Comments Technical Direction Company promotes Chris Fitzgerald to...

05/08/2026

Gravity Media, Live Media Group Form Strategic Partnership

Share Copy link Facebook X Linkedin Bluesky Email...

05/08/2026

Samsung Launches World's First HDR10+ Streaming on Prime Video

Share Copy link Facebook X Linkedin Bluesky Email...

05/08/2026

Profuz Digital unveils new advanced search capabilities a...

Profuz Digital is set to launch a powerful upgrade to its Profuz LAPIS digital media management platform at IBC this year at Stand 8.A88. France Galop, Canal+ F...

05/08/2026

KRK Brings Advanced Wireless Control to Professional Moni...

KRK, a Gibson brand and leader in professional studio monitoring for nearly four decades, launches its new V Series Five Studio Monitors, the latest evolution o...

05/08/2026

Stable performance in line with plan - Magyar Telekoms H1 and Q2 2026 results

Network expansion, continued customer growth and stable financial performance characterized Magyar Telekoms business results for the first half and second quart...

05/08/2026

The Sky Foundation launches new bursary programme to support the future of UK dance

Wednesday 5 August 2026 The Sky Foundation launches new bursary programme to su...

05/08/2026

Charlie Sheen Launches Mostly Sheenius in Partnership with Red Seat Ventures

Charlie Sheen Launches Mostly Sheenius in Partnership with Red Seat Ventures After Decades as One of the World's Most Recognizable and Interviewed Celebr...

04/08/2026

MLB's Jason Hedgcock on Using the Cloud To Produce World Baseball Classic Games for Netflix Japan

Live production in the cloud is a growing trend across the industry, and Major L...

04/08/2026

Major League Soccer Names Larry Berg as Next Commissioner; Will Take Helm in 2027

LAFC co-managing owner will succeed Don Garber on Jan. 1, 2027, as league prepar...

04/08/2026

Foresight: Driving Growth Through Broadcast Flexibility - A Calrec + PGA TOUR Case Study

In 2022, when the PGA TOUR was breaking ground on its new, state-of-the-art broa...

04/08/2026

Dealer Cam Debuts as Omaha Productions Takes World Series of Poker Main Event Final Table Live on ESPN

Provided by POVORA, the new camera angle offers a dealer's-eye view of the c...

04/08/2026

USL To Trial FIFA's Football Video Support System During 2026 USL Cup

Spiideo will provide the camera setup, with NEP handling broadcast integration and off-site video operations...

04/08/2026

Game On Live Studio Deploys Blackmagic Design Workflow Across 15 Fields at Tropical 7s

Production team streamed more than 750 youth rugby matches using ATEM switchers,...

04/08/2026

NDI, KTC Partner To Develop Network-Native Commercial Displays

Agreement gives KTC early access to NDI's Advanced SDK for integration across commercial displays, digital signage, and interactive panels...

04/08/2026

IBC2026: IBC Innovation Awards Finalists Highlight Advances in AI, Cloud, Open Standards, and Accessibility

Twelve projects across four categories recognize collaborative work in productio...

04/08/2026

Ekstraklasa, Live Park Select Hawk-Eye for Tracking, Offside, and Broadcast Technology

Multi-year agreement will introduce SkeleTRACK, semi-automated offside, automate...

04/08/2026

Radio France Deploys Redundant Lawo AoIP Workflow for 2026 Concert de Paris

Three mc 56 consoles, dual A__UHD Core engines, RAVENNA networking, and HOME management supported live sound, monitoring, and international broadcast production...

04/08/2026

IBC2026: Haivision to Debut Makito ONE, Falkon X4 in Europe

New contribution products support ultra-low-latency video transport across fixed, remote, and mobile production environments...

04/08/2026

IBC2026: Chyron to Highlight AI, Cloud, HTML Graphics, and Sports-Analysis Tools

Company will showcase updates across PRIME, Chyron LIVE, Paint, virtual production, newsroom workflows, and venue control...

04/08/2026

TAB Show 2026: MultiDyne to Showcase Fiber-Transport and IP Infrastructure Tools

Exhibit will feature SilverBack, VersaBrix, VersaFrame, VF-9100, and FiberSaver-10G platforms...

04/08/2026

IBC2026: TAG Video Systems Adds MXL Support, Containerized Deployment, Expanded QC Tools

Platform updates target cloud-based production, high-density ST 2110 monitoring,...

04/08/2026

Gravity Media, Live Media Group Partner To Expand U.S. Mobile Production Facility Operations

Live Media Group will take control of Gravity Media's U.S. mobile-unit fleet...

04/08/2026

HeadRush update Prime, Core & Flex Core firmware

Amp-modelling rigs get major upgrade HeadRush have recently released a major firmware update that brings some powerful new features to their Prime, Core and...

04/08/2026

Am I Audio launch I Am Choir

Generate a choir from a single voice track Following years of development, Am I Audio have just officially launched their debut plug-in. Created in collabor...

04/08/2026

Universal Audio's EMT 140 & 250 plug-ins go native

Official EMT-endorsed reverbs now available to all Two more of Universal Audio's renowned UAD plug-ins have just gained new native versions that can be ...

04/08/2026

SBS helps Australians understand why every Census counts

SBS helps Australians understand why every Census counts Media releases As Australia prepares for the 2026 Census on Tuesday 11 August, SBS is helping ensur...

04/08/2026

Canada invests in the future of HF communications with Rohde & Schwarz solutions

Canada invests in the future of HF communications with Rohde & Schwarz solutions After selection by the Canadian Department of National Defence (DND) as the p...

04/08/2026

Blendkit Expands Beyond Blender

Blendkit Expands Beyond Blender Brie Clayton August 4, 2026 0 Comments Popular 3D Asset Library Launches Alpha Plugins for Maya, Godot, and Rhino Blend...