Sony Pixel Power calrec Sony

Crowning Achievement: NVIDIA Research Model Enables Fast, Efficient Dynamic Scene Reconstruction

09/12/2024

Content streaming and engagement are entering a new dimension with QUEEN, an AI model by NVIDIA Research and the University of Maryland that makes it possible to stream free-viewpoint video, which lets viewers experience a 3D scene from any angle.

QUEEN could be used to build immersive streaming applications that teach skills like cooking, put sports fans on the field to watch their favorite teams play from any angle, or bring an extra level of depth to video conferencing in the workplace. It could also be used in industrial environments to help teleoperate robots in a warehouse or a manufacturing plant.

The model will be presented at NeurIPS, the annual conference for AI research that begins Tuesday, Dec. 10, in Vancouver.

To stream free-viewpoint videos in near real time, we must simultaneously reconstruct and compress the 3D scene, said Shalini De Mello, director of research and a distinguished research scientist at NVIDIA. QUEEN balances factors including compression rate, visual quality, encoding time and rendering time to create an optimized pipeline that sets a new standard for visual quality and streamability.

Reduce, Reuse and Recycle for Efficient Streaming Free-viewpoint videos are typically created using video footage captured from different camera angles, like a multicamera film studio setup, a set of security cameras in a warehouse or a system of videoconferencing cameras in an office.

Prior AI methods for generating free-viewpoint videos either took too much memory for livestreaming or sacrificed visual quality for smaller file sizes. QUEEN balances both to deliver high-quality visuals - even in dynamic scenes featuring sparks, flames or furry animals - that can be easily transmitted from a host server to a client's device. It also renders visuals faster than previous methods, supporting streaming use cases.

In most real-world environments, many elements of a scene stay static. In a video, that means a large share of pixels don't change from one frame to another. To save computation time, QUEEN tracks and reuses renders of these static regions - focusing instead on reconstructing the content that changes over time.

Using an NVIDIA Tensor Core GPU, the researchers evaluated QUEEN's performance on several benchmarks and found the model outperformed state-of-the-art methods for online free-viewpoint video on a range of metrics. Given 2D videos of the same scene captured from different angles, it typically takes under five seconds of training time to render free-viewpoint videos at around 350 frames per second.

This combination of speed and visual quality can support media broadcasts of concerts and sports games by offering immersive virtual reality experiences or instant replays of key moments in a competition.

In warehouse settings, robot operators could use QUEEN to better gauge depth when maneuvering physical objects. And in a videoconferencing application - such as the 3D videoconferencing demo shown at SIGGRAPH and NVIDIA GTC - it could help presenters demonstrate tasks like cooking or origami while letting viewers pick the visual angle that best supports their learning.

The code for QUEEN will soon be released as open source and shared on the project page.

QUEEN is one of over 50 NVIDIA-authored NeurIPS posters and papers that feature groundbreaking AI research with potential applications in fields including simulation, robotics and healthcare. Among them is a best paper runner-up, titled Guiding a Diffusion Model With a Bad Version of Itself - one of just five papers recognized among the over 4,500 accepted this year.

Generative Adversarial Nets, the paper that first introduced GAN models, won the NeurIPS 2024 Test of Time Award. Cited more than 85,000 times, the paper was coauthored by Bing Xu, distinguished engineer at NVIDIA. Hear more from its lead author, Ian Goodfellow, research scientist at DeepMind, on the AI Podcast:

The AI Podcast Ep. 25: Google's Ian Goodfellow on How an Argument in a Bar Led to Generative Adversarial Networks

Learn more about NVIDIA Research at NeurIPS.

See the latest work from NVIDIA Research, which has hundreds of scientists and engineers worldwide, with teams focused on topics including AI, computer graphics, computer vision, self-driving cars and robotics.

Academic researchers working on large language models, simulation and modeling, edge AI and more can apply to the NVIDIA Academic Grant Program.

See notice regarding software product information.
LINK: https://blogs.nvidia.com/blog/neurips-2024-research/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

18/11/2026

The National Film and Video Foundation (NFVF) Call for Festival and Market Hosting Funding Applications is Open

The National Film and Video Foundation (NFVF), an agency of the Department of Sp...

18/11/2026

DIRECTV teams with Gracenote to enhance the sports programming experiences that fans want

Strategic renewal builds on existing engagement, supports DIRECTV's objectiv...

18/11/2026

SMPTE Publishes Engineering Report on Content Authenticity

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Senate Commerce Committee Approves Thumann Severs for FCC Commissioner

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

wTVision Qualifies NEOiD PTZ Cameras for Virtual Sets, AR

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Spanish Media Group Deploys Lawo IP Audio Infrastructure

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Library of American Broadcasting Foundation Honors 2026 Giants of Broadcasting & Electronic Arts

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , er...

18/11/2026

Monthly Subscription Streaming Service Prices Up Just 8% Since 2022

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Spectrum News Streamlines Workflow With Amagi

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Media Links Joins Panasonic Kairos Partner Program

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Signiant Acquires Burnlab's Ceivo AI-Driven Media Platform

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Democrats Urge Delay of Thumann Severs FCC Confirmation

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Hisong Unveils AirStudio Muse Audio Capture System

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Spectrum Reach Taps Debbie Presser to Lead Sales at AdLink New York

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

Nationwide EAS Test Runs Without Apparent Hiccups

Share {alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...

18/11/2026

SMPTE Identifies Key Requirements for Content Provenance...

SMPTE, the home for media professionals, technologists and engineers, today announced the publication of an engineering report with the summarized results of an...

18/11/2026

TeleData Accelerates XGS-PON Service Rollout in Germany and Unifies its Network with Harmonic's cOS Broadband Platform

Key Takeaways TeleData is building a unified DOCSIS and fiber network with Ha...

18/11/2026

Beeble launches SwitchX 3.0 with dynamic environments and greater creative control

Beeble recently launched SwitchX 3.0, a major update to its AI-powered video-to-...

18/11/2026

Film London to lead new industry PhD program in partnership with Brunel University

Film London will lead and manage a long-term industry research program in partne...

18/11/2026

Hexagon launches MANTIS to capture 3D scene data as cameras roll

Hexagon, the global leader in measurement technologies, recently launched MANTIS by Leica Geosystems, a camera-mounted 3D capture and tracking system to give fi...

18/11/2026

Successfully achieved BSI C5 attestation

Arvato Systems Expands BSI-C5 Testing Greater Confidence in Cloud Security and Regulatory Resilience Successful BSI-C5 Retest Certification in accordance wi...

18/11/2026

RT to premiere new three-part documentary series Able

Brendan O'Connor joins six disabled job seekers as they navigate the barriers of finding work and challenge perceptions of disability in the workplace Begi...

18/11/2026

When writing becomes a weapon : New true-crime podcast series to reveal the impact of anonymous poison-pen letter case

Poison Ink begins Friday 20 November 2026 on RT Listen and RT Radio 1 Listen ...

18/11/2026

RT Choice Music Prize 2027 Key Dates Announcement

RT Choice Music Prize In association with IMRO and IRMA 2027 KEY DATES Irish Album of the Year 2026 Shortlist 18th January Irish Artist of the Year 2026 ...

18/11/2026

NVIDIA RTX Spark Compact Desktops Out Now

New NVIDIA RTX Spark compact desktops from ASUS, Dell and MSI are available today, bringing local AI agents, image and video generation, and model fine-tuning t...

17/11/2026

Fabric Insights and Fonn Group Partner to Connect Newsroom Planning and Media Operations

Fabric Insights and Fonn Group have announced a partnership to connect editorial...

17/11/2026

NABJ Online Seminar Highlights Black Sports Production Legends on Nov. 25

The National Association of Black Journalists is offering up a Thanksgiving Eve treat on Wednesday, Nov. 25, in the form of a live streaming event featuring for...

17/11/2026

Bridge Technologies Adds Apple ProRes Support to VB440 Production Probe

Bridge Technologies has announced that its VB440 production probe now supports both monitoring and signal generation for Apple ProRes video streams. The additio...

17/11/2026

Canon U.S.A. Hosts Jericho High School Journalism Students at Melville Headquarters

Canon U.S.A., Inc. recently welcomed 21 Jericho High School students to its Melv...

17/11/2026

Amagi to Power Live Newscasts for Spectrum News

Spectrum News has deployed the Amagi platform to support its live news operations. Amagi says the platform lets Spectrum News streamline newsroom workflows, red...

17/11/2026

Microsoft Expands Haivision Video Contribution Deployment, Adds Makito ONE for Global Live Events

Microsoft has expanded its use of Haivision's live video contribution ecosys...

17/11/2026

Setanta Sports Deploys Ateme TITAN Live for OTT Sports Streaming

Setanta Sports has deployed Ateme's TITAN Live platform to support its OTT video delivery operations. Ateme's transcoding technology is integrated into ...

17/11/2026

Media Links Joins Panasonic's KAIROS Alliance Partner Program

Media Links has joined the KAIROS Alliance Partner program for Panasonic's KAIROS IT/IP live video production platform. The company completed interoperabili...

17/11/2026

Behind The Mic: Dan Hicks, Iconic NBC Sports Broadcaster, Passes Away at 64

Behind The Mic provides a roundup of recent news regarding on-air talent, including new deals, departures, and assignments compiled from press releases and repo...

17/11/2026

2026 Sports Broadcasting Hall of Fame: Jim Dove, an Innovator Offering Second Looks at the Greatest Plays in Sports

In live sports and in life, things go by in the blink of an eye. Savoring the mo...

17/11/2026

SVG Sit-Down: Eutelsat's Gonzalez, Carroccia on C-Band Clearing, LEO Contribution, the Future of Satellite in Sports

Satellite operators are adding a new generation of low-Earth-orbit (LEO) network...

17/11/2026

Clear-Com Delivers Portable Wireless Systems for Aircraft Communications Security Testing

Clear-Com has delivered five customized test systems for electromagnetic securit...

17/11/2026

Stephen Arnold Music Creates Sonic Identity for WTGL

Stephen Arnold Music (SAM) has created the sonic identity for WTGL, the women's team golf league from TMRW Sports, created in partnership with the LPGA and ...

17/11/2026

PAMA Introduces Give Sound a Voice Initiative to Promote the Value of Sound

The Professional Audio Manufacturers Alliance (PAMA) has introduced Give Sound a Voice, an initiative aimed at raising awareness of the value of sound among the...

17/11/2026

Fanatics Studios Names Jamal Henderson First President of Film & TV

Fanatics Studios, the sports entertainment joint venture between Fanatics and OBB, has named Jamal Henderson its first President of Film & TV, announced Michael...

17/11/2026

NFL Thanksgiving Week Halftime Shows to Feature U2, Wu-Tang Clan, Miranda Lambert and Goo Goo Dolls

The NFL's Thanksgiving Week games will feature halftime performances by U2, ...

17/11/2026

FanSided Launches Redesigned App with Live Scores and Personalized Feeds

FanSided, a Minute brand with more than 250 fan communities, has launched a redesigned mobile app that combines team news, analysis, live game coverage, stats a...

17/11/2026

Major League Volleyball Announces 2027 Media Lineup with NBC Sports, ESPN, ION and CBS Sports

Major League Volleyball (MLV), the longest-running indoor professional volleybal...

17/11/2026

Prime Video to Present Black Friday Sports Lineup in 200+ Countries, No Subscription Required

Prime Video will present a 15-hour slate of live sports on Black Friday, Nov. 27...

17/11/2026

DIOR X SUNDANCE INSTITUTE PARTNERSHIP ANNOUNCED AHEAD OF 2027 SUNDANCE FILM FESTIVAL

Catherine Dior was more than a muse. A Resistance fighter, a survivor of deporta...

17/11/2026

Freqport introduce FreqLift plug-in

Multiband dynamics plug-in boasts FT1 integration Freqport's latest plug-in combines multiband upward and downward dynamics with the company's renow...

17/11/2026

Studio 5X: Debut monitors from Teufel Audio

Hi-fi experts launch two pro-audio models Berlin-based hi-fi experts Teufel Audio have recently branched out into the pro-audio world with their first pairs...

17/11/2026

EastWest Black Friday 2026

A year of ComposerCloud+ for $99 EastWest's annual Black Friday Sale is already live, and new users are being offered one year of the all-encompassing C...

17/11/2026

SOS Awards: Make Your Vote Count!

Pick your favourite gear from 2026 The annual SOS Awards are now underway, and as always, we're asking readers to help us choose the best pro-audio rele...