Sony Pixel Power calrec Sony

Crowning Achievement: NVIDIA Research Model Enables Fast, Efficient Dynamic Scene Reconstruction

09/12/2024

Content streaming and engagement are entering a new dimension with QUEEN, an AI model by NVIDIA Research and the University of Maryland that makes it possible to stream free-viewpoint video, which lets viewers experience a 3D scene from any angle.

QUEEN could be used to build immersive streaming applications that teach skills like cooking, put sports fans on the field to watch their favorite teams play from any angle, or bring an extra level of depth to video conferencing in the workplace. It could also be used in industrial environments to help teleoperate robots in a warehouse or a manufacturing plant.

The model will be presented at NeurIPS, the annual conference for AI research that begins Tuesday, Dec. 10, in Vancouver.

To stream free-viewpoint videos in near real time, we must simultaneously reconstruct and compress the 3D scene, said Shalini De Mello, director of research and a distinguished research scientist at NVIDIA. QUEEN balances factors including compression rate, visual quality, encoding time and rendering time to create an optimized pipeline that sets a new standard for visual quality and streamability.

Reduce, Reuse and Recycle for Efficient Streaming Free-viewpoint videos are typically created using video footage captured from different camera angles, like a multicamera film studio setup, a set of security cameras in a warehouse or a system of videoconferencing cameras in an office.

Prior AI methods for generating free-viewpoint videos either took too much memory for livestreaming or sacrificed visual quality for smaller file sizes. QUEEN balances both to deliver high-quality visuals - even in dynamic scenes featuring sparks, flames or furry animals - that can be easily transmitted from a host server to a client's device. It also renders visuals faster than previous methods, supporting streaming use cases.

In most real-world environments, many elements of a scene stay static. In a video, that means a large share of pixels don't change from one frame to another. To save computation time, QUEEN tracks and reuses renders of these static regions - focusing instead on reconstructing the content that changes over time.

Using an NVIDIA Tensor Core GPU, the researchers evaluated QUEEN's performance on several benchmarks and found the model outperformed state-of-the-art methods for online free-viewpoint video on a range of metrics. Given 2D videos of the same scene captured from different angles, it typically takes under five seconds of training time to render free-viewpoint videos at around 350 frames per second.

This combination of speed and visual quality can support media broadcasts of concerts and sports games by offering immersive virtual reality experiences or instant replays of key moments in a competition.

In warehouse settings, robot operators could use QUEEN to better gauge depth when maneuvering physical objects. And in a videoconferencing application - such as the 3D videoconferencing demo shown at SIGGRAPH and NVIDIA GTC - it could help presenters demonstrate tasks like cooking or origami while letting viewers pick the visual angle that best supports their learning.

The code for QUEEN will soon be released as open source and shared on the project page.

QUEEN is one of over 50 NVIDIA-authored NeurIPS posters and papers that feature groundbreaking AI research with potential applications in fields including simulation, robotics and healthcare. Among them is a best paper runner-up, titled Guiding a Diffusion Model With a Bad Version of Itself - one of just five papers recognized among the over 4,500 accepted this year.

Generative Adversarial Nets, the paper that first introduced GAN models, won the NeurIPS 2024 Test of Time Award. Cited more than 85,000 times, the paper was coauthored by Bing Xu, distinguished engineer at NVIDIA. Hear more from its lead author, Ian Goodfellow, research scientist at DeepMind, on the AI Podcast:

The AI Podcast Ep. 25: Google's Ian Goodfellow on How an Argument in a Bar Led to Generative Adversarial Networks

Learn more about NVIDIA Research at NeurIPS.

See the latest work from NVIDIA Research, which has hundreds of scientists and engineers worldwide, with teams focused on topics including AI, computer graphics, computer vision, self-driving cars and robotics.

Academic researchers working on large language models, simulation and modeling, edge AI and more can apply to the NVIDIA Academic Grant Program.

See notice regarding software product information.
LINK: https://blogs.nvidia.com/blog/neurips-2024-research/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

07/10/2026

Dalet Flex LTS Delivers Smarter Media Operations from Ingest to Distribution

Dalet, a leading technology and service provider for media-rich organizations, today announced the latest Long-Term Supported (LTS) release of Dalet Flex. Build...

08/09/2026

Orlando Magic, Gray Media Ink Full-Season Broadcast Deal

Share Copy link Facebook X Linkedin Bluesky Email...

07/09/2026

Get to Know This Fall and Winter's Filmmakers Through These 24 Sundance Institute-Supported Films

By Ramona Flume and Lucy Spicer There's a particular kind of magic that hap...

07/09/2026

HeadRush launch the FRFR Cab Series

Two new full-range, flat-response cabinets introduced HeadRush have recently introduced a pair of new full-range, flat-response cabinets designed for player...

07/09/2026

GameSoundCon 2026

Answering questions about AI in game audio GameSoundCon, the largest professional conference for video game music and sound design, have announced that they...

07/09/2026

DPA to launch the 6388 Core+

New directional headset mic on show at IBC Just ahead of this week's IBC show, DPA have announced the launch of a new directional headset that's bee...

07/09/2026

Accurate.Video rallies Iyuno to rework 35,000 media assets ahead of schedule

We are not building systems to replace human decision-making; we are building systems that execute our decisions at machine speed. The media companies that win ...

07/09/2026

Blackmagic Design Announces New Lower Price for Blackmagic PYXIS 6K and Blackmagic Cinema Camera 6K

Blackmagic Design Announces New Lower Price for Blackmagic PYXIS 6K and Blackmag...

07/09/2026

GB Labs to Unveil Core.ONE at IBC2026, an Intelligent Storage Platform for Media Teams

GB Labs to Unveil Core.ONE at IBC2026, an Intelligent Storage Platform for Media...

07/09/2026

Freddie Meredith joins cast of SNL UK ahead of start of Season 2

Monday 7 September 2026 Freddie Meredith joins cast of SNL UK ahead of start of Season 2 Freddie Meredith Images ZIP (3MB) Saturday Night Live UK announces ...

07/09/2026

Sky and NFTS team up to launch new routes into screen careers

Monday 7 September 2026 Sky and NFTS team up to launch new routes into screen careers Sky has today announced a new apprenticeship programme, with application...

07/09/2026

Sky Sports announces most comprehensive NFL coverage to date for the 2026/27 season

Monday 7 September 2026 Sky Sports announces most comprehensive NFL coverage to...

07/09/2026

Jnger Audio Showcases Industry-First S-ADM Cloud Workflow at IBC 2026

Dolby, MainConcept, Techex and J nger Audio will showcase the industry's first demonstration of Serial Audio Definition Model (S-ADM) in a cloud-based produ...

07/09/2026

dB Broadcast to Showcase APIS Hive at IBC 2026

dB Broadcast will showcase APIS Hive, its software platform for designing, deploying and managing modern broadcast infrastructures, at IBC 2026. Developed from...

07/09/2026

A KNOCKOUT 1.6 MILLION VIEWERS WATCHED KATIE TAYLOR'S FINAL BOUT FREE -TO -AIR ON RT2

552,000 streams on RT Player The historic fight, which saw Katie Taylor become...

06/09/2026

BS-203 MacroAcidizer from Born Second

Recreates Rolands TB-303 & MC-202 BS-203 MacroAcidizer is the first virtual instrument from Born Second, and it offers a recreation of two Roland classics -...

06/09/2026

PlayBox Technology Opens Design Partner Programme for New...

London, UK September 2026 PlayBox Technology is inviting broadcasters, OTT and FAST operators, content owners and media organisations worldwide to become De...

06/09/2026

Dolby and MagentaTV Bring Fans Closer to the FIFA World Cup 2026 in Germany with Dolby Vision and Dolby Atmos

June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...

05/09/2026

HEDD Audio update CTRL software

MK2 & Bass system control app improved HEDD Audio have just launched a new and improved version of the control software that pairs with their MK2 and Bass m...

05/09/2026

CALL FOR ENTRIES IS NOW OPEN FOR THE 20TH ANNUAL SOUTH AFRICAN FILM & TELEVISION AWARDS (SAFTAs20) EXTENDED

The National Film and Video Foundation (NFVF), an agency of the Department of Sp...

05/09/2026

Orlando Magic, Gray Media Ink Ful-Season Broadcast Deal

Share Copy link Facebook X Linkedin Bluesky Email...

05/09/2026

YouTube TV Adds ESPN Unlimited

Share Copy link Facebook X Linkedin Bluesky Email...

05/09/2026

InSync Brings Conversion Workflow Flexibility to IBC 2026

InSync Technology, the leading provider of premium video conversion solutions, will showcase a new generation of products at IBC 2026, designed to help broadcas...

05/09/2026

GlobalM supports quattro media and World Rowing with Glob...

GlobalM successfully supported quattro media GmbH and World Rowing, the world governing body of the Olympic sport of Rowing, with the international distribution...

05/09/2026

Scripps Research professor Chi-Huey Wong to receive ACS Alfred Bader Award

LA JOLLA, CA-In recognition of his exceptional work to understand the chemistry of carbohydrates and the role they play in health and disease, Scripps Family Ch...

04/09/2026

Business Innovation Synergizer FAQ: What applicants need to know

Media organisations interested in the Media Innovation Europe 3.0 Business Innovation Synergizer (BIS) can now consult a new FAQ document providing additional a...

04/09/2026

IBC 2026: Eluvio Showcases Global Broadcast Fabric Updates

Eluvio will demonstrate updates to its Content Fabric platform at IBC2026 (Hall 8, Stand 8.MS5, and Future Tech Zone, Hall 14, Stand 14.A58, September 11-14). C...

04/09/2026

IBC 2026: LiveU Joins IBC Accelerator Smart Stories Consortium

LiveU has joined the IBC Accelerator's Smart Stories consortium, a multi-vendor initiative developing an open standard for Story Context interoperability in...

04/09/2026

IBC 2026: Telos Alliance to Demonstrate Audio Observability Capabilities

Telos Alliance will demonstrate new audio observability capabilities at IBC2026 (Stand 8.D37, RAI Amsterdam, September 11-14), covering monitoring of audio heal...

04/09/2026

World Team Tennis Announces Broadcast Partnership with Versants USA Sports

World Team Tennis (WTT) has announced a broadcast partnership with USA Sports, the sports division of Versant. USA Sports will air all six of WTT's live mat...

04/09/2026

Gravity Media Appoints Alex Pannell as Chief Business Officer

Gravity Media has appointed Alex Pannell as Chief Business Officer. Pannell joins from Appear, where he served as Chief Commercial Officer. He will be based in ...

04/09/2026

IBC 2026: Dalet Expands Dalia AI Platform with Agentic Workflow Capabilities

By Carrie Bowden Staff Writer Dalet has announced new agentic workflow capabilities for Dalia, its AI platform for media organizations, ahead of IBC2026. The ...

04/09/2026

IBC 2026: L-Acoustics Achieves EN 54 Certification Across Loudspeaker and Amplifier Portfolio

L-Acoustics has achieved EN 54 certification across a portfolio of loudspeakers,...

04/09/2026

IBC 2026: EVS and Matrox Video Announce Technology Partnership for Dynamic Media Facility Deployments

EVS and Matrox Video have announced a technology partnership integrating Matrox ...

04/09/2026

IBC 2026: NEP to Showcase Software-Defined Live Production

NEP Group will exhibit at IBC2026 (Hall 7, Stand 7.B55, RAI Amsterdam, September 11-14), demonstrating software-defined production capabilities across NEP Platf...

04/09/2026

FoMaSystems and ARRI Honored with Engineering Emmy for ARRI TRINITY 2

The Television Academy has awarded an Engineering Emmy to Dr. Roman Foltyn, Chief Engineer of FoMaSystems, and Curt O. Schaller for the development of the ARRI ...

04/09/2026

IBC 2026: InSync Technology to Showcase Frame Rate and Standards Conversion

InSync Technology will exhibit at IBC2026 (Stand 2.C30, RAI Amsterdam, September 11-14), demonstrating its FrameFormer and InGear conversion platforms for frame...

04/09/2026

Professional Triathletes Organization Signs Agreement with Sela to Host Events in Saudi Arabia

The Professional Triathletes Organisation (PTO) has signed a multi-year agreemen...

04/09/2026

IBC 2026: Accedo and MediaKind Launch Joint Streaming Solution for Operators

Accedo and MediaKind have announced a white-label, end-to-end streaming solution for operators, combining Accedo's native application framework and cloud-ba...

04/09/2026

IBC 2026: TAG Video Systems Launches TAG Blue Cloud Monitoring Platform

TAG Video Systems has announced TAG Blue, a cloud platform for shared monitoring visibility across video delivery chains, with AI-powered root-cause analysis. T...

04/09/2026

IBC 2026: TwelveLabs Launches Video Compliance Platform with NVIDIA Deepfake Detection

TwelveLabs has launched a video compliance platform for media organizations, inc...

04/09/2026

Television Academy and NATAS Announce 2026 Engineering, Science & Technology Emmy Award Recipients

The Television Academy and the National Academy of Television Arts & Sciences ha...

04/09/2026

KeySolutions Sounds release Traveler

Powerful new soft synth announced Traveler is the first software instrument to be released by KeySolutions Sounds, and it's an ambitious one! Said to co...

04/09/2026

e-instruments launch Velvet Guitars Fragment

Free virtual instrument range expanded The latest expansion of e-instruments' free Fragments series introduces a packed-down version of the company'...

04/09/2026

Accentize Chameleon wins Emmy Award

Company recognised for second time Two years after winning an Engineering, Science & Technology Emmy Award for dxRevive Pro, Accentize have been recognised ...

04/09/2026

AgileTV and Friend MTS Complete CDN Watermarking Integration to Fast-Track Anti-Piracy Protection for Operators

Bilbao - 3rd September 2026 -AgileTV, a leading provider of end-to-end TV and vi...

04/09/2026

Net Insight to Launch Nimbra 1020G at 2026 IBC Show

Share Copy link Facebook X Linkedin Bluesky Email...

04/09/2026

Disney, Optimum Renew Carriage Deal

Share Copy link Facebook X Linkedin Bluesky Email...

04/09/2026

ACEMAGIC First to Introduce Ryzen AI Max+ PRO 495 AI Mini Workstation at IFA 2026

ACEMAGIC First to Introduce Ryzen AI Max PRO 495 AI Mini Workstation at IFA 202...