Sony Pixel Power calrec Sony

Crowning Achievement: NVIDIA Research Model Enables Fast, Efficient Dynamic Scene Reconstruction

09/12/2024

Content streaming and engagement are entering a new dimension with QUEEN, an AI model by NVIDIA Research and the University of Maryland that makes it possible to stream free-viewpoint video, which lets viewers experience a 3D scene from any angle.

QUEEN could be used to build immersive streaming applications that teach skills like cooking, put sports fans on the field to watch their favorite teams play from any angle, or bring an extra level of depth to video conferencing in the workplace. It could also be used in industrial environments to help teleoperate robots in a warehouse or a manufacturing plant.

The model will be presented at NeurIPS, the annual conference for AI research that begins Tuesday, Dec. 10, in Vancouver.

To stream free-viewpoint videos in near real time, we must simultaneously reconstruct and compress the 3D scene, said Shalini De Mello, director of research and a distinguished research scientist at NVIDIA. QUEEN balances factors including compression rate, visual quality, encoding time and rendering time to create an optimized pipeline that sets a new standard for visual quality and streamability.

Reduce, Reuse and Recycle for Efficient Streaming Free-viewpoint videos are typically created using video footage captured from different camera angles, like a multicamera film studio setup, a set of security cameras in a warehouse or a system of videoconferencing cameras in an office.

Prior AI methods for generating free-viewpoint videos either took too much memory for livestreaming or sacrificed visual quality for smaller file sizes. QUEEN balances both to deliver high-quality visuals - even in dynamic scenes featuring sparks, flames or furry animals - that can be easily transmitted from a host server to a client's device. It also renders visuals faster than previous methods, supporting streaming use cases.

In most real-world environments, many elements of a scene stay static. In a video, that means a large share of pixels don't change from one frame to another. To save computation time, QUEEN tracks and reuses renders of these static regions - focusing instead on reconstructing the content that changes over time.

Using an NVIDIA Tensor Core GPU, the researchers evaluated QUEEN's performance on several benchmarks and found the model outperformed state-of-the-art methods for online free-viewpoint video on a range of metrics. Given 2D videos of the same scene captured from different angles, it typically takes under five seconds of training time to render free-viewpoint videos at around 350 frames per second.

This combination of speed and visual quality can support media broadcasts of concerts and sports games by offering immersive virtual reality experiences or instant replays of key moments in a competition.

In warehouse settings, robot operators could use QUEEN to better gauge depth when maneuvering physical objects. And in a videoconferencing application - such as the 3D videoconferencing demo shown at SIGGRAPH and NVIDIA GTC - it could help presenters demonstrate tasks like cooking or origami while letting viewers pick the visual angle that best supports their learning.

The code for QUEEN will soon be released as open source and shared on the project page.

QUEEN is one of over 50 NVIDIA-authored NeurIPS posters and papers that feature groundbreaking AI research with potential applications in fields including simulation, robotics and healthcare. Among them is a best paper runner-up, titled Guiding a Diffusion Model With a Bad Version of Itself - one of just five papers recognized among the over 4,500 accepted this year.

Generative Adversarial Nets, the paper that first introduced GAN models, won the NeurIPS 2024 Test of Time Award. Cited more than 85,000 times, the paper was coauthored by Bing Xu, distinguished engineer at NVIDIA. Hear more from its lead author, Ian Goodfellow, research scientist at DeepMind, on the AI Podcast:

The AI Podcast Ep. 25: Google's Ian Goodfellow on How an Argument in a Bar Led to Generative Adversarial Networks

Learn more about NVIDIA Research at NeurIPS.

See the latest work from NVIDIA Research, which has hundreds of scientists and engineers worldwide, with teams focused on topics including AI, computer graphics, computer vision, self-driving cars and robotics.

Academic researchers working on large language models, simulation and modeling, edge AI and more can apply to the NVIDIA Academic Grant Program.

See notice regarding software product information.
LINK: https://blogs.nvidia.com/blog/neurips-2024-research/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

07/10/2026

Dalet Flex LTS Delivers Smarter Media Operations from Ingest to Distribution

Dalet, a leading technology and service provider for media-rich organizations, today announced the latest Long-Term Supported (LTS) release of Dalet Flex. Build...

06/09/2026

Dolby and MagentaTV Bring Fans Closer to the FIFA World Cup 2026 in Germany with Dolby Vision and Dolby Atmos

June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...

20/08/2026

Nielsen Incorporates New Enhancements To Improve Its Data Measurement Leading Into The New Fall TV Season

New York August 19, 2026 Nielsen today shared a number of key enhancements t...

20/08/2026

FAST Sports Programming Surges at More Than Twice the Rate of Sports Channel Growth

New Gracenote Data Hub analysis finds sports program titles up 31.2% and individ...

20/08/2026

Charter Completes $34.5 Billion Cox Acquisition

Share Copy link Facebook X Linkedin Bluesky Email...

20/08/2026

SubtitleNEXT makes a significant leap in localisation eff...

At IBC, Profuz Digital will showcase at stand 8.A88 the SubtitleNEXT platform delivering innovative AI transcription integrations and stunning new features boos...

20/08/2026

Transformation Church Elevates Production Communications...

Riedel Communications today announced that Transformation Church, one of the fastest-growing and most influential digital ministries in the United States, has d...

20/08/2026

QuickLink Showcases New 4K Capabilities Across StudioEdge and StudioCall at IBC 2026

QuickLink Showcases New 4K Capabilities Across StudioEdge and StudioCall at IBC ...

20/08/2026

The Boys Season Five Graded with DaVinci Resolve Studio

The Boys Season Five Graded with DaVinci Resolve Studio Brie Clayton August 20, 2026 0 Comments Colorist Siggy Ferstl uses Depth Map to bring a three ...

20/08/2026

Adobe Firefly expands its creative AI studio: generate music, speech, and sound effects in one place

Adobe Firefly expands its creative AI studio: generate music, speech, and sound ...

20/08/2026

Broadpeak launches Multiview as a Service to accelerate m...

Multiview available via SaaS for streaming platforms and operators through broadpeak.io, with live demonstrations at IBC2026 Broadpeak , a leader in streaming ...

20/08/2026

Techex Appoints Stuart Almond as Chief Revenue Officer

Techex, a leading provider of live broadcast infrastructure solutions, has appointed Stuart Almond as Chief Revenue Officer, further strengthening its executive...

20/08/2026

MediaCity Studios upgrades with Avid

Avid today announced that MediaCity Studios has upgraded its production infrastructure with an enhanced end-to-end Avid production ecosystem, including the dep...

20/08/2026

MS Now to Launch $7.99 DTC Service

Share Copy link Facebook X Linkedin Bluesky Email...

20/08/2026

Nielsen Details Improvements to Audience Measurement Capabilities

Share Copy link Facebook X Linkedin Bluesky Email...

20/08/2026

Study: 174 Billion Illegal Streams During FIFA World Cup 2026

Share Copy link Facebook X Linkedin Bluesky Email...

20/08/2026

Prasad Opens New 8K/4K Film Preservation & Restoration Center

Share Copy link Facebook X Linkedin Bluesky Email...

20/08/2026

IPC Expands Brazilian Presence with New Channel Partner C...

IPC Expands Brazilian Presence with New Channel Partner CCATech Industry veterans Carlos Abrah o and Bruno Lopes strengthen local support and expand IPCs prese...

20/08/2026

Zixi Enters Mauritius Market Through Strategic Deployment...

Zixi Enters Mauritius Market Through Strategic Deployment with MTC Jockey Club Worlds Oldest Horse Racing Club in the Southern Hemisphere Selects Zixis Softwar...

20/08/2026

Scripps Research scientist to receive $2.76 million NIAID New Innovators Award

LA JOLLA, CA-Scripps Research assistant professor Tiantian Liu has received a five-year NIAID DP2 award from the National Institute of Allergy and Infectious Di...

20/08/2026

Audiences reveal appetite for multicultural Australia on-screen

Audiences reveal appetite for multicultural Australia on-screen 20 August 2026 Dr Georgie McClean, Dr Rebecca Huntley (L-R) By Dr Georgie McClean and Dr Rebec...

20/08/2026

How Mavis Studio Is Expanding Live Production Teaching at Sussex

How Mavis Studio Is Expanding Live Production Teaching at Sussex Brie Clayton August 19, 2026 0 Comments Hero image displays a student about to mix be...

20/08/2026

Vertical Unscripted Series Legal Queens Captured with Blackmagic Design

Vertical Unscripted Series Legal Queens Captured with Blackmagic Design Brie Clayton August 19, 2026 0 Comments Reality TV veteran Eric Gardner create...

20/08/2026

Arvato Systems brings together digital transformation leaders in the manufacturing industry

Arvato Systems Brings Together Digital Transformation Leaders in the Manufacturi...

20/08/2026

Bring the Fire: Play Games on GeForce NOW With New Firefox Browser Support

It's a new way into the cloud. GeForce NOW welcomes Firefox support to the cloud, opening up another way to jump into high-performance PC gaming straight ...

19/08/2026

Imagine Communications Steve Reynolds on Kaseya Centers Hybrid IP Build and the Growth of ST 2110

As the cost of IP nears price parity with traditional SDI, Reynolds argues that ...

19/08/2026

European Athletics Championships 2026: BBC, Event Organizers Bring Birmingham Action to Viewers in Style

A private 5G bubble was chosen instead of traditional broadcast RF for the race ...

19/08/2026

Filipiana Sets the Class Divide on a Golf Course in the Philippines

(L-R) Rafael Manuel and Isabel Sicat attend the Filipi ana premiere during the 2026 Sundance Film Festival at The Ray Theatre on January 23, 2026, in Park Cit...

19/08/2026

MPG launch Save Our Studios survey

Studio owners invited to have their say on rates cuts The Music Producers Guild (MPG) are inviting UK-based studios to take part in an in-depth survey that ...

19/08/2026

Harrison unveil the Flex-10

Vintage-inspired desktop interface revealed Harrison's latest hardware offering sees them head into the world of audio interfaces, delivering a 10-in/12...

19/08/2026

Minimal Audio release Lucid

Granular effect remains locked to scale and tempo Described as your dream granular effect , Minimal Audio's latest creation is capable of turning any s...

19/08/2026

Audio Modeling open PolySWAM pre-orders

First polyphonic SWAM instrument incoming Audio Modeling's latest physical-modelling orchestral instrument is just around the corner, and is now availab...

19/08/2026

FOX and NBCUniversal Score with FIFA World Cup 2026 Success in Nielsen's June Gauge Reports

World Cup Games Generate 84 Billion Minutes Across FOX, Fox Sports 1 and NBCU...

19/08/2026

The Gauge: Poland | July 2026

July brought typical peak-holiday drops in viewership, driven primarily by viewers stepping away from screens and a visible decline in reach across both traditi...

19/08/2026

Carr Calls ABC Suit 'Meritless

Share Copy link Facebook X Linkedin Bluesky Email...

19/08/2026

Henry Alexander Joins BCE as Sales Director

Henry joins BCE as the company accelerates its commercial strategy and expands its international presence across the media, broadcast and technology markets. He...

19/08/2026

iWedia Announces Advanced Android DRAM Optimisation Proce...

iWedia, a leading provider of software solutions for connected TV devices, today announced a major milestone in Android platform optimization with the successfu...

19/08/2026

NTP Technology announces ST 2110-30-RAVENNA 128-channel A...

NTP Technology announces that ST 2110-30 audio-over-IP connectivity with AES67 and RAVENNA capabilities is now available for the brand's world-class audio i...

19/08/2026

ORF goes full IP with Net Insight in nationwide network o...

Net Insight has been selected to provide its Nimbra platform by the service provider Big Blue Marble, formerly ORS Group, as the core media transport solution f...

19/08/2026

Coventry City Trophy Parade Extends LiveU LIQ Beyond the...

LiveU has supported Coventry City FC and several other top-flight UK football clubs through one of their most demanding live production challenges of the year, ...

19/08/2026

Nature Seekers Documentary Shot with Blackmagic URSA Cine 12K LF

Nature Seekers Documentary Shot with Blackmagic URSA Cine 12K LF Brie Clayton August 18, 2026 0 Comments Filming endangered turtles required camera...

19/08/2026

Neyrinck Brings D-Control Consoles Back to Life on Apple Silicon with D-Control Basic

Neyrinck Brings D-Control Consoles Back to Life on Apple Silicon with D-Control ...

19/08/2026

ABC Files First Amendment Lawsuit Against FCC

Share Copy link Facebook X Linkedin Bluesky Email...

19/08/2026

Nielsen: Fox and NBCUniversal Score with FIFA World Cup 2026

Share Copy link Facebook X Linkedin Bluesky Email...

19/08/2026

ESPN Co-Founder Bill Rasmussen Dies At 93

Share Copy link Facebook X Linkedin Bluesky Email...

19/08/2026

Study: NBA Ad Revenue Hit $2.1 Billion Last Season

Share Copy link Facebook X Linkedin Bluesky Email...

19/08/2026

Nexstar's WNCN Donates ENG Van for Training

Share Copy link Facebook X Linkedin Bluesky Email...

19/08/2026

Riedel Brings Professional Live Production Into the Class...

Riedel Communications today announced that Edith Cowan University (ECU) in Perth, Australia, implemented a comprehensive communications infrastructure that brin...

19/08/2026

QuickLink Showcases New 4K Capabilities Across StudioEdge...

QuickLink, a leading provider of award-winning video production and remote guest contribution solutions, will spotlight a major leap forward in remote guest con...