Sony Pixel Power calrec Sony

Seamless in Seattle: NVIDIA Research Showcases Advancements in Visual Generative AI at CVPR

17/06/2024

NVIDIA researchers are at the forefront of the rapidly advancing field of visual generative AI, developing new techniques to create and interpret images, videos and 3D environments.

More than 50 of these projects will be showcased at the Computer Vision and Pattern Recognition (CVPR) conference, taking place June 17-21 in Seattle. Two of the papers - one on the training dynamics of diffusion models and another on high-definition maps for autonomous vehicles - are finalists for CVPR's Best Paper Awards.

NVIDIA is also the winner of the CVPR Autonomous Grand Challenge's End-to-End Driving at Scale track - a significant milestone that demonstrates the company's use of generative AI for comprehensive self-driving models. The winning submission, which outperformed more than 450 entries worldwide, also received CVPR's Innovation Award.

NVIDIA's research at CVPR includes a text-to-image model that can be easily customized to depict a specific object or character, a new model for object pose estimation, a technique to edit neural radiance fields (NeRFs) and a visual language model that can understand memes. Additional papers introduce domain-specific innovations for industries including automotive, healthcare and robotics.

Collectively, the work introduces powerful AI models that could enable creators to more quickly bring their artistic visions to life, accelerate the training of autonomous robots for manufacturing, and support healthcare professionals by helping process radiology reports.

Artificial intelligence, and generative AI in particular, represents a pivotal technological advancement, said Jan Kautz, vice president of learning and perception research at NVIDIA. At CVPR, NVIDIA Research is sharing how we're pushing the boundaries of what's possible - from powerful image generation models that could supercharge professional creators to autonomous driving software that could help enable next-generation self-driving cars.

At CVPR, NVIDIA also announced NVIDIA Omniverse Cloud Sensor RTX, a set of microservices that enable physically accurate sensor simulation to accelerate the development of fully autonomous machines of every kind.

Forget Fine-Tuning: JeDi Simplifies Custom Image Generation Creators harnessing diffusion models, the most popular method for generating images based on text prompts, often have a specific character or object in mind - they may, for example, be developing a storyboard around an animated mouse or brainstorming an ad campaign for a specific toy.

Prior research has enabled these creators to personalize the output of diffusion models to focus on a specific subject using fine-tuning - where a user trains the model on a custom dataset - but the process can be time-consuming and inaccessible for general users.

JeDi, a paper by researchers from Johns Hopkins University, Toyota Technological Institute at Chicago and NVIDIA, proposes a new technique that allows users to easily personalize the output of a diffusion model within a couple of seconds using reference images. The team found that the model achieves state-of-the-art quality, significantly outperforming existing fine-tuning-based and fine-tuning-free methods.

JeDi can also be combined with retrieval-augmented generation, or RAG, to generate visuals specific to a database, such as a brand's product catalog.

https://blogs.nvidia.com/wp-content/uploads/2024/06/JeDi-cow-sculpture.mp4

New Foundation Model Perfects the Pose NVIDIA researchers at CVPR are also presenting FoundationPose, a foundation model for object pose estimation and tracking that can be instantly applied to new objects during inference, without the need for fine-tuning.

The model, which set a new record on a popular benchmark for object pose estimation, uses either a small set of reference images or a 3D representation of an object to understand its shape. It can then identify and track how that object moves and rotates in 3D across a video, even in poor lighting conditions or complex scenes with visual obstructions.

FoundationPose could be used in industrial applications to help autonomous robots identify and track the objects they interact with. It could also be used in augmented reality applications where an AI model is used to overlay visuals on a live scene.

NeRFDeformer Transforms 3D Scenes With a Single Snapshot A NeRF is an AI model that can render a 3D scene based on a series of 2D images taken from different positions in the environment. In fields like robotics, NeRFs can be used to generate immersive 3D renders of complex real-world scenes, such as a cluttered room or a construction site. However, to make any changes, developers would need to manually define how the scene has transformed - or remake the NeRF entirely.

Researchers from the University of Illinois Urbana-Champaign and NVIDIA have simplified the process with NeRFDeformer. The method, being presented at CVPR, can successfully transform an existing NeRF using a single RGB-D image, which is a combination of a normal photo and a depth map that captures how far each object in a scene is from the camera.

VILA Visual Language Model Gets the Picture A CVPR research collaboration between NVIDIA and the Massachusetts Institute of Technology is advancing the state of the art for vision language models, which are generative AI models that can process videos, images and text.

The group developed VILA, a family of open-source visual language models that outperforms prior neural networks on key benchmarks that test how well AI models answer questions about images. VILA's unique pretraining process unlocked new model capabilities, including enhanced world knowledge, stronger in-context learning and the ability to reason across multiple images.

VILA can understand memes and reason based on multiple images or video frames. The VILA model fa
LINK: https://blogs.nvidia.com/blog/visual-generative-ai-cvpr-research/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

01/06/2026

Dolby Sets the New Standard for Premium Entertainment at CES 2026

January 6 2026, 05:30 (PST) Dolby Sets the New Standard for Premium Entertainment at CES 2026 Throughout the week, Dolby brings to life the latest innovatio...

02/05/2026

Dalet Flex LTS Delivers Smarter Search, Faster Editing, and an AI-Ready Foundation for Modern Media

Dalet, a leading technology and service provider for media-rich organizations, t...

01/05/2026

NBCUniversal's Peacock to Be First Streamer to Integrate Dolby's Full Suite of Premium Picture and Sound Innovations

January 5 2026, 18:30 (PST) NBCUniversal's Peacock to Be First Streamer to ...

01/04/2026

DOLBY AND DOUYIN EMPOWER THE NEXT GENERATON OF CREATORS WITH DOLBY VISION

January 4 2026, 18:00 (PST) DOLBY AND DOUYIN EMPOWER THE NEXT GENERATON OF CREATORS WITH DOLBY VISION Douyin Users Can Now Create And Share Videos With Stun...

13/03/2026

Missionized Business Jets: Fielded and Flying

Over a year ago, L3Harris delivered the first missionized Bombardier Global 6500 aircraft for U.S. Indo-Pacific Command. Two ATHENA-R platforms now average 400+...

13/03/2026

GFiber and Stonepeak's Astound Broadband to Merge

Share Copy link Facebook X Linkedin Bluesky Email...

13/03/2026

Harmonic Redefines the Economics of Video Playout with Ne...

Harmonic (NASDAQ: HLIT) today announced Spectrum X Plus, the newest generation of its Spectrum X media server, offering double the channel density of previous ...

13/03/2026

Historic Ewing Covenant Church Revitalizes Worship Experi...

When Ewing Covenant Church made the decision to return to its original, historic building, affectionately called 1867 Sanctuary for weekly worship, the congre...

13/03/2026

Marshall Electronics Launches CV574 WP its First 4K All I...

Marshall Electronics introduces its first all-IP 4K POV camera, the CV574-WP, at NAB 2026 (Booth C8339). The CV574-WP supports NDI |HX, providing ultra-efficien...

13/03/2026

Net Insight launches Nimbra 520 - Predictable Live Media...

At NAB Show 2026, Net Insight introduces Nimbra 520, a high-density media processing node designed to simplify live contribution and distribution across both ma...

13/03/2026

Abandon Editorial Signs With Michal Dimitri for West Coast Representation

Abandon Editorial Signs With Micha l Dimitri for West Coast Representation Brie Clayton March 12, 2026 0 Comments Abandon Editorial is excited to part...

13/03/2026

Documentary The Bulldogs Shot and Edited with Blackmagic Design

Documentary The Bulldogs Shot and Edited with Blackmagic Design Brie Clayton March 12, 2026 0 Comments Editorial tools helped shape film in real time,...

13/03/2026

AE Captions as Fast as CapCut - No Plugins

AE Captions as Fast as CapCut - No Plugins Graham Quince March 12, 2026 0 Comments Stop wasting hours clicking through nested compositions and manuall...

13/03/2026

New Music USA and Berklee Institute of Jazz and Gender Justice Announce 2026 Next Jazz Legacy Cohort

New Music USA and Berklee Institute of Jazz and Gender Justice Announce 2026 Nex...

13/03/2026

RT announces Rick O'Shea as new presenter on RT Radio 1's Arena

RT has announced today that Rick O'Shea is the new presenter of Arena RT Radio 1's flagship weeknight arts and culture programme. Rick has been pres...

13/03/2026

Lights! Camera! Action! The 98th Oscars set to air live as RT backs the Irish nominees

Lights! Camera! Action! The 98th Oscars set to air live as RT backs the Irish n...

12/03/2026

Milano Cortina 2026: Yospace helps ad-funded rights-holders claim advertising gold

Staines-upon-Thames, UK, 11th March, 2026 - Yospace, the trusted leader in Dyna...

12/03/2026

Utah Scientific Expands Technology Partner Program With Integrations From Audinate, Bitfocus, and Skaarhoj

Utah Scientific Expands Technology Partner Program With Integrations From Audina...

12/03/2026

Techex Hires Matt McKee as Senior Director, Sales, Americas

Techex, a global expert in live video solutions over IP and cloud, announces the appointment of Matt McKee as Senior Director, Sales, Americas, further strength...

12/03/2026

KOKUSAI DENKI Electric America Welcomes Mondae Hott as Regional Sales Manager, Northeast

KOKUSAI DENKI Electric America has appointed Mondae Hott as Regional Sales Manag...

12/03/2026

Interra Systems Focuses on Streaming-First QC, Monitoring, and Responsible AI Workflows at 2026 NAB Show

At the 2026 NAB Show, Interra Systems will showcase its latest advancements in a...

12/03/2026

15th National Games of China Features Expansive Unified Broadcast Network

The 15th National Games of China concluded after a two-week celebration of athletic excellence and regional collaboration. Held from Nov. 9-21 across Guangdong,...

12/03/2026

SVG in Indy: Butler University Students Get Live-Sports Experience via On-Campus Studio, Production Truck

Live-production academic program Butler Sports Live produced a total of 40 fall-...

12/03/2026

Lawo Delivers Fully IPBased Video Infrastructure for University of Nebraska's HuskerVision

The University of Nebraska's HuskerVision has completed the second phase of ...

12/03/2026

University of Illinois Upgrades Game-Day Production with New Hybrid IP Control Room

Grass Valley and integration partner Tab M Solutions have completed Phase 1 of a...

12/03/2026

CBS Sports Turns to University of Cincinnati's Control Room and Staff for Two Big 12 Men's Basketball Broadcasts

The broadcaster expands its campus-production model as the university handles tw...

12/03/2026

Disney+ to Stream Men's and Women's NCAA March Madness Across Europe, South Africa

Disney has announced the addition of March Madness - the NCAA Division I Men...

12/03/2026

Apple TV Further Integrating iPhones into Friday Night Baseball' Camera Lineup This Season

Apple TV's Friday Night Baseball MLB doubleheader series returns for its f...

12/03/2026

SVG Students To Watch: Jack Rinaldi, University of Notre Dame

The senior from New Jersey is making his mark in South Bend, both on the mic and behind it...

12/03/2026

Ross Video's Kevin Dresser on Powering Graphics for Major Sporting Events Throughout a Busy February

After a relatively quiet January, the month of February was jammed packed with l...

12/03/2026

X Games Launches New Era With First League Draft at Cosm L.A.

Long-time production partner Echo Entertainment is producing the broadcast, while Cosm played a vital role in the collaboration...

12/03/2026

Film Festival Watch: 24 Sundance Institute-Supported Films to Catch at the SXSW Film Festival

By Jessica Herndon We love kicking off each year by introducing the world to po...

12/03/2026

Didn't Die Brings Improv to the Zombie Apocalypse

Samrat Chakrabarti, George Basil, Kiran Deol, Katie McCuen and Vishal Vijayakumar attend the 2025 Sundance Film Festival premiere of Didn't Die at the Lib...

12/03/2026

Mon Laferte Leads All-Women Spotify Session as EQUAL Celebrations Kick Off in Latin America

In Latin America, women are shaping music and defining its future. To kick off t...

12/03/2026

Mon Laferte lidera la edicin EQUAL de Spotify Sessions, mientras comienzan las celebraciones de EQUAL en Latinoamrica

En Am rica Latina, las mujeres est n moldeando la m sica y definiendo su futuro....

12/03/2026

As Spotify Turns 20, the Most Global and Diverse Music Industry in History Has Taken Shape

Let's turn back the clock 20 years: The music landscape was a world away fro...

12/03/2026

Bad Bunny Brings the Sounds of Puerto Rico to Tokyo for Spotify's Billions Club Live

Bad Bunny is no stranger to Spotify's Billions Club. In fact, he has a whopp...

12/03/2026

At the London Book Fair, Spotify Shares Our Vision for the Future of Reading

Spotify was at the London Book Fair this week, joining conversations across the publishing industry about how people can make reading part of their daily lives....

12/03/2026

Ohlhorst Digital & Tokyo Dawn Labs launch Ancora

Mastering tool improves mono compatibility Tokyo Dawn Labs' Ohlhorst Digital range is a series of mastering-focused plug-ins developed by Jan Ohlhorst, ...

12/03/2026

Lewitt partner with Elgato

Wave FX processor integrated into four products Lewitt have teamed up with Elgato to create a new processor for the company's Wave Next product range, i...

12/03/2026

Mix Notes iOS App by David Thomas

Free tool for annotating audio files Mix Notes is a new, free iOS App that provides users with a simple way to annotate their audio files. It's been cre...

12/03/2026

Duck 2 from Devious Machines

Side-chain ducking tool gets an upgrade Devious Machines' popular side-chaining and envelope-shaping tool has just been kitted out with an improved enve...

12/03/2026

MPG Awards 2026: Shortlist announced

Ceremony to take place on 16 April 2026 The MPG (Music Producers Guild) have revealed the full shortlist for this year's MPG Awards, which will be takin...

12/03/2026

Overloud introduce Gem Comp160

Emulates three classic dbx 160 variants The latest arrival to Overloud's Gem Series plug-in range faithfully recreates not one, but three versions of th...

12/03/2026

Grainferno from Baby Audio

New granular soft synth announced Said to be their most advanced software synthesizer to date, Baby Audio's latest release has been built on a new granu...

12/03/2026

Bitwig Studio 6 launches

Latest version now live! Edit 11 March 2026 - Bitwig Studio 6 is now live, and available for all to download! The latest version of Bitwig's DAW softwa...

12/03/2026

Stereo Miking: The Sound On Sound Guide

Latest free eBook now available! Designed for recording engineers, audio-technology students and technically minded musicians, our latest free eBook deliver...

12/03/2026

AFL and NITV partner to launch new First Nations led program Inside the Huddle'

AFL and NITV partner to launch new First Nations led program Inside the Huddle&...

12/03/2026

Rohde & Schwarz Cybersecurity expands SITLine network encryptor portfolio - more bandwidth, higher port density, future-proof architecture

Rohde & Schwarz Cybersecurity expands SITLine network encryptor portfolio - more...