Sony Pixel Power calrec Sony

Seamless in Seattle: NVIDIA Research Showcases Advancements in Visual Generative AI at CVPR

17/06/2024

NVIDIA researchers are at the forefront of the rapidly advancing field of visual generative AI, developing new techniques to create and interpret images, videos and 3D environments.

More than 50 of these projects will be showcased at the Computer Vision and Pattern Recognition (CVPR) conference, taking place June 17-21 in Seattle. Two of the papers - one on the training dynamics of diffusion models and another on high-definition maps for autonomous vehicles - are finalists for CVPR's Best Paper Awards.

NVIDIA is also the winner of the CVPR Autonomous Grand Challenge's End-to-End Driving at Scale track - a significant milestone that demonstrates the company's use of generative AI for comprehensive self-driving models. The winning submission, which outperformed more than 450 entries worldwide, also received CVPR's Innovation Award.

NVIDIA's research at CVPR includes a text-to-image model that can be easily customized to depict a specific object or character, a new model for object pose estimation, a technique to edit neural radiance fields (NeRFs) and a visual language model that can understand memes. Additional papers introduce domain-specific innovations for industries including automotive, healthcare and robotics.

Collectively, the work introduces powerful AI models that could enable creators to more quickly bring their artistic visions to life, accelerate the training of autonomous robots for manufacturing, and support healthcare professionals by helping process radiology reports.

Artificial intelligence, and generative AI in particular, represents a pivotal technological advancement, said Jan Kautz, vice president of learning and perception research at NVIDIA. At CVPR, NVIDIA Research is sharing how we're pushing the boundaries of what's possible - from powerful image generation models that could supercharge professional creators to autonomous driving software that could help enable next-generation self-driving cars.

At CVPR, NVIDIA also announced NVIDIA Omniverse Cloud Sensor RTX, a set of microservices that enable physically accurate sensor simulation to accelerate the development of fully autonomous machines of every kind.

Forget Fine-Tuning: JeDi Simplifies Custom Image Generation Creators harnessing diffusion models, the most popular method for generating images based on text prompts, often have a specific character or object in mind - they may, for example, be developing a storyboard around an animated mouse or brainstorming an ad campaign for a specific toy.

Prior research has enabled these creators to personalize the output of diffusion models to focus on a specific subject using fine-tuning - where a user trains the model on a custom dataset - but the process can be time-consuming and inaccessible for general users.

JeDi, a paper by researchers from Johns Hopkins University, Toyota Technological Institute at Chicago and NVIDIA, proposes a new technique that allows users to easily personalize the output of a diffusion model within a couple of seconds using reference images. The team found that the model achieves state-of-the-art quality, significantly outperforming existing fine-tuning-based and fine-tuning-free methods.

JeDi can also be combined with retrieval-augmented generation, or RAG, to generate visuals specific to a database, such as a brand's product catalog.

https://blogs.nvidia.com/wp-content/uploads/2024/06/JeDi-cow-sculpture.mp4

New Foundation Model Perfects the Pose NVIDIA researchers at CVPR are also presenting FoundationPose, a foundation model for object pose estimation and tracking that can be instantly applied to new objects during inference, without the need for fine-tuning.

The model, which set a new record on a popular benchmark for object pose estimation, uses either a small set of reference images or a 3D representation of an object to understand its shape. It can then identify and track how that object moves and rotates in 3D across a video, even in poor lighting conditions or complex scenes with visual obstructions.

FoundationPose could be used in industrial applications to help autonomous robots identify and track the objects they interact with. It could also be used in augmented reality applications where an AI model is used to overlay visuals on a live scene.

NeRFDeformer Transforms 3D Scenes With a Single Snapshot A NeRF is an AI model that can render a 3D scene based on a series of 2D images taken from different positions in the environment. In fields like robotics, NeRFs can be used to generate immersive 3D renders of complex real-world scenes, such as a cluttered room or a construction site. However, to make any changes, developers would need to manually define how the scene has transformed - or remake the NeRF entirely.

Researchers from the University of Illinois Urbana-Champaign and NVIDIA have simplified the process with NeRFDeformer. The method, being presented at CVPR, can successfully transform an existing NeRF using a single RGB-D image, which is a combination of a normal photo and a depth map that captures how far each object in a scene is from the camera.

VILA Visual Language Model Gets the Picture A CVPR research collaboration between NVIDIA and the Massachusetts Institute of Technology is advancing the state of the art for vision language models, which are generative AI models that can process videos, images and text.

The group developed VILA, a family of open-source visual language models that outperforms prior neural networks on key benchmarks that test how well AI models answer questions about images. VILA's unique pretraining process unlocked new model capabilities, including enhanced world knowledge, stronger in-context learning and the ability to reason across multiple images.

VILA can understand memes and reason based on multiple images or video frames. The VILA model fa
LINK: https://blogs.nvidia.com/blog/visual-generative-ai-cvpr-research/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

01/06/2026

Dolby Sets the New Standard for Premium Entertainment at CES 2026

January 6 2026, 05:30 (PST) Dolby Sets the New Standard for Premium Entertainment at CES 2026 Throughout the week, Dolby brings to life the latest innovatio...

02/05/2026

Dalet Flex LTS Delivers Smarter Search, Faster Editing, and an AI-Ready Foundation for Modern Media

Dalet, a leading technology and service provider for media-rich organizations, t...

01/05/2026

NBCUniversal's Peacock to Be First Streamer to Integrate Dolby's Full Suite of Premium Picture and Sound Innovations

January 5 2026, 18:30 (PST) NBCUniversal's Peacock to Be First Streamer to ...

01/04/2026

DOLBY AND DOUYIN EMPOWER THE NEXT GENERATON OF CREATORS WITH DOLBY VISION

January 4 2026, 18:00 (PST) DOLBY AND DOUYIN EMPOWER THE NEXT GENERATON OF CREATORS WITH DOLBY VISION Douyin Users Can Now Create And Share Videos With Stun...

12/03/2026

Blue Lucy's 6 Key Tenets

Modern media operations demand a platform that unites automation, orchestration, and human oversight without compromise. In this post, we explore the six key te...

12/03/2026

Blue Lucy Technology

A deep dive into the platform Architecture The Blue Lucy platform follows a distributed microservices architecture, meaning the overall operational capability...

12/03/2026

Blue Lucy Brings Order to the AI Wild West at NAB 2026

Orchestration platform enables broadcasters to deploy multiple AI models safely with full auditability, rights protection, and regulatory oversight. LONDON, En...

12/03/2026

Australia's financial sector ad spend surged by 20% in 2025 according to Nielsen

Cost pressures, switching intent and demand for savings and credit products are ...

12/03/2026

Nielsen Launches 2026 Upfront Planning Series With New Data To Help Marketers and Agencies Unlock Growth, Innovation Opportunities

For the first time, Nielsen breaks out demographic information about FAST and AV...

12/03/2026

Disney+ Goes Vertical with Verts

Share Copy link Facebook X Linkedin Bluesky Email...

12/03/2026

LTN and Appear Collaborate to Enable Scalable IP-Native L...

Appear's high-performance, ultra-low latency encoding platform augments LTN's fully managed global IP network and orchestration platform LTN, a leader ...

12/03/2026

Boland Communications to Showcase New OLED Displays and V...

2026 NAB Show Exhibitor Preview April 18-22 Las Vegas Booth C3519 Summary: At the 2026 NAB Show in Las Vegas, Boland Communications will be showing the bro...

12/03/2026

Riedel Expands Managed Technology Division in the America...

Riedel Communications today announced the continued expansion of its Managed Technology Division in the Americas and the appointment of Jan Schaffner as Vice Pr...

12/03/2026

Grass Valley Powers Hybrid IP Game-Day Production Hub for...

Grass Valley and integration partner Tab M Solutions have completed Phase 1 of a new broadcast production control room for the University of Illinois Division o...

12/03/2026

NAKIVO Expands Platform Support and Elevates Security in...

NAKIVO, a global provider of backup and ransomware recovery solutions, announces the general availability of NAKIVO Backup & Replication v11.2. This release exp...

12/03/2026

BIScience Adds Linear TV to its AdClarity Analysis Platform

Share Copy link Facebook X Linkedin Bluesky Email...

12/03/2026

Gray Stresses Importance of DRM for NextGen TV in FCC Sports Probe

Share Copy link Facebook X Linkedin Bluesky Email...

12/03/2026

Nebraska's HuskerVision Deploy Lawo IP Tech for Studio Upgrade

Share Copy link Facebook X Linkedin Bluesky Email...

12/03/2026

Comcast NBCU, Telemundo Station Group Announce $600,000 In Grants

Share Copy link Facebook X Linkedin Bluesky Email...

12/03/2026

EditShare To Highlight Analytical AI Capabilities At 2026 NAB Show

Share Copy link Facebook X Linkedin Bluesky Email...

12/03/2026

FreeWheel Launches AI Agent Infrastructure

Share Copy link Facebook X Linkedin Bluesky Email...

12/03/2026

COW Jobs: Editor de Vdeo - Direct Response, Performance Ads - Brazil, Remote

COW Jobs: Editor de V deo - Direct Response, Performance Ads - Brazil, Remote Brie Clayton March 11, 2026 0 Comments Editor(a) de V deo (Direct Respon...

12/03/2026

Avatar: Fire and Ash Graded with DaVinci Resolve Studio

Avatar: Fire and Ash Graded with DaVinci Resolve Studio Brie Clayton March 11, 2026 0 Comments Colorist delivers premium cinematic color across 2D, 3D...

12/03/2026

Boston Conservatory to Timothe Chalamet: We Care About Ballet and Opera

Boston Conservatory to Timoth e Chalamet: We Care About Ballet and Opera Boston Conservatory at Berklee students and faculty respond to the actors recent comm...

12/03/2026

Into the Omniverse: How Industrial AI and Digital Twins Accelerate Design, Engineering and Manufacturing Across Industries

Editor's note: This post is part of Into the Omniverse, a series focused on ...

12/03/2026

Jam-packed weekend of Irish entertainment, sport and music across RT for St Patrick's Day

The Late Late Show Show St Patrick's Day special Dancing with the Stars f...

12/03/2026

GeForce NOW Raises the Game at the Game Developers Conference

GeForce NOW is bringing the game to the Game Developers Conference (GDC), running this week in San Francisco. While developers build the future of gaming, GeFor...

11/03/2026

Calrec To Unlock Hybrid Workflows At 2026 NAB Show

Share Copy link Facebook X Linkedin Bluesky Email...

11/03/2026

Matrox Video Enables the Next Era of Software-Defined Med...

Matrox Video will showcase its vision for the future of live production at NAB 2026 in Las Vegas, April 19-22, highlighting how broadcasters and media organizat...

11/03/2026

GlobalM Showcases Distributed Video Gateway Architecture...

Geneva-based technology company, GlobalM SA, is presenting its GMX Distributed Video Gateway, a software-defined IP media transport platform designed to replace...

11/03/2026

Video is King - 2026 Iconik Media Stats Report Finds Vide...

Backlight (booth #N2829), the company behind Iconik and Wildmoka, which power video workflows for large media and entertainment organizations, has released the ...

11/03/2026

QuickLinks Latest StudioEdge Models to Make North America...

QuickLink, a leading provider of award-winning video production and remote guest contribution solutions, presents its latest StudioEdge models at The NAB Show ...

11/03/2026

Telestream Expands Its Cloud Services with the Introducti...

Telestream, a global leader in media workflow technologies, today announced the expansion of Telestream Cloud Services with the introduction of UP, a new cloud-...

11/03/2026

Operative Launches AOS Configuration for Digital-First Mo...

Operative, the preferred advertising management provider for the world's leading media brands, today announced the launch of AOS for digital media, an AI-po...

11/03/2026

Calrec Redefines Broadcast Workflows at NAB 2026

Calrec will be located in Central Hall, on Booth C6907 Choice without compromise The broadcast industry is going through a rapid evolution that s signalling a...

11/03/2026

Worldstream and Cubbit launch sovereign S3 cloud storage...

The new service is hosted and operated entirely in the Netherlands, combining data sovereignty, resilience, scalability, and predictable costs without relying...

11/03/2026

Ease Live powers interactive Premier Padel experiences on...

Ease Live, an Evertz company and leader in interactive graphical overlays, today announced the successful deployment of its platform on Red Bull TV for Premier ...

11/03/2026

Mediagenix Title Management Accelerates Content Monetizat...

Mediagenix, a global leader in smart content solutions to profitably connect the right content to the right audience, is advancing its Semantic Intelligence cap...

11/03/2026

Emergent Launches Fusion- The Interactive Anything Platfo...

Emergent, a leading provider of AI-enhanced media production solutions, today announced the official launch of Fusion, a powerful, no-code application builder d...

11/03/2026

Techex Names Matt McKee as Senior Director of Sales, Americas

Share Copy link Facebook X Linkedin Bluesky Email...

11/03/2026

IAB Tech Lab Announces Content Monetization Protocol for AI LLMs

Share Copy link Facebook X Linkedin Bluesky Email...

11/03/2026

Mondae Hott Joins Kokusai Denki as Northeast Sales Manager

Share Copy link Facebook X Linkedin Bluesky Email...

11/03/2026

Gray Media to Air Cincinnati Reds' Games on WXIX FOX19

Share Copy link Facebook X Linkedin Bluesky Email...

11/03/2026

Shure Audio Solutions Deliver Super Bowl Win

Share Copy link Facebook X Linkedin Bluesky Email...

11/03/2026

UK's First Live Broadcast Using New n40 Private 5G Spectrum

Share Copy link Facebook X Linkedin Bluesky Email...

11/03/2026

Utah Scientific Expands Technology Partner Program With I...

Utah Scientific today announced the expansion of its Technology Partner Program with the addition of Audinate, Bitfocus, and Skaarhoj, three industry leaders wh...

11/03/2026

DigitalGlue Ends the Post Production Tax creativespace In...

DigitalGlue, creator of the creative.space on-premise managed storage platform, today revealed plans to launch creative.space Intelligence (CSI) at NAB 2026 (Bo...

11/03/2026

Maxon and Tencent Cloud Partner to Integrate HY 3D into C...

Maxon, maker of powerful, approachable software solutions for creators working in 2D and 3D design, motion graphics, visual effects, gaming, and more, has annou...

11/03/2026

NUGEN Audio Halo Vision Plug In Serves as Spatial Compass...

Composer and Re-recording Mixer Michael Phillips Keeley has built his career around immersive storytelling. Working from his Dolby Atmos-equipped studio, Sound ...