Sony Pixel Power calrec Sony

NVIDIA Advances Physical AI at CVPR With Largest Indoor Synthetic Dataset

17/06/2024

NVIDIA contributed the largest ever indoor synthetic dataset to the Computer Vision and Pattern Recognition (CVPR) conference's annual AI City Challenge - helping researchers and developers advance the development of solutions for smart cities and industrial automation.

The challenge, garnering over 700 teams from nearly 50 countries, tasks participants to develop AI models to enhance operational efficiency in physical settings, such as retail and warehouse environments, and intelligent traffic systems.

Teams tested their models on the datasets that were generated using NVIDIA Omniverse, a platform of application programming interfaces (APIs), software development kits (SDKs) and services that enable developers to build Universal Scene Description (OpenUSD)-based applications and workflows.

Creating and Simulating Digital Twins for Large Spaces In large indoor spaces like factories and warehouses, daily activities involve a steady stream of people, small vehicles and future autonomous robots. Developers need solutions that can observe and measure activities, optimize operational efficiency, and prioritize human safety in complex, large-scale settings.

Researchers are addressing that need with computer vision models that can perceive and understand the physical world. It can be used in applications like multi-camera tracking, in which a model tracks multiple entities within a given environment.

To ensure their accuracy, the models must be trained on large, ground-truth datasets for a variety of real-world scenarios. But collecting that data can be a challenging, time-consuming and costly process.

AI researchers are turning to physically based simulations - such as digital twins of the physical world - to enhance AI simulation and training. These virtual environments can help generate synthetic data used to train AI models. Simulation also provides a way to run a multitude of what-if scenarios in a safe environment while addressing privacy and AI bias issues.

Creating synthetic data is important for AI training because it offers a large, scalable, and expandable amount of data. Teams can generate a diverse set of training data by changing many parameters including lighting, object locations, textures and colors.

Building Synthetic Datasets for the AI City Challenge This year's AI City Challenge consists of five computer vision challenge tracks that span traffic management to worker safety.

NVIDIA contributed datasets for the first track, Multi-Camera Person Tracking, which saw the highest participation, with over 400 teams. The challenge used a benchmark and the largest synthetic dataset of its kind - comprising 212 hours of 1080p videos at 30 frames per second spanning 90 scenes across six virtual environments, including a warehouse, retail store and hospital.

Created in Omniverse, these scenes simulated nearly 1,000 cameras and featured around 2,500 digital human characters. It also provided a way for the researchers to generate data of the right size and fidelity to achieve the desired outcomes.

The benchmarks were created using Omniverse Replicator in NVIDIA Isaac Sim, a reference application that enables developers to design, simulate and train AI for robots, smart spaces or autonomous machines in physically based virtual environments built on NVIDIA Omniverse.

Omniverse Replicator, an SDK for building synthetic data generation pipelines, automated many manual tasks involved in generating quality synthetic data, including domain randomization, camera placement and calibration, character movement, and semantic labeling of data and ground-truth for benchmarking.

Ten institutions and organizations are collaborating with NVIDIA for the AI City Challenge:

Australian National University, Australia

Emirates Center for Mobility Research, UAE

Indian Institute of Technology Kanpur, India

Iowa State University, U.S.

Johns Hopkins University, U.S.

National Yung-Ming Chiao-Tung University, Taiwan

Santa Clara University, U.S.

The United Arab Emirates University, UAE

University at Albany - SUNY, U.S.

Woven by Toyota, Japan

Driving the Future of Generative Physical AI Researchers and companies around the world are developing infrastructure automation and robots powered by physical AI - which are models that can understand instructions and autonomously perform complex tasks in the real world.

Generative physical AI uses reinforcement learning in simulated environments, where it perceives the world using accurately simulated sensors, performs actions grounded by laws of physics, and receives feedback to reason about the next set of actions.

Developers can tap into developer SDKs and APIs, such as the NVIDIA Metropolis developer stack - which includes a multi-camera tracking reference workflow - to add enhanced perception capabilities for factories, warehouses and retail operations. And with the latest release of NVIDIA Isaac Sim, developers can supercharge robotics workflows by simulating and training AI-based robots in physically based virtual spaces before real-world deployment.

Researchers and developers are also combining high-fidelity, physics-based simulation with advanced AI to bridge the gap between simulated training and real-world application. This helps ensure that synthetic training environments closely mimic real-world conditions for more seamless robot deployment.

NVIDIA is taking the accuracy and scale of simulations further with the recently announced NVIDIA Omniverse Cloud Sensor RTX, a set of microservices that enable physically accurate sensor simulation to accelerate the development of fully autonomous machines.

This technology will allow autonomous systems, whether a factory, vehicle or robot, to gather essential data to effectively perceive, navigate and interact with the real world. Using these microservices, developers can run large-scale te
LINK: https://blogs.nvidia.com/blog/ai-city-challenge-omniverse-cvpr/...
See more stories from nvidia

Most recent headlines

05/01/2027

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be demoed at CES 2026

Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...

07/10/2026

Dalet Flex LTS Delivers Smarter Media Operations from Ingest to Distribution

Dalet, a leading technology and service provider for media-rich organizations, today announced the latest Long-Term Supported (LTS) release of Dalet Flex. Build...

06/09/2026

Dolby and MagentaTV Bring Fans Closer to the FIFA World Cup 2026 in Germany with Dolby Vision and Dolby Atmos

June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...

05/08/2026

Thomson toolkit added to virtual hub built to strengthen climate storytelling

Thomson's Climate Crisis Toolkit for Media in Tanzania is being included in a new virtual library designed to ensure accurate and engaging storytelling arou...

05/08/2026

SVG Regional Sports Production Summit 2026: All Sessions Now Available to Watch on SVG PLAY

RSNs, teams, leagues, and streamers explore the present and future of local spor...

05/08/2026

SVG New Sponsor Spotlight: Hitachi Vantara's Lenny Khaitov on Building Resilient Data Infrastructure for Sports Production

As sports-production workflows generate larger volumes of unstructured data and ...

05/08/2026

NESN's Women's Celebration Night Returns With Expanded Focus on the Next Generation

Three aspiring broadcasters will contribute on-camera and inside the production ...

05/08/2026

Union County Paints an Authentic Picture of Recovery and Community

Adam Meeks attends the premiere of his film Union County, an official selection of the 2026 Sundance Film Festival. (Photo by Jemal Countess/Sundance Institut...

05/08/2026

Steinberg celebrate VST & HALion anniversaries

Major discounts, updates & remakes of early VST plug-ins Steinberg are currently celebrating two significant anniversaries, with this week marking the 30th ...

05/08/2026

KRK introduce the V Series Five

Flagship monitor range gains wireless remote control KRK's flagship studio monitor range has just been upgraded, with three new fifth-generation models ...

05/08/2026

Rohde & Schwarz launches next phase of Airways New Zealand Air Traffic Control system upgrade

Rohde & Schwarz launches next phase of Airways New Zealand Air Traffic Control s...

05/08/2026

Rohde & Schwarz presents the R&SMP007, a man-portable tactical direction finding system for seamless signal detection from 9 MHz to 8 GHz

Rohde & Schwarz presents the R&S MP007, a man-portable tactical direction findin...

05/08/2026

Australia's retirement sector advertising rises more than 10% as millions plan for life after work according to Nielsen

Advertisers invested $53.6 million in the retirement sector in the last 12 month...

05/08/2026

New Nielsen data shows IKEA's New Zealand launch attracted more than 243,000 shoppers in the last month

Nielsen CMI and Ad Intel data reveals who's visiting IKEA, where else they s...

05/08/2026

Nielsen expands Four-Screen Ad Deduplication across CTV in Japan

Media buyers and sellers can now compare deduplicated campaign reach across computer, mobile, connected TV and linear TV across all measurable CTV publishers in...

05/08/2026

Stranger Things,' Bluey,' The Pitt' Top Nielsen's Streaming Charts Through First Half of 2026

Stranger Things' Leads All Streaming Titles with 23.3 Billion Minutes in Fir...

05/08/2026

Live Media Group, Gravity Media Strike Deal on U.S. OB Fleets

Share Copy link Facebook X Linkedin Bluesky Email...

05/08/2026

Beyond VVC: Call for Proposals for Next-Gen Video Coding

Share Copy link Facebook X Linkedin Bluesky Email...

05/08/2026

QuickLink Expands North American Team With Appointment of...

QuickLink, the leading provider of award-winning video production and remote guest integration solutions, announces the appointment of Kane Peterson to its Nort...

05/08/2026

Seven in 10 UK advertisers exceed production budgets desp...

XR research reveals marketers believe they have control over production costs, yet budget overruns, campaign delays and creative cutbacks remain commonplace Se...

05/08/2026

Game On Live Studio Uses Blackmagic Design to Stream Tropical 7s

Game On Live Studio Uses Blackmagic Design to Stream Tropical 7s Brie Clayton August 4, 2026 0 Comments Every match of world's largest youth rugby...

05/08/2026

Chris Fitzgerald to Head TDC's Experiential Growth

Chris Fitzgerald to Head TDC's Experiential Growth Brie Clayton August 4, 2026 0 Comments Technical Direction Company promotes Chris Fitzgerald to...

05/08/2026

Gravity Media, Live Media Group Form Strategic Partnership

Share Copy link Facebook X Linkedin Bluesky Email...

05/08/2026

Samsung Launches World's First HDR10+ Streaming on Prime Video

Share Copy link Facebook X Linkedin Bluesky Email...

05/08/2026

Profuz Digital unveils new advanced search capabilities a...

Profuz Digital is set to launch a powerful upgrade to its Profuz LAPIS digital media management platform at IBC this year at Stand 8.A88. France Galop, Canal+ F...

05/08/2026

KRK Brings Advanced Wireless Control to Professional Moni...

KRK, a Gibson brand and leader in professional studio monitoring for nearly four decades, launches its new V Series Five Studio Monitors, the latest evolution o...

05/08/2026

Stable performance in line with plan - Magyar Telekoms H1 and Q2 2026 results

Network expansion, continued customer growth and stable financial performance characterized Magyar Telekoms business results for the first half and second quart...

05/08/2026

The Sky Foundation launches new bursary programme to support the future of UK dance

Wednesday 5 August 2026 The Sky Foundation launches new bursary programme to su...

05/08/2026

Charlie Sheen Launches Mostly Sheenius in Partnership with Red Seat Ventures

Charlie Sheen Launches Mostly Sheenius in Partnership with Red Seat Ventures After Decades as One of the World's Most Recognizable and Interviewed Celebr...

04/08/2026

MLB's Jason Hedgcock on Using the Cloud To Produce World Baseball Classic Games for Netflix Japan

Live production in the cloud is a growing trend across the industry, and Major L...

04/08/2026

Major League Soccer Names Larry Berg as Next Commissioner; Will Take Helm in 2027

LAFC co-managing owner will succeed Don Garber on Jan. 1, 2027, as league prepar...

04/08/2026

Foresight: Driving Growth Through Broadcast Flexibility - A Calrec + PGA TOUR Case Study

In 2022, when the PGA TOUR was breaking ground on its new, state-of-the-art broa...

04/08/2026

Dealer Cam Debuts as Omaha Productions Takes World Series of Poker Main Event Final Table Live on ESPN

Provided by POVORA, the new camera angle offers a dealer's-eye view of the c...

04/08/2026

USL To Trial FIFA's Football Video Support System During 2026 USL Cup

Spiideo will provide the camera setup, with NEP handling broadcast integration and off-site video operations...

04/08/2026

Game On Live Studio Deploys Blackmagic Design Workflow Across 15 Fields at Tropical 7s

Production team streamed more than 750 youth rugby matches using ATEM switchers,...

04/08/2026

NDI, KTC Partner To Develop Network-Native Commercial Displays

Agreement gives KTC early access to NDI's Advanced SDK for integration across commercial displays, digital signage, and interactive panels...

04/08/2026

IBC2026: IBC Innovation Awards Finalists Highlight Advances in AI, Cloud, Open Standards, and Accessibility

Twelve projects across four categories recognize collaborative work in productio...

04/08/2026

Ekstraklasa, Live Park Select Hawk-Eye for Tracking, Offside, and Broadcast Technology

Multi-year agreement will introduce SkeleTRACK, semi-automated offside, automate...

04/08/2026

Radio France Deploys Redundant Lawo AoIP Workflow for 2026 Concert de Paris

Three mc 56 consoles, dual A__UHD Core engines, RAVENNA networking, and HOME management supported live sound, monitoring, and international broadcast production...

04/08/2026

IBC2026: Haivision to Debut Makito ONE, Falkon X4 in Europe

New contribution products support ultra-low-latency video transport across fixed, remote, and mobile production environments...

04/08/2026

IBC2026: Chyron to Highlight AI, Cloud, HTML Graphics, and Sports-Analysis Tools

Company will showcase updates across PRIME, Chyron LIVE, Paint, virtual production, newsroom workflows, and venue control...

04/08/2026

TAB Show 2026: MultiDyne to Showcase Fiber-Transport and IP Infrastructure Tools

Exhibit will feature SilverBack, VersaBrix, VersaFrame, VF-9100, and FiberSaver-10G platforms...

04/08/2026

IBC2026: TAG Video Systems Adds MXL Support, Containerized Deployment, Expanded QC Tools

Platform updates target cloud-based production, high-density ST 2110 monitoring,...

04/08/2026

Gravity Media, Live Media Group Partner To Expand U.S. Mobile Production Facility Operations

Live Media Group will take control of Gravity Media's U.S. mobile-unit fleet...

04/08/2026

HeadRush update Prime, Core & Flex Core firmware

Amp-modelling rigs get major upgrade HeadRush have recently released a major firmware update that brings some powerful new features to their Prime, Core and...

04/08/2026

Am I Audio launch I Am Choir

Generate a choir from a single voice track Following years of development, Am I Audio have just officially launched their debut plug-in. Created in collabor...

04/08/2026

Universal Audio's EMT 140 & 250 plug-ins go native

Official EMT-endorsed reverbs now available to all Two more of Universal Audio's renowned UAD plug-ins have just gained new native versions that can be ...

04/08/2026

SBS helps Australians understand why every Census counts

SBS helps Australians understand why every Census counts Media releases As Australia prepares for the 2026 Census on Tuesday 11 August, SBS is helping ensur...

04/08/2026

Canada invests in the future of HF communications with Rohde & Schwarz solutions

Canada invests in the future of HF communications with Rohde & Schwarz solutions After selection by the Canadian Department of National Defence (DND) as the p...

04/08/2026

Blendkit Expands Beyond Blender

Blendkit Expands Beyond Blender Brie Clayton August 4, 2026 0 Comments Popular 3D Asset Library Launches Alpha Plugins for Maya, Godot, and Rhino Blend...