
Idea generation, not hardware or software, needs to be the bottleneck to the advancement of AI, Bryan Catanzaro, vice president of applied deep learning research at NVIDIA, said this week at the AI Hardware Summit.
We want the inventors, the researchers and the engineers that are coming up with future AI to be limited only by their own thoughts, Catanzaro told the audience.
Catanzaro leads a team of researchers working to apply the power of deep learning to everything from video games to chip design. At the annual event held in Silicon Valley, he described the work that NVIDIA is doing to enable advancements in AI, with a focus on large language modeling.
CUDA Is for the Dreamers Training and deploying large neural networks is a tough computational problem, so hardware that's both incredibly fast and highly efficient is a necessity, according to Catanzaro.
But, he explained, the software that accompanies that hardware might be even more important to unlocking further advancements in AI.
The core of the work that we do involves optimizing hardware and software together, all the way from chips, to systems, to software, frameworks, libraries, compilers, algorithms and applications, he said. We optimize all of these things to give transformational capabilities to scientists, researchers and engineers around the world.
This end-to-end approach yields chart-topping performance in industry-standard benchmarks, such as MLPerf. It also ensures that developers aren't constrained by the platform as they aim to advance AI.
CUDA is for the dreamers, CUDA is for the people who are thinking new thoughts, said Catanzaro. How do they think those thoughts and test them efficiently? They need something general and flexible, and that's why we build what we build.
Large Language Models Are Changing the World One of the most exciting areas of AI is language modeling, which is enabling groundbreaking applications in natural language understanding and conversational AI.
The complexity of large language models is growing at an incredible rate, with parameter counts doubling every two months.
A well-known example of a large and powerful language model is GPT-3, developed by OpenAI. Packing 175 billion parameters, it required 314 zettaflops (1021 floating point operations) to train.
It's a staggering amount of compute, Catanzaro said. And that means language modeling is now becoming constrained by economics.
Estimates suggest that GPT-3 would cost about $12 million to train and, Catanzaro observed, the rapid growth in model complexity means that, despite NVIDIA's tireless work to advance the performance and efficiency of its hardware and software, the cost to train these models is set to grow.
And, according to Catanzaro, this trend suggests that it might not be too long before a single model might require more than a billion dollars' worth of computer time to train.
What would it look like to build a model that took a billion dollars to train a single model? Well, it would need to reinvent an entire company, and you'd need to be able to use it in a lot of different contexts, Catanzaro explained.
Catanzaro expects that these models will unlock an incredible amount of value, inspiring continued innovation. During his talk, Catanzaro showed an example of the surprising capabilities of large language models to solve new tasks without being explicitly trained to do so.
After inputting just a few examples into a large language model - four sentences, with two written in English and their corresponding translations into Spanish - he then entered an English sentence, which the model then translated into Spanish properly.
The model was able to do this despite never being trained to do translation. Instead, it was trained - using, as Catanzaro described, an enormous amount of data from the internet - to predict the next word that should follow a given sequence of text.
To perform that very generic task, the model needed to come up with higher-level representations of concepts, such as the existence of languages in general, English and Spanish vocabularies and grammar, and the concept of a translation task, in order to understand the query and properly respond.
These language models are first steps towards generalized artificial intelligence with few shot learning, and that is enormously valuable and very exciting, explained Catanzaro.
A Full-Stack Approach to Language Modeling Catanzaro then went on to describe NVIDIA Megatron, a framework created by NVIDIA using PyTorch for efficiently training the world's largest, transformer-based language models.
A key feature of NVIDIA Megatron, which Catanzaro notes has already been used by various companies and organizations to train large transformer-based models, is model parallelism.
Megatron supports both inter-layer (pipeline) parallelism, which allows different layers of a model to be processed on different devices, as well as intra-layer (tensor) parallelism, which allows a single layer to be processed by multiple different devices.
Catanzaro further described some of the optimizations that NVIDIA applies to maximize the efficiency of pipeline parallelism and minimize so-called pipeline bubbles, during which a GPU is not performing useful work.
A batch is split into microbatches, the execution of which is pipelined. This boosts the utilization of the GPU resources in a system during training. With further optimizations, pipeline bubbles can be reduced even more.
Catanzaro described an optimization, recently published, that entails round-robining each (pipeline) stage among multiple GPUs so that we can further reduce the amount of pipeline bubble overhead in this schedule.
Although this optimization puts additional stress on the communication fabric within the system, Catanzaro showed that, by l
Most recent headlines
06/10/2025
France T l visions, France's leading broadcaster, has received the 2025 EBU ...
04/09/2025
Monumental Sports & Entertainment (MSE), in collaboration with Dalet, has been a...
15/06/2025
July 2025 in Dublin, Berlin, Amsterdam & London
Photo: Thea Martre
Music Production for Women (MPW) have announced that they will be running a series of fo...
15/06/2025
Composer/producer launches free virtual instruments
Sulcata Sound is the latest venture of Jason Graves, a two-time British Academy Award-winnning composer,...
14/06/2025
NEW YORK Pluto TV and the All Womens Sports Network have launched a free ad-supported streaming TV (FAST) AWSN channel in the U.S., Canada, the U.K. and the Nor...
14/06/2025
NEW YORK and CINCINNATI E.W. Scripps has announced a new, multiyear agreement with the WNBA that will continue Ions regular-season coverage of the league on Fri...
14/06/2025
WASHINGTON The National Association of Broadcasters highlighted the hidden importance of spectrum in the production of major sporting events and described wha...
14/06/2025
WASHINGTON Sunsetting ATSC 1.0, expanding business opportunities for NextGen Broadcast and increasing international adoption of the ATSC 3.0 standard were top o...
14/06/2025
SAN FRANCISCO Samba TV and Acxiom have announced that they will dramatically expand their longstanding relationship....
14/06/2025
July 2025 in Dublin, Berlin, Amsterdam & London
Photo: Thea Martre
Music Production for Women (MPW) have announced that they will be running a series of fo...
14/06/2025
San Francisco State University's School of Cinema Uses Blackmagic Design
Brie Clayton June 13, 2025
0 Comments
More than 40 Blackmagic Design came...
14/06/2025
Boris FX Mocha Pro Adds New AI Tools To Tackle VFX Tasks Fast
Jessie Electa Petrov June 13, 2025
0 Comments
The 2025.5 release helps artists work more...
14/06/2025
AJA Debuts DRM2-Plus Mini-Converter Frame at InfoComm 2025
Brie Clayton June 13, 2025
0 Comments
Next-gen frame addresses diverse rackmount needs wit...
13/06/2025
(L-R) Lindsay Utz, Michelle Walshe, and The Right Honourable Dame Jacinda Ardern attend the 2025 Sundance Film Festival premiere of Prime Minister at Eccles T...
13/06/2025
Photo credit: Atsushi Nishijima
If you're a true lover of rom-coms, chances...
13/06/2025
Pure Drama and Fierce Rivalries set to dominate the world's most iconic spor...
13/06/2025
Johannesburg, 12 June 2025 - The National Film and Video Foundation (NFVF), an a...
13/06/2025
ABILENE. Texas A severe storm knocked down the tower and severely damaged the news studio and main facility of Sinclair-owned KTXS here on Sunday, June 8....
13/06/2025
Berklee's Music Business/Management Department Recognized by the Music Biz A...
13/06/2025
WASHINGTON The ATSC, the Broadcast Standards Association, honored veteran technologist Aldo Cugnini and Clarence Hau, Senior Vice President of Standards, Policy...
13/06/2025
(Editor's note: The 2025 UFL Championship Game between the D.C. Defenders and Michigan Panthers kicks off Saturday, June 14, at 8 p.m. Eastern. The game wil...
13/06/2025
New iPad/iPhone synth App announced
Following on from last year's release of Gradient Synth - which reached #6 on the App Store's Paid Music charts ...
13/06/2025
LONDON Warner Bros. Discovery has announced that HBO Max will launch direct-to-consumer in multiple new countries this July as the streamer becomes available in...
13/06/2025
AI voice transcription and captioning platform Verbit has added a new feature to its Captivate ASR solution the ability to identify specific features in automat...
13/06/2025
WASHINGTON Federal Communications Commission member Anna Gomez has wrapped up two weeks in California visiting broadcasters, television studio executives, enter...
13/06/2025
WASHINGTON The U.S. House of Representatives voted mostly along party lines to approve a rescission package that would cancel $9.4 billion in previously approve...
13/06/2025
At InfoComm 2025, AJA Video Systems announced DRM2-Plus, an intuitive, high-capacity 3RU frame that can neatly house up to 24 AJA Mini-Converters. Tailored to s...
13/06/2025
Cinema advertising leader to leverage AOS and suite of AI-enabled solutions to optimize forecasting, yield management, and streamlined ad sales and operations a...
13/06/2025
Manfrotto has launched the ONE Hybrid Tripod, a new support system designed specifically for professional content creators working with mirrorless cameras acros...
13/06/2025
Leading video software provider, Synamedia, today announced that its Media Edge Gateway (MEG), an ATSC 3.0 software-based IRD, now supports Device Security requ...
13/06/2025
LiveU, the global leader in live IP-video contribution, production and distribution solutions, is deepening its commitment to the German-speaking market with th...
13/06/2025
Chaos, the leader in architectural visualisation software, today announces Chaos Corona 13, giving archviz designers new ways to add eye-catching style and flai...
13/06/2025
PALI's Nena Music Video Shot with Blackmagic Design
Brie Clayton June 12, 2025
0 Comments
Blackmagic Cinema Camera 6K and DaVinci Resolve Studio b...
13/06/2025
OddBeast Powers Up iRobot's Newest Roombas with Suite of CGI Launch Assets
Brie Clayton June 12, 2025
0 Comments
The motion design and production ...
13/06/2025
On Chick Coreas Birthday, a Newly Uncovered Archival Release The Visitors, composed by Corea and performed by vibraphonist Gary Burton and pianist Kirill Gers...
13/06/2025
In fulfilment of a recommendation by the Government's Expert Advisory Commit...
13/06/2025
SVG Sit-Down: Backblaze's Gleb Budman Talks Products, Partnerships, and the ...
13/06/2025
SVG Sit-Down: DAZN's Walker Jacobs Calls Streaming the FIFA Club World Cup ...
13/06/2025
New Sponsor Spotlight: Vecima Networks' Paul Strickland on How Improving QoE...
13/06/2025
Pitch Perspective: Where's Next for Specialty Cameras in Soccer? Leaders from Sky Austria and ACS discuss the possibilities of camera placement pitchside B...
13/06/2025
Premiership Rugby Final 2025: Vintage clash between Bath and Leicester gets full...
13/06/2025
Premiership Rugby Final 2025: TNT Sports gears up for Bath vs Leicester battle w...
13/06/2025
NCAA Men's College World Series: ESPN Adds Two-Point SupraCam, Invests in Ne...
13/06/2025
New FSWX signal and spectrum analyzer with novel architecture overcomes limits o...
13/06/2025
Apple today announced the addition of iPad to Self Service Repair, providing iPad owners with access to repair manuals, genuine Apple parts, Apple Diagnostics t...
13/06/2025
CUPERTINO, CALIFORNIA Apple today previewed iOS 26, a major update that brings a beautiful new design, intelligent experiences, and improvements to the apps use...
13/06/2025
At Apple's Worldwide Developers Conference (WWDC), Apple unveiled Apple Games, an all-new destination designed to help players jump back into the games they...
13/06/2025
Industrial AI isn't slowing down. Germany is ready.
Following London Tech Week and GTC Paris at VivaTech, NVIDIA founder and CEO Jensen Huang's Europea...
12/06/2025
In 2018, Spotify launched Heart & Soul, a mental health initiative developed to ...
12/06/2025
50 Years Strong: SBS and NITV Supercharge NAIDOC Week 2025 in a joint 50th celeb...