Idea generation, not hardware or software, needs to be the bottleneck to the advancement of AI, Bryan Catanzaro, vice president of applied deep learning research at NVIDIA, said this week at the AI Hardware Summit.
We want the inventors, the researchers and the engineers that are coming up with future AI to be limited only by their own thoughts, Catanzaro told the audience.
Catanzaro leads a team of researchers working to apply the power of deep learning to everything from video games to chip design. At the annual event held in Silicon Valley, he described the work that NVIDIA is doing to enable advancements in AI, with a focus on large language modeling.
CUDA Is for the Dreamers Training and deploying large neural networks is a tough computational problem, so hardware that's both incredibly fast and highly efficient is a necessity, according to Catanzaro.
But, he explained, the software that accompanies that hardware might be even more important to unlocking further advancements in AI.
The core of the work that we do involves optimizing hardware and software together, all the way from chips, to systems, to software, frameworks, libraries, compilers, algorithms and applications, he said. We optimize all of these things to give transformational capabilities to scientists, researchers and engineers around the world.
This end-to-end approach yields chart-topping performance in industry-standard benchmarks, such as MLPerf. It also ensures that developers aren't constrained by the platform as they aim to advance AI.
CUDA is for the dreamers, CUDA is for the people who are thinking new thoughts, said Catanzaro. How do they think those thoughts and test them efficiently? They need something general and flexible, and that's why we build what we build.
Large Language Models Are Changing the World One of the most exciting areas of AI is language modeling, which is enabling groundbreaking applications in natural language understanding and conversational AI.
The complexity of large language models is growing at an incredible rate, with parameter counts doubling every two months.
A well-known example of a large and powerful language model is GPT-3, developed by OpenAI. Packing 175 billion parameters, it required 314 zettaflops (1021 floating point operations) to train.
It's a staggering amount of compute, Catanzaro said. And that means language modeling is now becoming constrained by economics.
Estimates suggest that GPT-3 would cost about $12 million to train and, Catanzaro observed, the rapid growth in model complexity means that, despite NVIDIA's tireless work to advance the performance and efficiency of its hardware and software, the cost to train these models is set to grow.
And, according to Catanzaro, this trend suggests that it might not be too long before a single model might require more than a billion dollars' worth of computer time to train.
What would it look like to build a model that took a billion dollars to train a single model? Well, it would need to reinvent an entire company, and you'd need to be able to use it in a lot of different contexts, Catanzaro explained.
Catanzaro expects that these models will unlock an incredible amount of value, inspiring continued innovation. During his talk, Catanzaro showed an example of the surprising capabilities of large language models to solve new tasks without being explicitly trained to do so.
After inputting just a few examples into a large language model - four sentences, with two written in English and their corresponding translations into Spanish - he then entered an English sentence, which the model then translated into Spanish properly.
The model was able to do this despite never being trained to do translation. Instead, it was trained - using, as Catanzaro described, an enormous amount of data from the internet - to predict the next word that should follow a given sequence of text.
To perform that very generic task, the model needed to come up with higher-level representations of concepts, such as the existence of languages in general, English and Spanish vocabularies and grammar, and the concept of a translation task, in order to understand the query and properly respond.
These language models are first steps towards generalized artificial intelligence with few shot learning, and that is enormously valuable and very exciting, explained Catanzaro.
A Full-Stack Approach to Language Modeling Catanzaro then went on to describe NVIDIA Megatron, a framework created by NVIDIA using PyTorch for efficiently training the world's largest, transformer-based language models.
A key feature of NVIDIA Megatron, which Catanzaro notes has already been used by various companies and organizations to train large transformer-based models, is model parallelism.
Megatron supports both inter-layer (pipeline) parallelism, which allows different layers of a model to be processed on different devices, as well as intra-layer (tensor) parallelism, which allows a single layer to be processed by multiple different devices.
Catanzaro further described some of the optimizations that NVIDIA applies to maximize the efficiency of pipeline parallelism and minimize so-called pipeline bubbles, during which a GPU is not performing useful work.
A batch is split into microbatches, the execution of which is pipelined. This boosts the utilization of the GPU resources in a system during training. With further optimizations, pipeline bubbles can be reduced even more.
Catanzaro described an optimization, recently published, that entails round-robining each (pipeline) stage among multiple GPUs so that we can further reduce the amount of pipeline bubble overhead in this schedule.
Although this optimization puts additional stress on the communication fabric within the system, Catanzaro showed that, by l
Most recent headlines
05/01/2027
Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...
06/09/2026
June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...
04/08/2026
Dalet, a leading technology and service provider for media-rich organizations, t...
04/07/2026
When Canada qualified for the FIFA World Cup 2026, OneSoccer knew this wasn'...
04/07/2026
Celebrates company's 80th anniversary
Rhodes have recently revealed that they will be producing a limited run of electric pianos in celebration of their...
04/07/2026
Blackmagic Design Cameras Empower Youth Broadcasting Program
Brie Clayton July 3, 2026
0 Comments
Blackmagic Pocket Cinema Camera 6K Pro and Blackmagi...
04/07/2026
iZotope Joins Boris FX
Boris Yamnitsky July 3, 2026
0 Comments
iZotope, the team behind RX and Ozone, is joining Boris FX. A letter from founder Boris...
04/07/2026
April 7 2026, 19:00 (PDT) Detective Conan: Fallen Angel of the Highway Opens in...
03/07/2026
New bundle & three Single Packs
Continuing to expand their already sizeable orchestral collection, VSL's latest release introduces three new Single Pack...
03/07/2026
Venue installs new ATC-equipped control room
LSO St Luke's, a unique music venue that also serves as the home of the London Symphony Orchestra, have rec...
03/07/2026
Caden Pearson appointed new Commissioning Editor for NITV
3 July, 2026
Media releases
National Indigenous Television (NITV) has strengthened its commitment...
03/07/2026
1 February 2023
SHARE Facebook Twitter Linkedin Email
Munich, Germany, 1st February 2023: Cinegy GmbH, the premier provider of software technology for digit...
03/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
03/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
03/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
03/07/2026
Death Star
Andy Marken July 2, 2026
0 Comments
Look, I can't get involved. I've got work to do. It's not that I like the Empire; I hate i...
03/07/2026
Berklee's New Visual Identity: Honoring Our History, Building for What's...
03/07/2026
Scripps Research scientists awarded $2M to advance global disease surveillance Two Gates Foundation grants will expand wastewater surveillance and AI-driven dis...
03/07/2026
Joan Pulupa joins Scripps Research faculty to study the organization of DNA in brain cells and its links to neurodegeneration Using smell-sensing neurons and ad...
02/07/2026
Entering her senior year, this hometown girl is paving a career in live sports production gaining experience in replay and audio and as a TD
In the live-sports...
02/07/2026
In-venue and creative video staffers at the professional and collegiate level ha...
02/07/2026
BLAST, a competitive entertainment company focused on esports, has announced more than $133 million in revenue for 2025, representing more than 40% year-over-ye...
02/07/2026
Riedel Communications has announced official SKAARHOJ panel support for SimplyLive production workflows, enabled through the SimplyLive 2.1 release. The integra...
02/07/2026
The Fire Rescue Service of the Czech Republic has deployed LiveU video-over-bond...
02/07/2026
Gravity Media USA has announced the appointment of Brittney Boston as Head of Business Development, effective July 1, 2026. Based in Nashville, Tennessee, Bosto...
02/07/2026
TwelveLabs, a video intelligence company, has announced $100 million in Series B funding co-led by NEA and NAVER Ventures, with participation from Amazon, Radic...
02/07/2026
The Pro Padel League (PPL) has announced a broadcast partnership with USA Sports that will air five PPL championship matches on CNBC during the 2026 season, the...
02/07/2026
LiveLike, a digital fan engagement platform, has announced eight confirmed FIFA ...
02/07/2026
Cobalt Digital has received Future's Best of Show Award, presented by AV Technology at InfoComm 2026, for its blueCORE family of standalone signal processor...
02/07/2026
Synamedia has announced the appointment of Dr. Tzvi Gerstl as Chief Executive Officer. Paul Segre, who has served as CEO for the past six years, will transition...
02/07/2026
The Esports Foundation (EF) and Sony Group Corporation have announced an expanded collaboration for the Esports World Cup 2026 (EWC), taking place in Paris, Fra...
02/07/2026
Zee Entertainment Enterprises Ltd. ( Z') has announced exclusive broadcast and digital rights for the Bundesliga in India for five years, beginning with the...
02/07/2026
An effort uniting News, Sports, Local, and Telemundo, the 50+-camera live produc...
02/07/2026
Zoey Deutch, John Slattery, Ken Marino, Miles Gutierrez-Riley, and Ben Wang appe...
02/07/2026
Stammering, stuttering, strangulated tones
The Crow Hill Company's latest creation promises to be the most original sound set they've produced to d...
02/07/2026
A new era in unmixing and spectral editing
The latest version of Steinberg's spectral audio-editing software has just arrived, building on the strength...
02/07/2026
Aims to simplify additive synthesis
Sine Machine is the debut launch from Melatonin, a Vienna-based developer who have spent the past six years creating wha...
02/07/2026
Products to remain fully active & supported
Following the news of Native Instruments joining the inMusic brand line-up, Academy and Emmy Award-winning visua...
02/07/2026
What you missed!
Last weekend, Saturday 27 June 2026, saw the debut of Sound On Sounds new GearExpo UK event, the largest dedicated pro-audio event to take ...
02/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
02/07/2026
Following the successful launch of its inaugural APAC Mentoring Programme last month, the Rise AV APAC Regional Council will bring the conversation around mento...
02/07/2026
Blackmagic PYXIS 6K Used to Shoot Director Takahisa Zeze's Cry Out
Brie Clayton July 2, 2026
0 Comments
Highly mobile camera supports tense and de...
02/07/2026
Broadcast Solutions acquires BFE, expanding its lead in European broadcast, medi...
02/07/2026
Berklee Alum and Faculty Perform at Boston Public Library's 250th Anniversar...
02/07/2026
Broadcast Solutions GmbH, a leading systems integrator and provider of innovative solutions for the broadcast media industry, is acquiring BFE Studio und Medien...
02/07/2026
Cinegy GmbH, the premier provider of software-defined television technology, has extended the ingest facility at leading Brazilian sports company LiveMode, work...
02/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
02/07/2026
Standalone processors acknowledged for the innovation and value they bring to Pro AV
Cobalt Digital, a leading designer and manufacturer of signal processing ...
02/07/2026
Synamedia announced today the appointment of Dr Tzvi Gerstl as Chief Executive Officer. Paul Segre, who has served as CEO for the past six years, will transitio...
Idea generation, not hardware or software, needs to be the bottleneck to the advancement of AI, Bryan Catanzaro, vice president of applied deep learning research at NVIDIA, said this week at the AI Hardware Summit.
