
Idea generation, not hardware or software, needs to be the bottleneck to the advancement of AI, Bryan Catanzaro, vice president of applied deep learning research at NVIDIA, said this week at the AI Hardware Summit.
We want the inventors, the researchers and the engineers that are coming up with future AI to be limited only by their own thoughts, Catanzaro told the audience.
Catanzaro leads a team of researchers working to apply the power of deep learning to everything from video games to chip design. At the annual event held in Silicon Valley, he described the work that NVIDIA is doing to enable advancements in AI, with a focus on large language modeling.
CUDA Is for the Dreamers Training and deploying large neural networks is a tough computational problem, so hardware that's both incredibly fast and highly efficient is a necessity, according to Catanzaro.
But, he explained, the software that accompanies that hardware might be even more important to unlocking further advancements in AI.
The core of the work that we do involves optimizing hardware and software together, all the way from chips, to systems, to software, frameworks, libraries, compilers, algorithms and applications, he said. We optimize all of these things to give transformational capabilities to scientists, researchers and engineers around the world.
This end-to-end approach yields chart-topping performance in industry-standard benchmarks, such as MLPerf. It also ensures that developers aren't constrained by the platform as they aim to advance AI.
CUDA is for the dreamers, CUDA is for the people who are thinking new thoughts, said Catanzaro. How do they think those thoughts and test them efficiently? They need something general and flexible, and that's why we build what we build.
Large Language Models Are Changing the World One of the most exciting areas of AI is language modeling, which is enabling groundbreaking applications in natural language understanding and conversational AI.
The complexity of large language models is growing at an incredible rate, with parameter counts doubling every two months.
A well-known example of a large and powerful language model is GPT-3, developed by OpenAI. Packing 175 billion parameters, it required 314 zettaflops (1021 floating point operations) to train.
It's a staggering amount of compute, Catanzaro said. And that means language modeling is now becoming constrained by economics.
Estimates suggest that GPT-3 would cost about $12 million to train and, Catanzaro observed, the rapid growth in model complexity means that, despite NVIDIA's tireless work to advance the performance and efficiency of its hardware and software, the cost to train these models is set to grow.
And, according to Catanzaro, this trend suggests that it might not be too long before a single model might require more than a billion dollars' worth of computer time to train.
What would it look like to build a model that took a billion dollars to train a single model? Well, it would need to reinvent an entire company, and you'd need to be able to use it in a lot of different contexts, Catanzaro explained.
Catanzaro expects that these models will unlock an incredible amount of value, inspiring continued innovation. During his talk, Catanzaro showed an example of the surprising capabilities of large language models to solve new tasks without being explicitly trained to do so.
After inputting just a few examples into a large language model - four sentences, with two written in English and their corresponding translations into Spanish - he then entered an English sentence, which the model then translated into Spanish properly.
The model was able to do this despite never being trained to do translation. Instead, it was trained - using, as Catanzaro described, an enormous amount of data from the internet - to predict the next word that should follow a given sequence of text.
To perform that very generic task, the model needed to come up with higher-level representations of concepts, such as the existence of languages in general, English and Spanish vocabularies and grammar, and the concept of a translation task, in order to understand the query and properly respond.
These language models are first steps towards generalized artificial intelligence with few shot learning, and that is enormously valuable and very exciting, explained Catanzaro.
A Full-Stack Approach to Language Modeling Catanzaro then went on to describe NVIDIA Megatron, a framework created by NVIDIA using PyTorch for efficiently training the world's largest, transformer-based language models.
A key feature of NVIDIA Megatron, which Catanzaro notes has already been used by various companies and organizations to train large transformer-based models, is model parallelism.
Megatron supports both inter-layer (pipeline) parallelism, which allows different layers of a model to be processed on different devices, as well as intra-layer (tensor) parallelism, which allows a single layer to be processed by multiple different devices.
Catanzaro further described some of the optimizations that NVIDIA applies to maximize the efficiency of pipeline parallelism and minimize so-called pipeline bubbles, during which a GPU is not performing useful work.
A batch is split into microbatches, the execution of which is pipelined. This boosts the utilization of the GPU resources in a system during training. With further optimizations, pipeline bubbles can be reduced even more.
Catanzaro described an optimization, recently published, that entails round-robining each (pipeline) stage among multiple GPUs so that we can further reduce the amount of pipeline bubble overhead in this schedule.
Although this optimization puts additional stress on the communication fabric within the system, Catanzaro showed that, by l
Most recent headlines
05/01/2027
Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...
16/11/2026
November 16, 2026
Hitachi system enables proton beams to be delivered from 360 degrees around the patient, expanding treatment capacity and supporting more pre...
16/11/2026
Share
{alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...
16/11/2026
Share
{alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...
16/11/2026
Share
{alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...
16/11/2026
Share
{alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...
16/11/2026
Share
{alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...
16/11/2026
Share
{alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...
16/11/2026
Share
{alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...
16/11/2026
Share
{alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...
16/11/2026
Hiltron Communications has received a repeat order from one of Europes longest established communications satellite operators. Central to this project is a Hilt...
16/11/2026
Fabric Insights and Fonn Group today announce a partnership to connect editorial planning and media operations across their complementary product portfolios.
...
16/11/2026
Density 1.0 brings a complete color grading workflow to Final Cut Pro
Brie Clayton November 16, 2026
0 Comments
Filmmaker Pascal Basel's plugin pr...
16/11/2026
Berklee Students to Perform at Boston's CITGO Sign Relighting Kuper Walker, Ollie Mann, and dreamcorp. will perform at the free public celebration in Kenm...
16/11/2026
The Hollywood Reporter Names Berklee Among World's Best Music Schools The annual list spotlights Berklee's screen scoring programs and new master'...
16/11/2026
Five years ago, Lightware launched the original Taurus UCX with a clear vision: USB-C would become the central connection in meeting rooms. The goal was simple:...
16/11/2026
Sonniss: bundles over 80% below their libraries' prices, plus up to 60% off ...
16/11/2026
Singaporean Comedy Luck My Life Shot with Blackmagic URSA Cine 17K 65
Brie Clayton November 16, 2026
0 Comments
Blackmagic PYXIS 12K and DaVinci Resol...
16/11/2026
As countries gather to discuss climate targets at COP 31 this week, one question that is at the heart of debates is how to measure the environmental impact of t...
16/11/2026
Announced today at SC26, NVIDIA Rubin HPC is a new GPU in the NVIDIA Vera Rubin ...
16/11/2026
This week at SC26, NVIDIA is marking two decades of CUDA by honoring the researchers, developers and engineers who helped make GPU computing a foundation for mo...
16/11/2026
With expanded NVIDIA CUDA-X libraries, including cuPhoton, cuPotential and cuSpa, researchers can sift through vast amounts of data to accelerate research, from...
16/11/2026
In its MLPerf benchmarks debut, with results released today, the NVIDIA Vera Rub...
16/11/2026
Quantum processor deployments at supercomputing centers have grown nearly sevenfold over the past five years, with nearly 30 supercomputing centers around the w...
16/11/2026
To train an AI model, thousands of GPUs have to work together, requiring a network that's fast, efficient and resilient.
Lambda is tackling the network pie...
15/11/2026
Mixers, interfaces & DAW controller receive new firmware
Yamaha have just launched a trio of firmware updates that bring some powerful new functionality to ...
15/11/2026
Can VideoProc AI Really Reduce Noise and Enhance Blurry, Old Footage? Real-World...
15/11/2026
Over two days at COEX, more than 2,000 attendees converged for AI Day Seoul, bri...
14/11/2026
Longtime MLB Network employee David Maybrown passed away Thursday at the age of 57. David joined MLB Network in 2009 shortly after its launch, beginning as an i...
14/11/2026
Modular innovators introduce two guitar-based products
ADDAC System's recently introduced Four String Series makes it possible to integrate physical str...
14/11/2026
Marcelo Dai's Live Session Captured with URSA Cine Immersive
Brie Clayton November 14, 2026
0 Comments
Artist takes viewers on an immersive journe...
14/11/2026
Share
{alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...
14/11/2026
Share
{alert(Link copied)}).catch(err => {console.error(Failed to copy: , err)}) data-platform=copy_link data-action= class=font-ui-heading py-2 text-sm tex...
14/11/2026
Newest Lenses, Horizon Anamorphic Seminar & Guided Lens Character Experience Highlight Creative Possibilities
At the 2026 Toru CAMERIMAGE Film Festival, ZEISS...
14/11/2026
Sonnet Announces Black Friday Sale on Select Products
Brie Clayton November 14, 2026
0 Comments
WHAT: Sonnet Technologies is having a sale on a select...
14/11/2026
Berklee Alumni Shine at the 2026 Latin Grammy Awards This year's winners included songwriters, engineers, and performers who contributed to acclaimed reco...
13/11/2026
The UIM E1 Series wraps its third season this week with the Bombay Sapphire E1 Miami GP, Nov. 11-14, on Biscayne Bay at the Miami Seaplane Base on Watson Island...
13/11/2026
In-venue and creative video staffers at the professional and collegiate level ha...
13/11/2026
Singer-songwriter Benson Boone will headline entertainment for the Discover NHL Winter Classic between the Utah Mammoth and Colorado Avalanche at Rice-Eccles St...
13/11/2026
Blackmagic Design has announced the Blackmagic Ambisonic Microphone, a 7th order ambisonic microphone array that captures spatial audio for immersive video prod...
13/11/2026
SMPTE has announced that The Studio B&H and Riedel Communications will provide the video infrastructure and communications technology for the Solutions Hub at t...
13/11/2026
The Hollywood Professional Association (HPA) has announced the theme and program focus for the 2027 HPA Tech Retreat, which takes place Feb. 14-18, 2027, in La ...
13/11/2026
Football Manager 27 Touch is available on Apple Arcade starting today. The game ...
13/11/2026
MotoGP has announced a two-year media rights agreement with ESPN covering the Un...
13/11/2026
Apple Sports, the free iPhone app, has added the Ultimate Fighting Championship (UFC) to its lineup of supported sports, in time for Saturday's UFC 334. The...
13/11/2026
The NBA and Fanatics Studios, the joint venture between Fanatics and entertainme...
13/11/2026
ESPN held its annual EDGE Innovation Conference last night and offered up an ove...
13/11/2026
Regional sports network gives Ithaca College, SUNY Cortland students a platform to thrive
One of the biggest games in Division III college football will be pla...
13/11/2026
Studer-inspired 500-series PWM compressor unveiled
The HLabs range already includes a couple of Studer-inspired offerings in the form of the PRE169 and EQ16...