Idea generation, not hardware or software, needs to be the bottleneck to the advancement of AI, Bryan Catanzaro, vice president of applied deep learning research at NVIDIA, said this week at the AI Hardware Summit.
We want the inventors, the researchers and the engineers that are coming up with future AI to be limited only by their own thoughts, Catanzaro told the audience.
Catanzaro leads a team of researchers working to apply the power of deep learning to everything from video games to chip design. At the annual event held in Silicon Valley, he described the work that NVIDIA is doing to enable advancements in AI, with a focus on large language modeling.
CUDA Is for the Dreamers Training and deploying large neural networks is a tough computational problem, so hardware that's both incredibly fast and highly efficient is a necessity, according to Catanzaro.
But, he explained, the software that accompanies that hardware might be even more important to unlocking further advancements in AI.
The core of the work that we do involves optimizing hardware and software together, all the way from chips, to systems, to software, frameworks, libraries, compilers, algorithms and applications, he said. We optimize all of these things to give transformational capabilities to scientists, researchers and engineers around the world.
This end-to-end approach yields chart-topping performance in industry-standard benchmarks, such as MLPerf. It also ensures that developers aren't constrained by the platform as they aim to advance AI.
CUDA is for the dreamers, CUDA is for the people who are thinking new thoughts, said Catanzaro. How do they think those thoughts and test them efficiently? They need something general and flexible, and that's why we build what we build.
Large Language Models Are Changing the World One of the most exciting areas of AI is language modeling, which is enabling groundbreaking applications in natural language understanding and conversational AI.
The complexity of large language models is growing at an incredible rate, with parameter counts doubling every two months.
A well-known example of a large and powerful language model is GPT-3, developed by OpenAI. Packing 175 billion parameters, it required 314 zettaflops (1021 floating point operations) to train.
It's a staggering amount of compute, Catanzaro said. And that means language modeling is now becoming constrained by economics.
Estimates suggest that GPT-3 would cost about $12 million to train and, Catanzaro observed, the rapid growth in model complexity means that, despite NVIDIA's tireless work to advance the performance and efficiency of its hardware and software, the cost to train these models is set to grow.
And, according to Catanzaro, this trend suggests that it might not be too long before a single model might require more than a billion dollars' worth of computer time to train.
What would it look like to build a model that took a billion dollars to train a single model? Well, it would need to reinvent an entire company, and you'd need to be able to use it in a lot of different contexts, Catanzaro explained.
Catanzaro expects that these models will unlock an incredible amount of value, inspiring continued innovation. During his talk, Catanzaro showed an example of the surprising capabilities of large language models to solve new tasks without being explicitly trained to do so.
After inputting just a few examples into a large language model - four sentences, with two written in English and their corresponding translations into Spanish - he then entered an English sentence, which the model then translated into Spanish properly.
The model was able to do this despite never being trained to do translation. Instead, it was trained - using, as Catanzaro described, an enormous amount of data from the internet - to predict the next word that should follow a given sequence of text.
To perform that very generic task, the model needed to come up with higher-level representations of concepts, such as the existence of languages in general, English and Spanish vocabularies and grammar, and the concept of a translation task, in order to understand the query and properly respond.
These language models are first steps towards generalized artificial intelligence with few shot learning, and that is enormously valuable and very exciting, explained Catanzaro.
A Full-Stack Approach to Language Modeling Catanzaro then went on to describe NVIDIA Megatron, a framework created by NVIDIA using PyTorch for efficiently training the world's largest, transformer-based language models.
A key feature of NVIDIA Megatron, which Catanzaro notes has already been used by various companies and organizations to train large transformer-based models, is model parallelism.
Megatron supports both inter-layer (pipeline) parallelism, which allows different layers of a model to be processed on different devices, as well as intra-layer (tensor) parallelism, which allows a single layer to be processed by multiple different devices.
Catanzaro further described some of the optimizations that NVIDIA applies to maximize the efficiency of pipeline parallelism and minimize so-called pipeline bubbles, during which a GPU is not performing useful work.
A batch is split into microbatches, the execution of which is pipelined. This boosts the utilization of the GPU resources in a system during training. With further optimizations, pipeline bubbles can be reduced even more.
Catanzaro described an optimization, recently published, that entails round-robining each (pipeline) stage among multiple GPUs so that we can further reduce the amount of pipeline bubble overhead in this schedule.
Although this optimization puts additional stress on the communication fabric within the system, Catanzaro showed that, by l
Most recent headlines
04/08/2024
Dalet, a leading technology and service provider for media-rich organizations, is excited to announce Santiago Solanas as its new Chief Executive Officer (CEO)....
03/06/2024
Dalet, a leading technology and service provider for media-rich organizations, a...
29/04/2024
L3Harris is delivering manpack radios to U.S. Army CH-47 Chinooks as part of the Air-to-Ground Networking Radio program, providing seamless, resilient communica...
29/04/2024
He writes in Defense One: Despite the near-universal acknowledgement throughout the U.S. government and defense industrial base of the criticality of resilient ...
29/04/2024
eds3_5_jq(document).ready(function($) { $(#eds_sliderM519).chameleonSlider_2_1({ content_source:......
29/04/2024
Codemill aims to revolutionize media workflow efficiency at this years NAB Show by introducing Just-In-Time (JIT) playback technology in Accurate.Video Validate...
29/04/2024
April 29, 2024
-- TF1 Chooses Broadpeak to Power Targeted Advertising for New...
29/04/2024
LOS ANGELES Allen Media Group (AMG) has partnered with LG Electronics to bring 223 Local Now FAST channels to LG's free streaming service, LG Channels, avai...
29/04/2024
OTTAWA Ross Video has announced the introduction of Raiden, a data-driven weather graphics software that combines data gathering, processing, and visualization ...
29/04/2024
Panalux, a leading rental provider of lighting and power solutions for the motion-picture industry and part of Panavision's end-to-end service offerings for...
29/04/2024
Cobalt Iron Inc., a leading provider of SaaS-based enterprise data protection, today announced that it has received a patent on its technology for dynamic autho...
29/04/2024
SDVI, the leading platform provider for cloud-native media supply chains, today announced that Rally Access Workstation, a fully managed solution for editing in...
29/04/2024
Premier Sports, a premium sports broadcaster, has selected QuickLink's Remote Commentary solution for introducing professional, high-quality remote commenta...
29/04/2024
Glookast has chosen Jigsaw24 Media as the only UK channel partner to represent their portfolio of ingest and workflow optimisation products. The agreement, sign...
29/04/2024
Clear-Com played a pivotal role in the seamless coverage of the recent solar eclipse on April 8, 2024. Leveraging its cutting-edge Eclipse HX Digital Matrix i...
29/04/2024
PlayBox Neo will promote its complete range of television channel management, graphic branding and playout solutions to EMEA region media content owners and bro...
29/04/2024
Lingo, RuPaul's word-twisting game show, returns for season two on CBS Friday, May 24. Two episodes air that night, and stream on Paramount Plus, too....
29/04/2024
ESPN and Amazon Prime Video are reportedly close to scoring television rights to the National Basketball Association, according to published reports....
29/04/2024
Judge Judy and Hot Bench, CBS Media Ventures' genre-leading court shows, have been renewed through the 2025-26 TV season in more than 95% of the country, Gr...
29/04/2024
Premier Sports, an Irish-based premium sports broadcaster, has selected QuickLink's Remote Commentary solution for introducing professional, high-quality re...
29/04/2024
The awards celebrate the craft of behind-the-scenes TV talent and the best programmes of 2023
By Matthew Corrigan
Published: April 29, 2024
The awards cel...
29/04/2024
The team at Milk had to create and deliver the VFX and environment work for the royal residences featured in the drama from scratch
By Jenny Priestley
Publis...
29/04/2024
CEO Bob Bakish is expected to leave the company as early as today, with a new leadership committee likely to run the company on an interim basis
By Jenny Pries...
29/04/2024
Albena Ivanova, director, media and entertainment and strategic products at CHAOS talks to TVBEurope about her route into the industry
By TVBEurope Staff
Pub...
29/04/2024
The acquisition gives Grand Technix the opportunity to expand its footprint in the audio visual and broadcast technology sectors
By Jenny Priestley
Published...
29/04/2024
29 04 2024 - Media release Screen Australia announces James J. Robinson's debut feature First Light
First Light
Principal Photography is underway on Firs...
29/04/2024
Capitol Broadcasting recently became the inaugural company honored with inductio...
29/04/2024
Back to All News
Netflix Announces a Diverse Lineup of Polish Titles to Debut i...
29/04/2024
Tonight on Smoke and Mirrors: Fanyana feels embarrassed about his sexual mishapDon't miss Monday, 29 April's riveting episode of South African soapie Sm...
29/04/2024
Talk about commitment. When startup SEA.AI, an NVIDIA Metropolis partner, set out to create a system that would use AI to scan the seas to enhance maritime safe...
28/04/2024
Mediahaus delivers the first SRT live-streaming sports production over 5G with U...
27/04/2024
On April 26, L3Harris Chair and CEO Christopher E. Kubasik joined CNBC's Mor...
27/04/2024
PORTLAND, Oregon Audinate Group Limited, the developer of the Dante AV-over-IP solution, announced significant new additions to Dante Connect, its cloud-based D...
27/04/2024
TORONTO Bell Media has launched 10 English and French-language FAST channels featuring entertainment, factual, news, and sports programming. The new free stream...
27/04/2024
An extensive new analysis of the news segments in the broadcast evening news programs of ABC, CBS, NBC and PBS has found that broadcasters devoted most of their...
27/04/2024
GERMANTOWN, Md. EchoStar's Hughes Network Systems has opened a new manufacturing facility and private 5G incubation center in Germantown, Maryland....
27/04/2024
Harry Pappas, one of three brothers who founded Pappas Telecasting Companies in 1971, died April 24. He was 78 years old....
27/04/2024
ATLANTA and LONDON Mexican telecommunications and broadcast company Televisa has selected Synamedia for an overhaul of its broadcast distribution....
27/04/2024
Participate in the Survey - The Impact of AI on Media and the Creative Industry
Pascal Wagner April 26, 2024
0 Comments
By participating in this surve...
27/04/2024
SDVI Rally Access Workstation Earns Two Top Awards at 2024 NAB Show
Brie Clayton April 26, 2024
0 Comments
SDVI, the leading platform provider for clo...
27/04/2024
Berklee's Music and Health Institute Launches Community Health Musician Cert...
27/04/2024
Charter Communications reported higher first-quarter profits despite continued cord-cutting and competition for broadband customers....
27/04/2024
Two environmentally-focused groups are partnering to engage the unscripted TV world in finding better ways to address climate change. Reality of Change is an ec...
27/04/2024
Sarah Garcia has been promoted to weekend anchor at KTLM McAllen, Texas, known as Telemundo 40. Starting April 27, she will anchor Noticias Telemundo 40 weekend...
27/04/2024
CBS Sports said it launched a new 24-hour free, ad supported streaming television (FAST) channel devoted to the UEFA Champions League....
27/04/2024
Comcast chairman and CEO Brian Roberts received $35.4 million in compensation in 2023, up 11% from the previous year, according to a proxy statement filed by th...
27/04/2024
Art Happens Here With John Lithgow, which sees the actor study dance, ceramics, silk-screen printing and vocal jazz with students in Los Angeles, debuts on PBS ...
27/04/2024
Remember Leave It to Beaver? Bewitched? Dragnet? When cable ratings were rising?...
27/04/2024
Next up for the weekend binge at Catchy Comedy is Gomer Pyle, U.S.M.C. Every weekend, Catchy Comedy features The Catchy Binge, a marathon of a classic sitcom....
Idea generation, not hardware or software, needs to be the bottleneck to the advancement of AI, Bryan Catanzaro, vice president of applied deep learning research at NVIDIA, said this week at the AI Hardware Summit.
