Why GPUs Are Great for AI
04/12/2023
Three technical reasons, and many stories, explain why that's so. Each reason has multiple facets well worth exploring, but at a high level:
GPUs employ parallel processing.
GPU systems scale up to supercomputing heights.
The GPU software stack for AI is broad and deep.
The net result is GPUs perform technical calculations faster and with greater energy efficiency than CPUs. That means they deliver leading performance for AI training and inference as well as gains across a wide array of applications that use accelerated computing.
In its recent report on AI, Stanford's Human-Centered AI group provided some context. GPU performance has increased roughly 7,000 times since 2003 and price per performance is 5,600 times greater, it reported.
A 2023 report captured the steep rise in GPU performance and price/performance. The report also cited analysis from Epoch, an independent research group that measures and forecasts AI advances.
GPUs are the dominant computing platform for accelerating machine learning workloads, and most (if not all) of the biggest models over the last five years have been trained on GPUs [they have] thereby centrally contributed to the recent progress in AI, Epoch said on its site.
A 2020 study assessing AI technology for the U.S. government drew similar conclusions.
We expect [leading-edge] AI chips are one to three orders of magnitude more cost-effective than leading-node CPUs when counting production and operating costs, it said.
NVIDIA GPUs have increased performance on AI inference 1,000x in the last ten years, said Bill Dally, the company's chief scientist in a keynote at Hot Chips, an annual gathering of semiconductor and systems engineers.
ChatGPT Spread the News ChatGPT provided a powerful example of how GPUs are great for AI. The large language model (LLM), trained and run on thousands of NVIDIA GPUs, runs generative AI services used by more than 100 million people.
Since its 2018 launch, MLPerf, the industry-standard benchmark for AI, has provided numbers that detail the leading performance of NVIDIA GPUs on both AI training and inference.
For example, NVIDIA Grace Hopper Superchips swept the latest round of inference tests. NVIDIA TensorRT-LLM, inference software released since that test, delivers up to an 8x boost in performance and more than a 5x reduction in energy use and total cost of ownership. Indeed, NVIDIA GPUs have won every round of MLPerf training and inference tests since the benchmark was released in 2019.
In February, NVIDIA GPUs delivered leading results for inference, serving up thousands of inferences per second on the most demanding models in the STAC-ML Markets benchmark, a key technology performance gauge for the financial services industry.
A RedHat software engineering team put it succinctly in a blog: GPUs have become the foundation of artificial intelligence.
AI Under the Hood A brief look under the hood shows why GPUs and AI make a powerful pairing.
An AI model, also called a neural network, is essentially a mathematical lasagna, made from layer upon layer of linear algebra equations. Each equation represents the likelihood that one piece of data is related to another.
For their part, GPUs pack thousands of cores, tiny calculators working in parallel to slice through the math that makes up an AI model. This, at a high level, is how AI computing works.
Highly Tuned Tensor Cores Over time, NVIDIA's engineers have tuned GPU cores to the evolving needs of AI models. The latest GPUs include Tensor Cores that are 60x more powerful than the first-generation designs for processing the matrix math neural networks use.
In addition, NVIDIA Hopper Tensor Core GPUs include a Transformer Engine that can automatically adjust to the optimal precision needed to process transformer models, the class of neural networks that spawned generative AI.
Along the way, each GPU generation has packed more memory and optimized techniques to store an entire AI model in a single GPU or set of GPUs.
Models Grow, Systems Expand The complexity of AI models is expanding a whopping 10x a year.
The current state-of-the-art LLM, GPT4, packs more than a trillion parameters, a metric of its mathematical density. That's up from less than 100 million parameters for a popular LLM in 2018.
In a recent talk at Hot Chips, NVIDIA Chief Scientist Bill Dally described how single-GPU performance on AI inference expanded 1,000x in the last decade. GPU systems have kept pace by ganging up on the challenge. They scale up to supercomputers, thanks to their fast NVLink interconnects and NVIDIA Quantum InfiniBand networks.
For example, the DGX GH200, a large-memory AI supercomputer, combines up to 256 NVIDIA GH200 Grace Hopper Superchips into a single data-center-sized GPU with 144 terabytes of shared memory.
Each GH200 superchip is a single server with 72 Arm Neoverse CPU cores and four petaflops of AI performance. A new four-way Grace Hopper systems configuration puts in a single compute node a whopping 288 Arm cores and 16 petaflops of AI performance with up to 2.3 terabytes of high-speed memory.
And NVIDIA H200 Tensor Core GPUs announced in November pack up to 288 gigabytes of the latest HBM3e memory technology.
Software Covers the Waterfront An expanding ocean of GPU software has evolved since 2007 to enable every facet of AI, from deep-tech features to high-level applications.
The NVIDIA AI platform includes hundreds of software libraries and apps. The CUDA programming language and the cuDNN-X library for deep learning provide a base on top of which developers have created software like NVIDIA NeMo, a framework to let users build, customize and run inference on their o
Most recent headlines
04/08/2024
Dalet Appoints Santiago Solanas as CEO to Lead Next Era of Growth and Innovation
Dalet, a leading technology and service provider for media-rich organizations, is excited to announce Santiago Solanas as its new Chief Executive Officer (CEO)....
03/06/2024
Dalet and Veritone Reach Agreement to Distribute, Transact and Monetize Media Archives
Dalet, a leading technology and service provider for media-rich organizations, a...
29/05/2024
ST Engineering iDirect Next-Generation Hub Infrastructure Selected for Indonesia's First Multifunction Satellite
Highly scalable and flexible solution supporting Satria-1 satellite to facilitat...
29/05/2024
L3Harris Empowers Future Canadian Leaders Through the CILA Program
In 2018, Rich Foster, Vice President of L3Harris Canada, envisioned a transformative initiative to address the gender and diversity gap in science, technology, ...
29/05/2024
Charting the Future of the U.S. Navy
The christening of OUSV Vanguard, the U.S. Navys newest Unmanned Surface Vehicle, marks a pivotal moment in Naval technology. Developed through the joint Strate...
29/05/2024
EditShare Boosts Sales Direction With Alumnus Grant Carroll
EditShare Boosts Sales Direction With Alumnus Grant Carroll Long-term leader moves up to head sales in the Americas Boston, MA, May 29, 2024 - EditShare, the...
29/05/2024
Broadcasters Foundation of America Designates June 13 its Annual Giving Day
The Broadcasters Foundation of America has announced its annual Giving Day will take place Thursday, June 13. The campaign's purpose is to raise money to su...
29/05/2024
Grant Carroll Returns to EditShare as its New SVP-Americas
BOSTON EditShare has hired Grant Carroll as its new Senior Vice President for Sales for the Americas....
29/05/2024
Vizrt joins forces with Dalet to enhance newsroom operations
Vizrt joins forces with Dalet to enhance newsroom operations Brie Clayton May 29, 2024 0 Comments The integration between Dalet Galaxy five and Viz Pi...
29/05/2024
Tucson TV Stations Launch NextGen TV Services
TUSCON Six stations have launched NextGen TV, aka ATSC 3.0 broadcasts in the Tucson, Ariz., area....
29/05/2024
France Tlvisions Upgrades to Grass Valley Kaleido-IP Video Multiviewer
MONTREAL Grass Valley is reporting that French National Public TV Broadcaster France T l visions, rebranded as france tv, has selected its next-generation Kalei...
29/05/2024
John Abbot Joins Google Fiber as Its First CFO
MOUNTAIN VIEW, Calif. Google Fiber has announced that John Abbot has recently joined its team as the company's first chief financial officer (CFO)....
29/05/2024
Obsidian Lighting Control ONYX 4.10 Software Now Available
Obsidian Control Systems has introduced ONYX 4.10, the latest iteration of the popular lighting control software for NX consoles and PC systems....
29/05/2024
ZOO Establishes ZOO Italy, Launches Dubbing Studios in Milan
LONDON ZOO Digital, a global provider of localization and media services to the entertainment industry, has launched ZOO dubbing studios in Milan and establishe...
29/05/2024
Vizrt Integrates HTML Graphics System with Dalet News Production System
BERGEN, Norway Vizrt has integrated Viz Pilot Edge, the company's newsroom HTML-based templated graphics system, with the Dalet Galaxy five news production ...
29/05/2024
Elettroformati Audio Post House Installs PMC Monitors In...
Italian audio production company Elettroformati has chosen PMC monitors and an Avid management system for its new Dolby Atmos music mixing studio in Milan. Fo...
29/05/2024
Pixotope Enables Remote Intercontinental Camera Tracking...
Pixotope, the leading software platform for end-to-end real-time virtual production solutions, is breaking new ground by enabling remote real time virtual produ...
29/05/2024
Vizrt joins forces with Dalet to enhance newsroom operati...
Vizrt, the leader in real-time graphics and live production solutions for content creators, today announces that its flagship newsroom HTML-based templated grap...
29/05/2024
Ateme Leads TVRIs Transition to 4K UHD OTT Streaming
Ateme, the global leader in video compression, delivery and streaming solutions with innovation at its core, today announced TVRI s historic transition to 4K UH...
29/05/2024
WRAL-TV's Shrader, Holland Talk Historic Hurricane Forecast on WRAL Daily Download
NOAA (the National Oceanic and Atmospheric Administration) issued a forecast las...
29/05/2024
VEON appoints UHY LLP as auditors for VEON Group's 2023 PCAOB Audit and shares compliance plan with Nasdaq
29 May 2024 VEON appoints UHY LLP as auditors for VEON Group's 2023 PCAOB A...
29/05/2024
Scripps Spelling Bee Is Its Own Kind Of Sport - and Has Its Own Kind of Broadcast on Ion Television
Scripps Spelling Bee Is Its Own Kind Of Sport - and Has Its Own Kind of Broadcas...
29/05/2024
TikTok's Tim Edwards Talks Long Form Content, Monetization and the Power of Search
TikTok's Tim Edwards Talks Long Form Content, Monetization and the Power of ...
29/05/2024
PWHL Finals: Raycom Sports, Sky Candy Studios Deploy Live Drone Over the Ice for Decisive Game 5 in Boston
PWHL Finals: Raycom Sports, Sky Candy Studios Deploy Live Drone Over the Ice for...
29/05/2024
Introducing R&SGSACSM: The most advanced communications system monitoring solution for armed forces
Introducing R&S GSACSM: The most advanced communications system monitoring solut...
29/05/2024
Netflix and Yash Raj Films Announce Maharaj': A Story of One Man's Courage in Pre-Independence India', Premiering June 14
Back to All News Netflix and Yash Raj Films Announce Maharaj': A Story of ...
29/05/2024
IBM Study: 6 Hard Truths CEOs Must Face - As CEOs rush to adopt generative AI adoption, workforce and culture concerns intensify
LONDON, UK, 29 May 2024 A new study by the IBM (NYSE: IBM) Institute for Busin...
29/05/2024
Arvato Systems wins Gold again at the Service Provider Awards
Arvato Systems wins Gold again at the Service Provider Awards Award in the Managed Cloud Service Provider category Arvato Systems receives Gold as Managed C...
29/05/2024
RT Brings You to the Heart of the Action this June Bank Holiday Weekend
An action-packed weekend of live sport, including the Women's Euro 2025 Qualifier, the GAA Championship, URC Live and the Champions League Final Catch al...
29/05/2024
Tidy Tech: How Two Stanford Students Are Building Robots for Handling Household Chores
Imagine having a robot that could help you clean up after a party - or fold heap...
29/05/2024
Decoding How NVIDIA RTX AI PCs and Workstations Tap the Cloud to Supercharge Generative AI
Editor's note: This post is part of the AI Decoded series, which demystifies...
29/05/2024
Thales' FlytEDGE - the first cloud-based IFE in the world Winner of Crystal Cabin Award
Facebook Twitter LinkedIn The Crystal Cabin Award Association recognized T...
29/05/2024
VIZIO and Dolby Usher in Premium Sound Era For All
29 May 2024, 05:30 (PDT) VIZIO and Dolby Usher in Premium Sound Era For All With Dolby Atmos across its entire 2024 soundbar lineup, VIZIO and Dolby are lea...
29/05/2024
SWR moves to software playout with integrated Pixel Power solution from Rohde & Schwarz
SWR moves to software playout with integrated Pixel Power solution from Rohde & ...
28/05/2024
AI and Disinformation in Taiwan's 2024 Election
This is a summary of the report commissioned by Thomson on AI Disinformation Attacks during Taiwans 2024 Presidential Elections, written by Professor Chen-ling ...
28/05/2024
In A Violent Nature: Festivalgoers Look Through the Eyes of a Murderer
PARK CITY, UTAH - JANUARY 22: Chris Nash attends the 2024 Sundance Film Festival In A Violent Nature premiere at the Library Center Theatre on January 22, 202...
28/05/2024
Aerojet Rocketdyne Expanding Huntsville Operations to Increase Solid Rocket Motor Deliveries
Aerojet Rocketdyne's Advanced Manufacturing Facility opened in 2019. The com...
28/05/2024
France Tlvisions Upgrades to Grass Valley Kaleido-IP Video Multiviewer to Support its Upcoming SMPTE-2110 Live UHD/3G Broadcast Transition
MONTREAL, CANADA -May 28, 2024 - Grass Valley , the leading innovator for live p...
28/05/2024
How Will Venu Sports Impact Pay-TV Subscriptions?
NEW ROCHELLE, NY Venu Sports, the new sports streaming bundle from Disney, Fox and WBD expected to launch later this year, could attract more than 4 in 10 sport...
28/05/2024
Tucson TV Stations Launch NextGen TV Service
TUSCON Six stations have launched NextGen TV, aka ATSC 3.0 broadcasts in the Tucson, Ariz., area....
28/05/2024
WSC Sports Unveils Trio Of AI-Driven Content Solutions
TEL AVIV, Israel WSC Sports has introduced three additions to its product portfolio for sports ecosystem companies that address content management, creation and...
28/05/2024
Pixotope Enables Remote Intercontinental Camera Tracking for NEP the Netherlands
Pixotope Enables Remote Intercontinental Camera Tracking for NEP the Netherlands Brie Clayton May 28, 2024 0 Comments Remote markerless through-the-le...
28/05/2024
Picture Shop's Mark Kueper Grades Billy the Kid With DaVinci Resolve Studio
Picture Shop's Mark Kueper Grades Billy the Kid With DaVinci Resolve Studio Brie Clayton May 28, 2024 0 Comments Blackmagic Design today announced...
28/05/2024
Profoto enters the cinema market with L1600D
Profoto enters the cinema market with L1600D Brie Clayton May 28, 2024 0 Comments Profoto enters the cinema market with uncompromising speed of use an...
28/05/2024
After Effects cameras and Unreal Engine
After Effects cameras and Unreal Engine Graham Quince May 28, 2024 0 Comments Welcome to my series on learning Unreal Engine for video production, esp...
28/05/2024
Apple Motion: Understanding Fixed Resolution
Apple Motion: Understanding Fixed Resolution Simon Ubsdell May 28, 2024 0 Comments An overview of this tricky but important topic which can hit you wi...
28/05/2024
OBS Taps Alibaba Cloud for AI-Enhanced MultiCamera Replays at Paris 2024
LONDON Olympic Broadcasting Services recently tested AI-enhanced multcamera replay tech from Alibaba Cloud at the Olympic Qualifier Series in Shanghai in prepar...
28/05/2024
Dune Part 2 and Avatar colourists to take part in DaVinci Resolve Live Tour
The events are for filmmakers, editors, colourists, and visual effects artists, whether theyre beginners and experienced users By Jenny Priestley Published: ...
28/05/2024
Meet the head of sound
1185 Films Mark Hodgkin explains his journey from studying classic guitar and piano to working on the sound of TV adverts, films and documentaries By TVBEurope...
28/05/2024
Alfalite presents its LED displays at InfoComm 2024
Alfalite, the European LED display manufacturer, returns for the second consecutive year to InfoComm with its LED displays for the rental, fixed installation an...