
Large language models and the applications they power enable unprecedented opportunities for organizations to get deeper insights from their data reservoirs and to build entirely new classes of applications.
But with opportunities often come challenges.
Both on premises and in the cloud, applications that are expected to run in real time place significant demands on data center infrastructure to simultaneously deliver high throughput and low latency with one platform investment.
To drive continuous performance improvements and improve the return on infrastructure investments, NVIDIA regularly optimizes the state-of-the-art community models, including Meta's Llama, Google's Gemma, Microsoft's Phi and our own NVLM-D-72B, released just a few weeks ago.
Relentless Improvements Performance improvements let our customers and partners serve more complex models and reduce the needed infrastructure to host them. NVIDIA optimizes performance at every layer of the technology stack, including TensorRT-LLM, a purpose-built library to deliver state-of-the-art performance on the latest LLMs. With improvements to the open-source Llama 70B model, which delivers very high accuracy, we've already improved minimum latency performance by 3.5x in less than a year.
We're constantly improving our platform performance and regularly publish performance updates. Each week, improvements to NVIDIA software libraries are published, allowing customers to get more from the very same GPUs. For example, in just a few months' time, we've improved our low-latency Llama 70B performance by 3.5x.
NVIDIA has increased performance on the Llama 70B model by 3.5x. In the most recent round of MLPerf Inference 4.1, we made our first-ever submission with the Blackwell platform. It delivered 4x more performance than the previous generation.
This submission was also the first-ever MLPerf submission to use FP4 precision. Narrower precision formats, like FP4, reduces memory footprint and memory traffic, and also boost computational throughput. The process takes advantage of Blackwell's second-generation Transformer Engine, and with advanced quantization techniques that are part of TensorRT Model Optimizer, the Blackwell submission met the strict accuracy targets of the MLPerf benchmark.
Blackwell B200 delivers up to 4x more performance versus previous generation on MLPerf Inference v4.1's Llama 2 70B workload. Improvements in Blackwell haven't stopped the continued acceleration of Hopper. In the last year, Hopper performance has increased 3.4x in MLPerf on H100 thanks to regular software advancements. This means that NVIDIA's peak performance today, on Blackwell, is 10x faster than it was just one year ago on Hopper.
These results track progress on the MLPerf Inference Llama 2 70B Offline scenario over the past year. Our ongoing work is incorporated into TensorRT-LLM, a purpose-built library to accelerate LLMs that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM is built on top of the TensorRT Deep Learning Inference library and leverages much of TensorRT's deep learning optimizations with additional LLM-specific improvements.
Improving Llama in Leaps and Bounds More recently, we've continued optimizing variants of Meta's Llama models, including versions 3.1 and 3.2 as well as model sizes 70B and the biggest model, 405B. These optimizations include custom quantization recipes, as well as efficient use of parallelization techniques to more efficiently split the model across multiple GPUs, leveraging NVIDIA NVLink and NVSwitch interconnect technologies. Cutting-edge LLMs like Llama 3.1 405B are very demanding and require the combined performance of multiple state-of-the-art GPUs for fast responses.
Parallelism techniques require a hardware platform with a robust GPU-to-GPU interconnect fabric to get maximum performance and avoid communication bottlenecks. Each NVIDIA H200 Tensor Core GPU features fourth-generation NVLink, which provides a whopping 900GB/s of GPU-to-GPU bandwidth. Every eight-GPU HGX H200 platform also ships with four NVLink Switches, enabling every H200 GPU to communicate with any other H200 GPU at 900GB/s, simultaneously.
Many LLM deployments use parallelism over choosing to keep the workload on a single GPU, which can have compute bottlenecks. LLMs seek to balance low latency and high throughput, with the optimal parallelization technique depending on application requirements.
For instance, if lowest latency is the priority, tensor parallelism is critical, as the combined compute performance of multiple GPUs can be used to serve tokens to users more quickly. However, for use cases where peak throughput across all users is prioritized, pipeline parallelism can efficiently boost overall server throughput.
The table below shows that tensor parallelism can deliver over 5x more throughput in minimum latency scenarios, whereas pipeline parallelism brings 50% more performance for maximum throughput use cases.
For production deployments that seek to maximize throughput within a given latency budget, a platform needs to provide the ability to effectively combine both techniques like in TensorRT-LLM.
Read the technical blog on boosting Llama 3.1 405B throughput to learn more about these techniques.
Different scenarios have different requirements, and parallelism techniques bring optimal performance for each of these scenarios. The Virtuous Cycle Over the lifecycle of our architectures, we deliver significant performance gains from ongoing software tuning and optimization. These improvements translate into additional value for customers who train and deploy on our platforms. They're able to create more capable models and applications and deploy their existing models using less infrastructure, enhancing th
Most recent headlines
05/01/2027
Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...
07/10/2026
Dalet, a leading technology and service provider for media-rich organizations, today announced the latest Long-Term Supported (LTS) release of Dalet Flex. Build...
06/09/2026
June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...
04/08/2026
Dalet, a leading technology and service provider for media-rich organizations, t...
31/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
31/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
31/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
31/07/2026
IBC has unveiled the finalists for the IBC2026 Innovation Awards, recognising collaborative projects from around the world that solve real-world challenges and ...
31/07/2026
From focus puller to colourist
Caroline Shawley July 30, 2026
0 Comments
Ana Mar a Ormaza shares her story of instinct, storytelling and finding a cre...
31/07/2026
Amazon Prime Documentary Andata e Ritorno Relies on Blackmagic Design
Brie Clayton July 30, 2026
0 Comments
DaVinci Resolve Studio and Blackmagic Clou...
31/07/2026
Krotos Launches Video to Sound Plugin for DaVinci Resolve
Brie Clayton July 30, 2026
0 Comments
AI-assisted workflow helps editors add synchronized, p...
30/07/2026
CP Communications provided RF audio, RF video, communications, RF coordination, ...
30/07/2026
OpenDrives has been included in CRN's 2026 Storage 100 list in the Software-Defined Storage category. The annual list, selected by the CRN editorial team, r...
30/07/2026
LATAM Airlines has selected SES to provide multi-orbit inflight connectivity to its fleet of Airbus and Embraer aircraft. More than 60 aircraft - including Airb...
30/07/2026
Tagboard has launched the Producer API, an open interface that allows key commands in Tagboard's live production environment to be triggered from external d...
30/07/2026
FOX Sports and the New York Racing Association (NYRA) have announced a multi-year agreement with Del Mar Thoroughbred Club that will make FOX Sports the exclusi...
30/07/2026
ABC Commercial has launched four free ad-supported streaming television (FAST) channels on LG Smart TVs across North America, Great Britain, and select countrie...
30/07/2026
The Six Kings Slam exhibition tennis tournament will return to Riyadh on October 21, 22, and 24, streaming live on Netflix at no additional cost to subscribers....
30/07/2026
Central Christian Church in Mt. Vernon, Illinois has deployed Clear-Com's LQ...
30/07/2026
Blackmagic Design has released the UltraStudio Express 3G family, a pair of USB4 capture and playback devices compatible with Mac, Windows, and Linux computers ...
30/07/2026
ESPN has reached a media rights agreement with the Women's Pro Baseball League (WPBL), making ESPN the national streaming home of the league's 2026 seas...
30/07/2026
Most Valuable Promotions (MVP) and the Professional Fighters League (PFL) have announced a merger that will operate under the MVP banner. PFL CEO John Martin wi...
30/07/2026
The Columbus Blue Jackets will simulcast all game broadcasts across television and radio beginning with the 2026-27 NHL season. Under the new format, the televi...
30/07/2026
TMRW Sports has selected Populous as architect for a purpose-built stadium for its professional flag football league, being developed in partnership with the NF...
30/07/2026
SiriusXM has announced SiriusXM Sports Pass, a new subscription plan launching September 1 that bundles the company's sports audio programming into a single...
30/07/2026
Leagues, broadcasters, and technologists gather to tackle cloud economics, distr...
30/07/2026
A major player in the cloud-based solutions, Amazon Web Services (AWS) has conti...
30/07/2026
The NBA will provide production for all local game broadcasts, including pregame and postgame coverage...
30/07/2026
Cosm will open its fourth immersive sports and entertainment venue in downtown Detroit on September 10. Located at 25 Cadillac Square, adjacent to Campus Martiu...
30/07/2026
NFL Network will be available on the ESPN App beginning July 30 for Unlimited plan subscribers. The addition brings NFL Network's live games, studio program...
30/07/2026
NEP Group has added Riedel's SimplyLive Production Suite to NEP Platform, giving customers access to the live production software through NEP's orchestr...
30/07/2026
LOS ANGELES, CA, July 30, 2026 - The nonprofit Sundance Institute today announce...
30/07/2026
A new take on classic algorithmic reverbs
Waves have just introduced a new flagship algorithmic reverb plug-in that offers the company's take on the ico...
30/07/2026
Save up to 30% on virtual instruments & plug-ins
Arturia have recently launched their annual Summer Sale, which sees discounts of up to 30% applied across a...
30/07/2026
ActiveAnalogue technology powers new flagship modular console
A year on from the launch of the Oracle, SSL have revealed a new flagship console which they s...
30/07/2026
Up to 22dBA of passive attenuation
The latest arrival to the beyerdynamic range introduces a new compact monitoring headphone that's been designed to ta...
30/07/2026
As part of its traditional graduation ceremony, SGL Carbon honored the achievements of a total of twelve young professionals who successfully completed their ap...
30/07/2026
Premiere date revealed for SBS Original drama The Airport Chaplain, starring Hug...
30/07/2026
TVNZ and Evergent are celebrating a major TVNZ milestone - the evolution from a free-to-view, ad-supported business into a hybrid model, offering premium sport...
30/07/2026
WebMiere Adds Native VP9/AV1 WebM/MKV Import to Adobe Premiere Pro
Brie Clayton July 29, 2026
0 Comments
Brought to you by KawaiiEngine, WebMiere is a...
30/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
30/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
30/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
30/07/2026
At IBC2026, GlobalM will showcase the latest evolution of its software-defined media platform, enabling broadcasters, telecom operators and media organisations ...
30/07/2026
At the 2026 TAB Show, Digital Alert Systems will highlight and demo the new DAS-Link AIR, an over-the-air common alert protocol (CAP) dissemination system for ...
30/07/2026
OpenDrives Named to the 2026 CRN Storage 100 List
Company Recognized Among the 50 Coolest Software-Defined Storage Vendors for Unlimited Capacity, S3 Hybrid M...
30/07/2026
New Scene Search capability in VIDA Content OS makes video content itself searchable, letting broadcasters and streamers find key moments across millions of ass...
30/07/2026
July 30th, 2026
TRIBECA FILMS ANNOUNCES SUMMER 2026 ACQUISITIONS INCLUDING L...
30/07/2026
Thursday 30 July 2026
Which WSL matches are Sky Sports showing on the 2026/27 opening weekend?
The season will kick off on Friday 4 September with Manchester ...
30/07/2026
Thursday 30 July 2026
Enjoy discounts on the iPhone Air, plus 5Gb speeds with U...