
Large language models and the applications they power enable unprecedented opportunities for organizations to get deeper insights from their data reservoirs and to build entirely new classes of applications.
But with opportunities often come challenges.
Both on premises and in the cloud, applications that are expected to run in real time place significant demands on data center infrastructure to simultaneously deliver high throughput and low latency with one platform investment.
To drive continuous performance improvements and improve the return on infrastructure investments, NVIDIA regularly optimizes the state-of-the-art community models, including Meta's Llama, Google's Gemma, Microsoft's Phi and our own NVLM-D-72B, released just a few weeks ago.
Relentless Improvements Performance improvements let our customers and partners serve more complex models and reduce the needed infrastructure to host them. NVIDIA optimizes performance at every layer of the technology stack, including TensorRT-LLM, a purpose-built library to deliver state-of-the-art performance on the latest LLMs. With improvements to the open-source Llama 70B model, which delivers very high accuracy, we've already improved minimum latency performance by 3.5x in less than a year.
We're constantly improving our platform performance and regularly publish performance updates. Each week, improvements to NVIDIA software libraries are published, allowing customers to get more from the very same GPUs. For example, in just a few months' time, we've improved our low-latency Llama 70B performance by 3.5x.
NVIDIA has increased performance on the Llama 70B model by 3.5x. In the most recent round of MLPerf Inference 4.1, we made our first-ever submission with the Blackwell platform. It delivered 4x more performance than the previous generation.
This submission was also the first-ever MLPerf submission to use FP4 precision. Narrower precision formats, like FP4, reduces memory footprint and memory traffic, and also boost computational throughput. The process takes advantage of Blackwell's second-generation Transformer Engine, and with advanced quantization techniques that are part of TensorRT Model Optimizer, the Blackwell submission met the strict accuracy targets of the MLPerf benchmark.
Blackwell B200 delivers up to 4x more performance versus previous generation on MLPerf Inference v4.1's Llama 2 70B workload. Improvements in Blackwell haven't stopped the continued acceleration of Hopper. In the last year, Hopper performance has increased 3.4x in MLPerf on H100 thanks to regular software advancements. This means that NVIDIA's peak performance today, on Blackwell, is 10x faster than it was just one year ago on Hopper.
These results track progress on the MLPerf Inference Llama 2 70B Offline scenario over the past year. Our ongoing work is incorporated into TensorRT-LLM, a purpose-built library to accelerate LLMs that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM is built on top of the TensorRT Deep Learning Inference library and leverages much of TensorRT's deep learning optimizations with additional LLM-specific improvements.
Improving Llama in Leaps and Bounds More recently, we've continued optimizing variants of Meta's Llama models, including versions 3.1 and 3.2 as well as model sizes 70B and the biggest model, 405B. These optimizations include custom quantization recipes, as well as efficient use of parallelization techniques to more efficiently split the model across multiple GPUs, leveraging NVIDIA NVLink and NVSwitch interconnect technologies. Cutting-edge LLMs like Llama 3.1 405B are very demanding and require the combined performance of multiple state-of-the-art GPUs for fast responses.
Parallelism techniques require a hardware platform with a robust GPU-to-GPU interconnect fabric to get maximum performance and avoid communication bottlenecks. Each NVIDIA H200 Tensor Core GPU features fourth-generation NVLink, which provides a whopping 900GB/s of GPU-to-GPU bandwidth. Every eight-GPU HGX H200 platform also ships with four NVLink Switches, enabling every H200 GPU to communicate with any other H200 GPU at 900GB/s, simultaneously.
Many LLM deployments use parallelism over choosing to keep the workload on a single GPU, which can have compute bottlenecks. LLMs seek to balance low latency and high throughput, with the optimal parallelization technique depending on application requirements.
For instance, if lowest latency is the priority, tensor parallelism is critical, as the combined compute performance of multiple GPUs can be used to serve tokens to users more quickly. However, for use cases where peak throughput across all users is prioritized, pipeline parallelism can efficiently boost overall server throughput.
The table below shows that tensor parallelism can deliver over 5x more throughput in minimum latency scenarios, whereas pipeline parallelism brings 50% more performance for maximum throughput use cases.
For production deployments that seek to maximize throughput within a given latency budget, a platform needs to provide the ability to effectively combine both techniques like in TensorRT-LLM.
Read the technical blog on boosting Llama 3.1 405B throughput to learn more about these techniques.
Different scenarios have different requirements, and parallelism techniques bring optimal performance for each of these scenarios. The Virtuous Cycle Over the lifecycle of our architectures, we deliver significant performance gains from ongoing software tuning and optimization. These improvements translate into additional value for customers who train and deploy on our platforms. They're able to create more capable models and applications and deploy their existing models using less infrastructure, enhancing th
Most recent headlines
05/01/2027
Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...
07/10/2026
Dalet, a leading technology and service provider for media-rich organizations, today announced the latest Long-Term Supported (LTS) release of Dalet Flex. Build...
06/09/2026
June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...
01/09/2026
Bridge Technologies will announce at IBC2026 (Booth 1.A71) that its VB440 produc...
01/09/2026
DAZN has announced an agreement to acquire EverPass Media, a commercial sports distribution platform serving bars, restaurants, hotels, and other venues in the ...
01/09/2026
The USTA and Kalshi have announced a multi-year partnership making Kalshi the Official Prediction Market Partner of the US Open, beginning with the 2026 US Open...
01/09/2026
MASV will exhibit at IBC2026 (Booth 5.B40, RAI Amsterdam, September 11-14). CTO Majed Alhajry will appear on a panel session, A Tale of Two Feeds: Automation o...
01/09/2026
FingerWorks Telestrators has announced a partnership with Broadcast Rental to ex...
01/09/2026
FOR-A has announced the expansion of viztrick AiDi into two dedicated products: Go Vertical! AiDi and AiDi Sports Hub. Developed by Nippon TV and delivered glob...
01/09/2026
Sony Pictures Television has launched Live TV on PS5, a free ad-supported streaming experience available to PS5 players in the United States. The service is acc...
01/09/2026
Friedrichstadt-Palast in Berlin has installed a Riedel production infrastructure spanning video, intercom, control, and I/O across the venue, implemented in col...
01/09/2026
Arkona technologies GmbH has announced an OEM partnership with BFE Studio und Medien Systeme GmbH to offer arkona-branded control panels for its SDI-IP routing ...
01/09/2026
Cricket Australia has launched Cricket TV, a direct-to-consumer streaming service built on Eluvio's Content Fabric platform. The service is currently in ear...
01/09/2026
Eluvio has announced it was selected to power Pasifika , a new direct-to-consume...
01/09/2026
Layercake, the company behind the intelligent media workflow platform Streamcake, has appointed Jon Marquard as Director and Chairperson. The appointment marks ...
01/09/2026
Gravity Media has announced a multi-year renewal of its agreement with Stawell Gift Event Management to produce the complete broadcast of the Powercor Stawell G...
01/09/2026
Beam Dynamics will exhibit at IBC2026 for the first time (Stand 14.FT29, Hall 14), introducing License Intelligence, a new module for managing software licenses...
01/09/2026
Interra Systems has announced that ZOO Digital has selected BATON for file-based quality control across its media workflows. ZOO Digital delivers localized ente...
01/09/2026
Bridge Technologies has announced the deployment of its VB330 monitoring probe at Sunrise, supporting the Swiss telecommunications provider's IP-based telev...
01/09/2026
Riedel Communications has announced the continued expansion of its managed coach communications partnership with Scottish Rugby Union, now entering its third se...
01/09/2026
Behind The Mic provides a roundup of recent news regarding on-air talent, includ...
01/09/2026
SMPTE today detailed the centerpiece of its IBC2026 program: the SMPTE Roadshow: Future of Media Infrastructure, a full-day workshop on Monday, 14 September, th...
01/09/2026
Inside the Music and Sound of Michael
Sound Particles have announced an upcoming webinar with John Warhurst, the Academy Award- and BAFTA-winning supervisin...
01/09/2026
Piano offers tutorials, lessons & feedback via app
Roland have just launched a new digital piano that integrates with a mobile app in an effort to encourage...
01/09/2026
Update introduces 7.1.4 support via Dante
Trinnov have just announced a software update that will allow Nova units with eight licensed Optimizer channels to...
01/09/2026
SBS secures exclusive rights to FIFA World Cup 2030
1 September, 2026
Media releases
Major sports rights deal ensures all Australians will be able to view...
01/09/2026
SBS announces the launch of outcomes-based measurement across its network
1 September, 2026
Media releases
New offer sees national broadcaster put outcomes...
01/09/2026
Revealed: The Booster Economy' how $140bn+ in everyday inheritances'...
01/09/2026
SBS census explorer to provide new insights into contemporary Australia
1 September, 2026
Media releases
The 2026 Census Explorer will give marketers new i...
01/09/2026
SBS and oOh!media launch exclusive SBS News content partnership across office, a...
01/09/2026
SBS offers marketers greater reach with extension into outdoor, creator economy ...
01/09/2026
SBS and The Guardian join forces to supersize sustainability challenge for 2027
1 September, 2026
Media releases
SBS Media and Guardian Australia have join...
01/09/2026
SBS goes deeper in 2027 with provocative Australian documentaries, star-studded ...
01/09/2026
Amazon MGM Studios have commissioned a new thrilling 6-part drama, Crouch End, b...
01/09/2026
New York August 31, 2026 Nielsen is announcing four new local methodology enhancements that will be activated into Local TV currency data starting today, Au...
01/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
01/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
01/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
01/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
01/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
01/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
01/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
01/09/2026
Pronology, a developer of cutting-edge broadcast workflow solutions, showcases the newest capabilities for its StreamFile Core platform at the 2026 IBC Show (S...
01/09/2026
PROTON Unveils Next-Generation FLEXY Camera at IBC 2026
Brie Clayton September 1, 2026
0 Comments
10-metre serialised coax unlocks new perspectives an...
01/09/2026
Hulu Drama The Season Graded with DaVinci Resolve Studio
Brie Clayton September 1, 2026
0 Comments
Senior Colorist Mark Kueper of Picture Shop crafts ...
01/09/2026
Pluxbox Launches CoreOC to Transform Mission-Critical Media Operations
Brie Clayton September 1, 2026
0 Comments
A vendor-agnostic operational control l...
01/09/2026
Layercake, the company behind the intelligent media workflow platform Streamcake, today announced the appointment of Jon Marquard as Director and Chairperson. T...
01/09/2026
Actus Digital, a LiveU company, will unveil the next generation of its Actus X platform at IBC2026 on LiveUs Stand 7.C24, demonstrating how broadcasters can sim...
01/09/2026
Broadcast Solutions, a leading systems integrator and provider of innovative solutions for the broadcast media industry, delivered transformational technology a...
01/09/2026
Quortex, the intelligent video technology company carved out from Synamedia by Lumine Group, today announced that Odisha Television Network (OTV), the leading r...