
Large language models and the applications they power enable unprecedented opportunities for organizations to get deeper insights from their data reservoirs and to build entirely new classes of applications.
But with opportunities often come challenges.
Both on premises and in the cloud, applications that are expected to run in real time place significant demands on data center infrastructure to simultaneously deliver high throughput and low latency with one platform investment.
To drive continuous performance improvements and improve the return on infrastructure investments, NVIDIA regularly optimizes the state-of-the-art community models, including Meta's Llama, Google's Gemma, Microsoft's Phi and our own NVLM-D-72B, released just a few weeks ago.
Relentless Improvements Performance improvements let our customers and partners serve more complex models and reduce the needed infrastructure to host them. NVIDIA optimizes performance at every layer of the technology stack, including TensorRT-LLM, a purpose-built library to deliver state-of-the-art performance on the latest LLMs. With improvements to the open-source Llama 70B model, which delivers very high accuracy, we've already improved minimum latency performance by 3.5x in less than a year.
We're constantly improving our platform performance and regularly publish performance updates. Each week, improvements to NVIDIA software libraries are published, allowing customers to get more from the very same GPUs. For example, in just a few months' time, we've improved our low-latency Llama 70B performance by 3.5x.
NVIDIA has increased performance on the Llama 70B model by 3.5x. In the most recent round of MLPerf Inference 4.1, we made our first-ever submission with the Blackwell platform. It delivered 4x more performance than the previous generation.
This submission was also the first-ever MLPerf submission to use FP4 precision. Narrower precision formats, like FP4, reduces memory footprint and memory traffic, and also boost computational throughput. The process takes advantage of Blackwell's second-generation Transformer Engine, and with advanced quantization techniques that are part of TensorRT Model Optimizer, the Blackwell submission met the strict accuracy targets of the MLPerf benchmark.
Blackwell B200 delivers up to 4x more performance versus previous generation on MLPerf Inference v4.1's Llama 2 70B workload. Improvements in Blackwell haven't stopped the continued acceleration of Hopper. In the last year, Hopper performance has increased 3.4x in MLPerf on H100 thanks to regular software advancements. This means that NVIDIA's peak performance today, on Blackwell, is 10x faster than it was just one year ago on Hopper.
These results track progress on the MLPerf Inference Llama 2 70B Offline scenario over the past year. Our ongoing work is incorporated into TensorRT-LLM, a purpose-built library to accelerate LLMs that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM is built on top of the TensorRT Deep Learning Inference library and leverages much of TensorRT's deep learning optimizations with additional LLM-specific improvements.
Improving Llama in Leaps and Bounds More recently, we've continued optimizing variants of Meta's Llama models, including versions 3.1 and 3.2 as well as model sizes 70B and the biggest model, 405B. These optimizations include custom quantization recipes, as well as efficient use of parallelization techniques to more efficiently split the model across multiple GPUs, leveraging NVIDIA NVLink and NVSwitch interconnect technologies. Cutting-edge LLMs like Llama 3.1 405B are very demanding and require the combined performance of multiple state-of-the-art GPUs for fast responses.
Parallelism techniques require a hardware platform with a robust GPU-to-GPU interconnect fabric to get maximum performance and avoid communication bottlenecks. Each NVIDIA H200 Tensor Core GPU features fourth-generation NVLink, which provides a whopping 900GB/s of GPU-to-GPU bandwidth. Every eight-GPU HGX H200 platform also ships with four NVLink Switches, enabling every H200 GPU to communicate with any other H200 GPU at 900GB/s, simultaneously.
Many LLM deployments use parallelism over choosing to keep the workload on a single GPU, which can have compute bottlenecks. LLMs seek to balance low latency and high throughput, with the optimal parallelization technique depending on application requirements.
For instance, if lowest latency is the priority, tensor parallelism is critical, as the combined compute performance of multiple GPUs can be used to serve tokens to users more quickly. However, for use cases where peak throughput across all users is prioritized, pipeline parallelism can efficiently boost overall server throughput.
The table below shows that tensor parallelism can deliver over 5x more throughput in minimum latency scenarios, whereas pipeline parallelism brings 50% more performance for maximum throughput use cases.
For production deployments that seek to maximize throughput within a given latency budget, a platform needs to provide the ability to effectively combine both techniques like in TensorRT-LLM.
Read the technical blog on boosting Llama 3.1 405B throughput to learn more about these techniques.
Different scenarios have different requirements, and parallelism techniques bring optimal performance for each of these scenarios. The Virtuous Cycle Over the lifecycle of our architectures, we deliver significant performance gains from ongoing software tuning and optimization. These improvements translate into additional value for customers who train and deploy on our platforms. They're able to create more capable models and applications and deploy their existing models using less infrastructure, enhancing th
Most recent headlines
05/01/2027
Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...
09/10/2026
September 10 2026, 06:00 (PDT) Dolby Expands Dolby OptiView Platform with New Capabilities at IBC 2026
New Sports Intelligence helps providers better unders...
07/10/2026
Dalet, a leading technology and service provider for media-rich organizations, today announced the latest Long-Term Supported (LTS) release of Dalet Flex. Build...
25/09/2026
In-venue and creative video staffers at the professional and collegiate level ha...
25/09/2026
Infront has named Ronald den Hollander as CEO of its Host Broadcast Services (HBS) subsidiary, effective 1 October 2026. The appointment follows the previously ...
25/09/2026
The North American Broadcasters Association (NABA) will hold Cyber University 2.0, a full-day cybersecurity summit for the broadcast and media industries, on Tu...
25/09/2026
Alpha Networks and Ateme have delivered a joint video platform for the digital d...
25/09/2026
Encompass Digital Media has announced the migration of multiple Viaplay free-to-air channels - TV3, TV6, TV8, and TV10 - to its Altitude Scheduling platform. Al...
25/09/2026
NAB Show New York 2026 will take place October 21-22 at the Jacob K. Javits Convention Center in New York City. Registration is open at nabshow.com/newyork.
Th...
25/09/2026
NHL Network has announced its studio programming lineup for the 2026-27 regular ...
25/09/2026
Clear-Com has announced that First Baptist Nashville has upgraded its production communications with a Clear-Com intercom system as part of a sanctuary technolo...
25/09/2026
Broadcast Management Group (BMG) has deployed EVS XT-VIA live video production s...
25/09/2026
BBright, a Hexaglobe Group company, has announced that dotCentral has joined as a sales and support partner in the United States.
BBright's portfolio cover...
25/09/2026
MeyerPro delivered audio, video, broadcast, and LED production for the Acumatica Summit 2026 at the Seattle Convention Center's Summit Building, January 25-...
25/09/2026
The Professional Women's Hockey League has announced its inaugural All-Star Weekend, to be held Saturday, March 7, 2026, at Scotiabank Arena in Toronto. The...
25/09/2026
The 2026 Presidents Cup is underway at Medinah Country Club outside of Chicago with Team USA and an International Team of men's golfers facing off for four ...
25/09/2026
Warner Bros. Discovery's Atlanta hub hosted a day of discussion on REMI prod...
25/09/2026
Algorithm update & new DVA vibrato-detection engine
A new flagship version of Antares' iconic pitch-correction software has arrived, and along with a co...
25/09/2026
New flagship unit to arrive in early 2027
Two notes Audio Engineering have revealed that a new and improved version of their hugely popular reactive load bo...
25/09/2026
AI-free mixing tool gets an upgrade
Launched in October 2025, Tim Exiles Finalist is an algorithmic mixing tool that's been designed for those who don...
25/09/2026
Rohde & Schwarz Mobile Test Summit 2026 puts the spotlight on 6G - registrations...
25/09/2026
In our latest blog, Tim Pearson reflects on IBC 2026 and considers the main learnings against an industry that saw AI finally coming out of the labs and into pr...
25/09/2026
Digital Anarchy Releases Newest Version of its Real-Time Skin Retouching Tool fo...
25/09/2026
Blackmagic Design Used on More Than 50 Summer Films in 2026
Brie Clayton September 25, 2026
0 Comments
Hero image: Disclosure Day
Blackmagic Design p...
25/09/2026
How I Automated Social Media Resizing in After Effects - Free Script
Graham Quince September 25, 2026
0 Comments
If your text keeps getting cut off on...
25/09/2026
IBC has unveiled the winners of the prestigious IBC2026 Innovation Awards at an event held at the RAI Amsterdam on Sunday 13 September, honouring collaborative ...
25/09/2026
IBC2026 brought the global media, entertainment and technology community together for four days of business, innovation and collaboration at the RAI Amsterdam. ...
25/09/2026
Lightware testing has revealed that each USB-C port on Apple's new MacBook Neo consumes three of the seven USB tiers available under the USB specification, ...
25/09/2026
Lightware is supporting educational institutions in building modern learning environments by combining professional training with education-focused AV technolog...
25/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
25/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
25/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
25/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
25/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
25/09/2026
Long-Standing Customer Renews Multiyear Contract for VO's Secure Video Player, Content Security, and Device Monitoring Solutions
Viaccess-Orca (VO), a glob...
25/09/2026
Automate Video Progress Bars in After Effects - No Keyframes!
Graham Quince September 24, 2026
0 Comments
Learn how to create an automated video playb...
25/09/2026
Friday 25 September 2026
How to watch an unmissable weekend of women's football on Sky Sports and NOW
Sky Sports and NOW will broadcast seven live games f...
25/09/2026
Friday 25 September 2026
YAGA trailer released, coming exclusively to Sky and N...
25/09/2026
Watch this space: a new strategic direction for UK media. Arqiva has submitted its response to the Governments Green Paper on the future of television distribut...
25/09/2026
Want to be part of this year's Late Late Toy Show?
Final call for applications in search for this year's stars
The most wonderful night of the year is...
24/09/2026
Broadcast Management Group (BMG) is supporting Virginia Public Media (VPM), the ...
24/09/2026
RED Digital Cinema's RED V-RAPTOR [X] and RED V-RAPTOR XL [X] cameras with the Phantom Track feature are the first cameras to receive GhostFrame Pro Certifi...
24/09/2026
Bringing together the full family of networks, FOX Sports is also adding Tubi, F...
24/09/2026
Virginia Polytechnic Institute and State University (Virginia Tech) has unveiled...
24/09/2026
The blog collects SVG's reporting on AI across U.S. and European sports broadcasting
Since SVG launched the AI Innovation Lab in April to advance the devel...
24/09/2026
SVG's annual TranSPORT conference, which will be held on Tuesday, Oct. 20 in...
24/09/2026
The self-built Master Control Room has cut streaming operations incident-respo...
24/09/2026
Ortana Media Group, the software systems integrator behind the Cubix workflow platform, has connected AudioShake's audio separation models to ScorePlay thro...
24/09/2026
AudioShake has released Multi-Speaker 2.0, the company's newest Multi-Speaker model, which removes 75% more noise and offers 32% better separation of overla...
24/09/2026
By Jon Morgan, Founder and Executive Producer, Ryval Studios
For Major League B...