
Large language models and the applications they power enable unprecedented opportunities for organizations to get deeper insights from their data reservoirs and to build entirely new classes of applications.
But with opportunities often come challenges.
Both on premises and in the cloud, applications that are expected to run in real time place significant demands on data center infrastructure to simultaneously deliver high throughput and low latency with one platform investment.
To drive continuous performance improvements and improve the return on infrastructure investments, NVIDIA regularly optimizes the state-of-the-art community models, including Meta's Llama, Google's Gemma, Microsoft's Phi and our own NVLM-D-72B, released just a few weeks ago.
Relentless Improvements Performance improvements let our customers and partners serve more complex models and reduce the needed infrastructure to host them. NVIDIA optimizes performance at every layer of the technology stack, including TensorRT-LLM, a purpose-built library to deliver state-of-the-art performance on the latest LLMs. With improvements to the open-source Llama 70B model, which delivers very high accuracy, we've already improved minimum latency performance by 3.5x in less than a year.
We're constantly improving our platform performance and regularly publish performance updates. Each week, improvements to NVIDIA software libraries are published, allowing customers to get more from the very same GPUs. For example, in just a few months' time, we've improved our low-latency Llama 70B performance by 3.5x.
NVIDIA has increased performance on the Llama 70B model by 3.5x. In the most recent round of MLPerf Inference 4.1, we made our first-ever submission with the Blackwell platform. It delivered 4x more performance than the previous generation.
This submission was also the first-ever MLPerf submission to use FP4 precision. Narrower precision formats, like FP4, reduces memory footprint and memory traffic, and also boost computational throughput. The process takes advantage of Blackwell's second-generation Transformer Engine, and with advanced quantization techniques that are part of TensorRT Model Optimizer, the Blackwell submission met the strict accuracy targets of the MLPerf benchmark.
Blackwell B200 delivers up to 4x more performance versus previous generation on MLPerf Inference v4.1's Llama 2 70B workload. Improvements in Blackwell haven't stopped the continued acceleration of Hopper. In the last year, Hopper performance has increased 3.4x in MLPerf on H100 thanks to regular software advancements. This means that NVIDIA's peak performance today, on Blackwell, is 10x faster than it was just one year ago on Hopper.
These results track progress on the MLPerf Inference Llama 2 70B Offline scenario over the past year. Our ongoing work is incorporated into TensorRT-LLM, a purpose-built library to accelerate LLMs that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM is built on top of the TensorRT Deep Learning Inference library and leverages much of TensorRT's deep learning optimizations with additional LLM-specific improvements.
Improving Llama in Leaps and Bounds More recently, we've continued optimizing variants of Meta's Llama models, including versions 3.1 and 3.2 as well as model sizes 70B and the biggest model, 405B. These optimizations include custom quantization recipes, as well as efficient use of parallelization techniques to more efficiently split the model across multiple GPUs, leveraging NVIDIA NVLink and NVSwitch interconnect technologies. Cutting-edge LLMs like Llama 3.1 405B are very demanding and require the combined performance of multiple state-of-the-art GPUs for fast responses.
Parallelism techniques require a hardware platform with a robust GPU-to-GPU interconnect fabric to get maximum performance and avoid communication bottlenecks. Each NVIDIA H200 Tensor Core GPU features fourth-generation NVLink, which provides a whopping 900GB/s of GPU-to-GPU bandwidth. Every eight-GPU HGX H200 platform also ships with four NVLink Switches, enabling every H200 GPU to communicate with any other H200 GPU at 900GB/s, simultaneously.
Many LLM deployments use parallelism over choosing to keep the workload on a single GPU, which can have compute bottlenecks. LLMs seek to balance low latency and high throughput, with the optimal parallelization technique depending on application requirements.
For instance, if lowest latency is the priority, tensor parallelism is critical, as the combined compute performance of multiple GPUs can be used to serve tokens to users more quickly. However, for use cases where peak throughput across all users is prioritized, pipeline parallelism can efficiently boost overall server throughput.
The table below shows that tensor parallelism can deliver over 5x more throughput in minimum latency scenarios, whereas pipeline parallelism brings 50% more performance for maximum throughput use cases.
For production deployments that seek to maximize throughput within a given latency budget, a platform needs to provide the ability to effectively combine both techniques like in TensorRT-LLM.
Read the technical blog on boosting Llama 3.1 405B throughput to learn more about these techniques.
Different scenarios have different requirements, and parallelism techniques bring optimal performance for each of these scenarios. The Virtuous Cycle Over the lifecycle of our architectures, we deliver significant performance gains from ongoing software tuning and optimization. These improvements translate into additional value for customers who train and deploy on our platforms. They're able to create more capable models and applications and deploy their existing models using less infrastructure, enhancing th
Most recent headlines
05/01/2027
Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...
07/10/2026
Dalet, a leading technology and service provider for media-rich organizations, today announced the latest Long-Term Supported (LTS) release of Dalet Flex. Build...
06/09/2026
June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...
27/08/2026
Advertisers gain granular CTV targeting and turnkey reporting across eligible in...
27/08/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
27/08/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
27/08/2026
Profuz Digital makes a mark by launching a powerful upgrade to its Profuz LAPIS digital media management platform this IBC in Amsterdam at Stand 8.A88. The late...
27/08/2026
DHD.audio has chosen IBC 2026 as the launch venue for what it defines as one of the most radical advances in the companys 30 year history.
Every broadcast trad...
27/08/2026
Direct connectivity between planning, production and graphics for creativity and streamlined workflows...
27/08/2026
New Photoshop innovations bring you more choice and control at every stage of yo...
27/08/2026
First Two Seasons of Vertical Series Vultures Shot on Blackmagic Design Cameras
Brie Clayton August 27, 2026
0 Comments
Director and DoP of Vultures ...
27/08/2026
For years, the media and entertainment industry has hesitated to use AI-voiced dubbing due to concern for audience experience. New data from 3Play Media, a glob...
27/08/2026
Starfish Technologies has announced the forthcoming V9 release of its Advantage AD workstation, adding support for voice synthesis services and giving Audio Des...
27/08/2026
Lightware's UBEX is a powerful AV-over-IP product well suited for rental and staging workflows where systems must be fast to deploy, easy to repurpose, and ...
27/08/2026
Lightware has expanded the scalability of its 10G TPN AV-over-IP platform with the introduction of the new TPN-MMU-X100 Matrix Management Unit, enabling systems...
27/08/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
27/08/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
27/08/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
27/08/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
27/08/2026
PlayBox Neo Stand 7.C29 IBC2026 - 11 14 September
Well-established global specialists in media playout, automation, and streaming solutions, PlayBox Neo ...
27/08/2026
Rise Academy, the broadcast and media skills charity dedicated to delivering practical media technology learning experiences, careers resources and work experie...
27/08/2026
Mediagenix, a global leader in smart content solutions to profitably connect the right content to the right audience, will showcase the latest evolution of its ...
27/08/2026
Screen Australia opens applications for Global Fellowships Program 27 August 2026
Applications are now open for the Screen Australia Global Fellowships progra...
27/08/2026
More Sasquatch, more fun: Today, the sneakiest crossover of the summer lands exc...
27/08/2026
Commissioned for RT KIDS as part of the EBU Kids' Drama Exchange The Note will premiere Sunday August 30, 5:10pm
RT One and RT Player
The Note tells a...
27/08/2026
August 27 2026, 06:00 (PDT) Dolby Laboratories Announces Leadership Transition
Kevin Yeaman to Retire from Dolby
Marc Whitten Named President, Chief Exe...
27/08/2026
NVIDIA's Gamescom announcements are revealing what's next for GeForce NOW, with new ways to play, more supported devices and platforms, and even more bi...
26/08/2026
Applicants interested in the Media Innovation Europe 3.0 Validation Booster call can now consult a new FAQ document addressing practical questions raised during...
26/08/2026
Roku Sports Channel will stream eight matches free beginning Aug. 30 as league shifts Sunday-night games away from Victory+...
26/08/2026
Evertz and Mobile TV Group detail how on-premises infrastructure and cloud resou...
26/08/2026
The Sennheiser Group will demonstrate its broadcast audio portfolio at IBC2026 (Hall 8, Booth D50), covering wireless microphone workflows, studio monitoring, a...
26/08/2026
Appear and Transmisiones Televisivas TVTEL Limitada have announced the deploymen...
26/08/2026
Leader Electronics has announced that Danish broadcast rental company Minitech has purchased a Leader LPX500 hybrid IP/SDI waveform monitor and LT4670 sync puls...
26/08/2026
Encompass will demonstrate its Altitude media services at IBC2026 (IAMT Hub, Diamond Lounge, Hall 3, RAI Amsterdam, September 11-14).
Key products on show incl...
26/08/2026
Daktronics has partnered with Massillon Washington High School in Ohio to install a new LED video display and implement the Daktronics Camino show control platf...
26/08/2026
Clear-Com has appointed Dubai-based Venuetech LLC as its main partner for the Middle East. Venuetech distributes professional audiovisual solutions across the r...
26/08/2026
NBC Sports NOW, NBC Sports' free 24/7 streaming channel, will launch a new weekday lineup beginning Monday, August 31, including two new shows.
Not So Fast...
26/08/2026
Calrec will demonstrate updates to its ImPulseV DSP software and Argo ecosystem at IBC2026 (Stand 8.C47), including previews aligned with the EBU's Dynamic ...
26/08/2026
Imagine Communications will demonstrate AI applications across its ad tech and video infrastructure portfolios at IBC2026 (Stand 1.B73, RAI Amsterdam, September...
26/08/2026
Dalet will demonstrate updates to Dalet Flex, Dalet Pyramid, and Dalia, its agentic AI platform, at IBC2026 (Hall 7, Stand 7.A43). The company will also partici...
26/08/2026
Team-owned platform will stream every locally carried Predators game beginning t...
26/08/2026
The 12-day tournament receives the broadcaster's full onsite-studio presence...
26/08/2026
A synth with no controls?
If you're into weird and wonderful instruments, then you may already be familiar with Warsaw-based SOMA Laboratory - if you...
26/08/2026
Physical modelling orchestral instrument announced
Following their recent cryptic hints, Dreamtonics have officially revealed the latest addition to their s...
26/08/2026
Discussions at the Meitingen site on competitiveness, energy supply, and fuel cell technology
Bertram Brossardt, CEO of the Bavarian metal and electrical emplo...
26/08/2026
Rohde & Schwarz USA celebrates grand opening of new manufacturing center in Fred...
26/08/2026
For large media companies, the content archive is a gold mine. There are thousands of hours of existing, high-value assets just waiting to be tapped for highlig...
26/08/2026
Ad Supported Viewing Represents 71.5% of TV in Q2
Streaming Captures Over 48% o...
26/08/2026
Charity sector ad spend reached $278 million as 44% of Australians donated to ch...
26/08/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...