Large language models and the applications they power enable unprecedented opportunities for organizations to get deeper insights from their data reservoirs and to build entirely new classes of applications.
But with opportunities often come challenges.
Both on premises and in the cloud, applications that are expected to run in real time place significant demands on data center infrastructure to simultaneously deliver high throughput and low latency with one platform investment.
To drive continuous performance improvements and improve the return on infrastructure investments, NVIDIA regularly optimizes the state-of-the-art community models, including Meta's Llama, Google's Gemma, Microsoft's Phi and our own NVLM-D-72B, released just a few weeks ago.
Relentless Improvements Performance improvements let our customers and partners serve more complex models and reduce the needed infrastructure to host them. NVIDIA optimizes performance at every layer of the technology stack, including TensorRT-LLM, a purpose-built library to deliver state-of-the-art performance on the latest LLMs. With improvements to the open-source Llama 70B model, which delivers very high accuracy, we've already improved minimum latency performance by 3.5x in less than a year.
We're constantly improving our platform performance and regularly publish performance updates. Each week, improvements to NVIDIA software libraries are published, allowing customers to get more from the very same GPUs. For example, in just a few months' time, we've improved our low-latency Llama 70B performance by 3.5x.
NVIDIA has increased performance on the Llama 70B model by 3.5x. In the most recent round of MLPerf Inference 4.1, we made our first-ever submission with the Blackwell platform. It delivered 4x more performance than the previous generation.
This submission was also the first-ever MLPerf submission to use FP4 precision. Narrower precision formats, like FP4, reduces memory footprint and memory traffic, and also boost computational throughput. The process takes advantage of Blackwell's second-generation Transformer Engine, and with advanced quantization techniques that are part of TensorRT Model Optimizer, the Blackwell submission met the strict accuracy targets of the MLPerf benchmark.
Blackwell B200 delivers up to 4x more performance versus previous generation on MLPerf Inference v4.1's Llama 2 70B workload. Improvements in Blackwell haven't stopped the continued acceleration of Hopper. In the last year, Hopper performance has increased 3.4x in MLPerf on H100 thanks to regular software advancements. This means that NVIDIA's peak performance today, on Blackwell, is 10x faster than it was just one year ago on Hopper.
These results track progress on the MLPerf Inference Llama 2 70B Offline scenario over the past year. Our ongoing work is incorporated into TensorRT-LLM, a purpose-built library to accelerate LLMs that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM is built on top of the TensorRT Deep Learning Inference library and leverages much of TensorRT's deep learning optimizations with additional LLM-specific improvements.
Improving Llama in Leaps and Bounds More recently, we've continued optimizing variants of Meta's Llama models, including versions 3.1 and 3.2 as well as model sizes 70B and the biggest model, 405B. These optimizations include custom quantization recipes, as well as efficient use of parallelization techniques to more efficiently split the model across multiple GPUs, leveraging NVIDIA NVLink and NVSwitch interconnect technologies. Cutting-edge LLMs like Llama 3.1 405B are very demanding and require the combined performance of multiple state-of-the-art GPUs for fast responses.
Parallelism techniques require a hardware platform with a robust GPU-to-GPU interconnect fabric to get maximum performance and avoid communication bottlenecks. Each NVIDIA H200 Tensor Core GPU features fourth-generation NVLink, which provides a whopping 900GB/s of GPU-to-GPU bandwidth. Every eight-GPU HGX H200 platform also ships with four NVLink Switches, enabling every H200 GPU to communicate with any other H200 GPU at 900GB/s, simultaneously.
Many LLM deployments use parallelism over choosing to keep the workload on a single GPU, which can have compute bottlenecks. LLMs seek to balance low latency and high throughput, with the optimal parallelization technique depending on application requirements.
For instance, if lowest latency is the priority, tensor parallelism is critical, as the combined compute performance of multiple GPUs can be used to serve tokens to users more quickly. However, for use cases where peak throughput across all users is prioritized, pipeline parallelism can efficiently boost overall server throughput.
The table below shows that tensor parallelism can deliver over 5x more throughput in minimum latency scenarios, whereas pipeline parallelism brings 50% more performance for maximum throughput use cases.
For production deployments that seek to maximize throughput within a given latency budget, a platform needs to provide the ability to effectively combine both techniques like in TensorRT-LLM.
Read the technical blog on boosting Llama 3.1 405B throughput to learn more about these techniques.
Different scenarios have different requirements, and parallelism techniques bring optimal performance for each of these scenarios. The Virtuous Cycle Over the lifecycle of our architectures, we deliver significant performance gains from ongoing software tuning and optimization. These improvements translate into additional value for customers who train and deploy on our platforms. They're able to create more capable models and applications and deploy their existing models using less infrastructure, enhancing th
Most recent headlines
05/01/2027
Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...
09/10/2026
September 10 2026, 06:00 (PDT) Dolby Expands Dolby OptiView Platform with New Capabilities at IBC 2026
New Sports Intelligence helps providers better unders...
07/10/2026
Dalet, a leading technology and service provider for media-rich organizations, today announced the latest Long-Term Supported (LTS) release of Dalet Flex. Build...
29/09/2026
Oak View Group (OVG) has announced Cisco as an Official Technology Partner, with plans to standardize infrastructure across its venue network on Cisco solutions...
29/09/2026
SportsNet Pittsburgh (SNP) won three honors at the 2026 Mid-Atlantic Emmy Awards.
Program awards went to Bucs BP in the Sports Program Live category (produced ...
29/09/2026
The Professional Women's Hockey League has announced home openers for all 12 teams for the 2026-27 season, along with a broadcast partnership with CBC under...
29/09/2026
Manifold technologies has been recognized as a contributor to the Media eXchange...
29/09/2026
FuboTV has announced My Local Sports, allowing subscribers in select markets to add their local regional sports network (RSN) as an add-on to a Fubo Sports subs...
29/09/2026
Asian News International (ANI), a wire agency and video news provider based in India, has deployed Grass Valley's AMPP Media Operating System and Framelight...
29/09/2026
South African Broadcasting Corporation (SABC) has deployed Ikegami UHK-X600 cameras as part of an upgrade to IP-based production infrastructure. The deployment ...
29/09/2026
REMI, REMCO models will drive the bulk of the broadcaster's NHL productions ...
29/09/2026
Three years in, the facility is supporting more centralized race production, com...
29/09/2026
NBC, Peacock, and NBSN are carrying all four Wild Card series with a production ...
29/09/2026
19 Nonfiction Projects Across All Stages of Production to Receive More Than $1 million in Support as Documentary Film Program Approaches 25th Anniversary
LOS ...
29/09/2026
Two new plug-ins launch alongside updated bundles
Zynaptiq have just announced the launch of two new plug-ins, alongside new and improved versions of two of...
29/09/2026
All-in-one bus processor revealed
Wes Audio's ng product range has come a long way since it was first introduced, and now includes a tasty selection of ...
29/09/2026
eds3_5_jq(document).ready(function($) { $(#eds_sliderM519).chameleonSlider_2_1({ content_source:......
29/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
29/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
29/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
29/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
29/09/2026
Ikegami announces that the South African Broadcasting Corporation, SABC, has chosen Ikegami UHK-X600 camera systems as part of an advance to IP-based operation ...
29/09/2026
Blendkit Offers 20% Discount to Celebrate Crossing 150K Asset Milestone
Brie Clayton September 29, 2026
0 Comments
Blendkit, the largest 3D asset libr...
29/09/2026
Interactive streaming specialist Ease Live supported more than 76.5 million fan sessions across three major European media operators during this summers FIFA Wo...
29/09/2026
Ease Live has added support for VAST (Video Ad Serving Template) to its graphical overlay technology, opening up new ways for broadcasters, streamers and rights...
29/09/2026
Rise AV has announced the launch of the inaugural Rise AV Awards, a new global awards programme celebrating the women, advocates and organisations across the AV...
29/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
29/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
29/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
29/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
29/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
29/09/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
29/09/2026
Clear-Com announced that First Baptist Nashville has upgraded its production communications with a Clear-Com intercom solution as part of a comprehensive sanct...
29/09/2026
Blackmagic Cinema Camera 6K Captured Action Shots for Sketch
Brie Clayton September 28, 2026
0 Comments
Camera's portable design and dynamic range...
29/09/2026
Kean Production Services Deploys AJA ColorBox to Deliver Consistent, Accurate Co...
29/09/2026
Berklee Alumni and Faculty Nominated for the 2026 Latin Grammy Awards A total of 28 Berklee alumni and faculty earned nominations across 20 categories this ye...
29/09/2026
Berklee's K-Pop & Beyond Explores Connections Between K-Pop and Hip-Hop Now in its fifth year, the series brings together artists, industry leaders, and s...
29/09/2026
We tend to think we live in a digital world, with hopes and fears around artificial intelligence (AI), quantum computing, data centres and the next big technolo...
29/09/2026
Today, in partnership with the American Academy of Pediatrics (AAP), Apple relea...
29/09/2026
Final Cut Camera, the professional filming app for iPhone, receives a major upda...
29/09/2026
RT lyric fm is proud to announce the annual Choirs for Christmas competition callout. Warm up those vocal chords and join us on RT lyric fm for a celebration ...
28/09/2026
The NHL has announced Adobe as an Official Creative, Marketing and AI Partner beginning with the 2026-27 season. The partnership will provide the league and its...
28/09/2026
MultiDyne will exhibit at CABSAT 2026 (October 5-7, Dubai Exhibition Centre), showcasing an expanded technology portfolio following its June 2026 acquisition of...
28/09/2026
ARRI and HONOR have announced the HONOR Magic9 Pro Max, a flagship smartphone carrying ARRI branding on the device and packaging as the first product-level expr...
28/09/2026
Net Insight has added the Nimbra 1020G to its pan-Asian live media trial network, giving customers and partners in the region a way to evaluate ST 2110 and JPEG...
28/09/2026
Vindral has been named a recipient of a 2026 Technology and Engineering Emmy Award by the National Academy of Television Arts and Sciences in the category of We...
28/09/2026
Interra Systems will exhibit at CABSAT 2026 (Stand B2-37, Dubai Exhibition Centre, October 5-7), demonstrating its BATON QC platform, ORION monitoring solutions...
28/09/2026
Two years into its new production facility, the TOUR is expanding centralized wo...
Large language models and the applications they power enable unprecedented opportunities for organizations to get deeper insights from their data reservoirs and to build entirely new classes of applications.
