
Large language models and the applications they power enable unprecedented opportunities for organizations to get deeper insights from their data reservoirs and to build entirely new classes of applications.
But with opportunities often come challenges.
Both on premises and in the cloud, applications that are expected to run in real time place significant demands on data center infrastructure to simultaneously deliver high throughput and low latency with one platform investment.
To drive continuous performance improvements and improve the return on infrastructure investments, NVIDIA regularly optimizes the state-of-the-art community models, including Meta's Llama, Google's Gemma, Microsoft's Phi and our own NVLM-D-72B, released just a few weeks ago.
Relentless Improvements Performance improvements let our customers and partners serve more complex models and reduce the needed infrastructure to host them. NVIDIA optimizes performance at every layer of the technology stack, including TensorRT-LLM, a purpose-built library to deliver state-of-the-art performance on the latest LLMs. With improvements to the open-source Llama 70B model, which delivers very high accuracy, we've already improved minimum latency performance by 3.5x in less than a year.
We're constantly improving our platform performance and regularly publish performance updates. Each week, improvements to NVIDIA software libraries are published, allowing customers to get more from the very same GPUs. For example, in just a few months' time, we've improved our low-latency Llama 70B performance by 3.5x.
NVIDIA has increased performance on the Llama 70B model by 3.5x. In the most recent round of MLPerf Inference 4.1, we made our first-ever submission with the Blackwell platform. It delivered 4x more performance than the previous generation.
This submission was also the first-ever MLPerf submission to use FP4 precision. Narrower precision formats, like FP4, reduces memory footprint and memory traffic, and also boost computational throughput. The process takes advantage of Blackwell's second-generation Transformer Engine, and with advanced quantization techniques that are part of TensorRT Model Optimizer, the Blackwell submission met the strict accuracy targets of the MLPerf benchmark.
Blackwell B200 delivers up to 4x more performance versus previous generation on MLPerf Inference v4.1's Llama 2 70B workload. Improvements in Blackwell haven't stopped the continued acceleration of Hopper. In the last year, Hopper performance has increased 3.4x in MLPerf on H100 thanks to regular software advancements. This means that NVIDIA's peak performance today, on Blackwell, is 10x faster than it was just one year ago on Hopper.
These results track progress on the MLPerf Inference Llama 2 70B Offline scenario over the past year. Our ongoing work is incorporated into TensorRT-LLM, a purpose-built library to accelerate LLMs that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM is built on top of the TensorRT Deep Learning Inference library and leverages much of TensorRT's deep learning optimizations with additional LLM-specific improvements.
Improving Llama in Leaps and Bounds More recently, we've continued optimizing variants of Meta's Llama models, including versions 3.1 and 3.2 as well as model sizes 70B and the biggest model, 405B. These optimizations include custom quantization recipes, as well as efficient use of parallelization techniques to more efficiently split the model across multiple GPUs, leveraging NVIDIA NVLink and NVSwitch interconnect technologies. Cutting-edge LLMs like Llama 3.1 405B are very demanding and require the combined performance of multiple state-of-the-art GPUs for fast responses.
Parallelism techniques require a hardware platform with a robust GPU-to-GPU interconnect fabric to get maximum performance and avoid communication bottlenecks. Each NVIDIA H200 Tensor Core GPU features fourth-generation NVLink, which provides a whopping 900GB/s of GPU-to-GPU bandwidth. Every eight-GPU HGX H200 platform also ships with four NVLink Switches, enabling every H200 GPU to communicate with any other H200 GPU at 900GB/s, simultaneously.
Many LLM deployments use parallelism over choosing to keep the workload on a single GPU, which can have compute bottlenecks. LLMs seek to balance low latency and high throughput, with the optimal parallelization technique depending on application requirements.
For instance, if lowest latency is the priority, tensor parallelism is critical, as the combined compute performance of multiple GPUs can be used to serve tokens to users more quickly. However, for use cases where peak throughput across all users is prioritized, pipeline parallelism can efficiently boost overall server throughput.
The table below shows that tensor parallelism can deliver over 5x more throughput in minimum latency scenarios, whereas pipeline parallelism brings 50% more performance for maximum throughput use cases.
For production deployments that seek to maximize throughput within a given latency budget, a platform needs to provide the ability to effectively combine both techniques like in TensorRT-LLM.
Read the technical blog on boosting Llama 3.1 405B throughput to learn more about these techniques.
Different scenarios have different requirements, and parallelism techniques bring optimal performance for each of these scenarios. The Virtuous Cycle Over the lifecycle of our architectures, we deliver significant performance gains from ongoing software tuning and optimization. These improvements translate into additional value for customers who train and deploy on our platforms. They're able to create more capable models and applications and deploy their existing models using less infrastructure, enhancing th
Most recent headlines
05/01/2027
Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...
06/09/2026
June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...
04/08/2026
Dalet, a leading technology and service provider for media-rich organizations, t...
08/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
08/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
08/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
08/07/2026
Foundry today announced that Reel FX Animation Studios has adopted Flix to standardize their pipeline and streamline processes across departments. Flix is a vis...
08/07/2026
The right team. The right time.' How The Killings at Parrish Station stuck t...
08/07/2026
Australians in Film and Screen Australia launch FUTURE VISION Forward to elevate...
07/07/2026
Applications are now open for the Validation Booster, a six-month acceleration programme supporting journalists, media organisations, content creators, media in...
07/07/2026
The Universal Talent Identifier (HAND) has issued HAND IDs to 50 top prospects ahead of the 2026 MLB Draft, including consensus top-ranked players Roch Cholowsk...
07/07/2026
Cisco and the McLaren Mastercard Formula 1 Team have announced an extension of their partnership, expanding the scope from networking and communications to incl...
07/07/2026
LucidLink has announced a workflow integration with Adobe Frame.io that gives production teams unified access to Frame.io projects alongside supporting producti...
07/07/2026
Chyron has announced the appointment of Ahmed Magd as Sales Director for the Middle East. Magd joins from previous senior commercial roles at Riedel Communicati...
07/07/2026
Leader Electronics of Europe has announced that its PHABRIX Qx rasterizer and Le...
07/07/2026
eCLUTCH has announced it has acquired broadcast rights to the Esports World Cup 2026 for EMEA and MENA, covering the Middle East, Africa, the Balkans, Sweden, a...
07/07/2026
Riedel Communications and ARRI supported ORF's host broadcast of the Eurovis...
07/07/2026
ARRI has announced the ALEXA 35 Live Xtreme, a camera that adds high frame rate ...
07/07/2026
LiveU will exhibit at IBC 2026 (Stand 7.C24), showcasing the LU900Q Intelligent Production Unit, expanded Sony collaboration, and additions to its open IP-video...
07/07/2026
Telemundo's coverage is packaged with Multiview, Catch Up with Key Plays, in...
07/07/2026
The Australian Rugby League Commission (ARLC) has announced a seven-year media r...
07/07/2026
Produced onsite at The Star in Frisco, TX, live coverage of the two-day event airs on ESPN platforms
The season for Football Media Days is here, and the Big 12...
07/07/2026
Rogers Communications Inc. has announced an agreement to acquire the remaining 25% ownership stake in Maple Leaf Sports and Entertainment (MLSE) from Kilmer Spo...
07/07/2026
End-to-end broadcast solution includes fleet of mobile units, Cloud Control, and...
07/07/2026
Sports-production-tech experts sound off on the state of the industry...
07/07/2026
New mixer aimed squarely at instrument-level sources
Heritage Audio have clearly been busy recently, as alongside their all-new Tubestrip, they've also ...
07/07/2026
New channel strip combines popular hardware designs
The latest arrival to the Heritage Audio line-up brings together a selection of their praised hardware d...
07/07/2026
eds3_5_jq(document).ready(function($) { $(#eds_sliderM519).chameleonSlider_2_1({ content_source:......
07/07/2026
New Kessler 10 data now available in Nielsen CMI gives brands a deeper view of h...
07/07/2026
New Nielsen Ad Intel and CMV data reveals rising competition in Australia's ...
07/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
07/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
07/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
07/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
07/07/2026
Riedel Communications and ARRI played a key role in one of the most technically ambitious Eurovision Song Contest productions to date. For the first time in the...
07/07/2026
Berklee Valencia Honors Eric Whitacre at the 2026 Commencement Ceremony The Grammy-winning composer was celebrated alongside this years cohort of masters degr...
07/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
07/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
07/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
07/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
07/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
07/07/2026
The appointment will drive growth and innovation across the Media Business Unit
Fincons Group, a leading IT business consultancy and systems integrator company...
07/07/2026
International broadcast and streaming expert Big Blue Marble has announced a multi-month Proof of Concept (PoC) for Media over QUIC (MoQ) during a major interna...
07/07/2026
Test & measurement innovator, Leader Electronics of Europe, is proud to announce that two of its leading products, the PHABRIX Qx rasterizer and LeaderPhabrix L...
07/07/2026
Clear-Com announced that Ross Production Services (RPS), a leading provider of mobile production solutions for live events and broadcasts, has standardized on ...
07/07/2026
LiveU redefined live newsgathering with the invention of cellular bonding. At IBC2026, Stand 7.C24, the company will show how that foundation is evolving into t...
07/07/2026
LucidLink, a leader in cloud-based file services for content creators, today announced an enhanced workflow integration between LucidLink and Adobe Frame.io tha...
07/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
07/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
07/07/2026
New appointments to Screen Australia board 7 July 2026
The Albanese Labor Government has made new appointments to the Screen Australia board.
Dr Tania Chamber...