
Large language models and the applications they power enable unprecedented opportunities for organizations to get deeper insights from their data reservoirs and to build entirely new classes of applications.
But with opportunities often come challenges.
Both on premises and in the cloud, applications that are expected to run in real time place significant demands on data center infrastructure to simultaneously deliver high throughput and low latency with one platform investment.
To drive continuous performance improvements and improve the return on infrastructure investments, NVIDIA regularly optimizes the state-of-the-art community models, including Meta's Llama, Google's Gemma, Microsoft's Phi and our own NVLM-D-72B, released just a few weeks ago.
Relentless Improvements Performance improvements let our customers and partners serve more complex models and reduce the needed infrastructure to host them. NVIDIA optimizes performance at every layer of the technology stack, including TensorRT-LLM, a purpose-built library to deliver state-of-the-art performance on the latest LLMs. With improvements to the open-source Llama 70B model, which delivers very high accuracy, we've already improved minimum latency performance by 3.5x in less than a year.
We're constantly improving our platform performance and regularly publish performance updates. Each week, improvements to NVIDIA software libraries are published, allowing customers to get more from the very same GPUs. For example, in just a few months' time, we've improved our low-latency Llama 70B performance by 3.5x.
NVIDIA has increased performance on the Llama 70B model by 3.5x. In the most recent round of MLPerf Inference 4.1, we made our first-ever submission with the Blackwell platform. It delivered 4x more performance than the previous generation.
This submission was also the first-ever MLPerf submission to use FP4 precision. Narrower precision formats, like FP4, reduces memory footprint and memory traffic, and also boost computational throughput. The process takes advantage of Blackwell's second-generation Transformer Engine, and with advanced quantization techniques that are part of TensorRT Model Optimizer, the Blackwell submission met the strict accuracy targets of the MLPerf benchmark.
Blackwell B200 delivers up to 4x more performance versus previous generation on MLPerf Inference v4.1's Llama 2 70B workload. Improvements in Blackwell haven't stopped the continued acceleration of Hopper. In the last year, Hopper performance has increased 3.4x in MLPerf on H100 thanks to regular software advancements. This means that NVIDIA's peak performance today, on Blackwell, is 10x faster than it was just one year ago on Hopper.
These results track progress on the MLPerf Inference Llama 2 70B Offline scenario over the past year. Our ongoing work is incorporated into TensorRT-LLM, a purpose-built library to accelerate LLMs that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM is built on top of the TensorRT Deep Learning Inference library and leverages much of TensorRT's deep learning optimizations with additional LLM-specific improvements.
Improving Llama in Leaps and Bounds More recently, we've continued optimizing variants of Meta's Llama models, including versions 3.1 and 3.2 as well as model sizes 70B and the biggest model, 405B. These optimizations include custom quantization recipes, as well as efficient use of parallelization techniques to more efficiently split the model across multiple GPUs, leveraging NVIDIA NVLink and NVSwitch interconnect technologies. Cutting-edge LLMs like Llama 3.1 405B are very demanding and require the combined performance of multiple state-of-the-art GPUs for fast responses.
Parallelism techniques require a hardware platform with a robust GPU-to-GPU interconnect fabric to get maximum performance and avoid communication bottlenecks. Each NVIDIA H200 Tensor Core GPU features fourth-generation NVLink, which provides a whopping 900GB/s of GPU-to-GPU bandwidth. Every eight-GPU HGX H200 platform also ships with four NVLink Switches, enabling every H200 GPU to communicate with any other H200 GPU at 900GB/s, simultaneously.
Many LLM deployments use parallelism over choosing to keep the workload on a single GPU, which can have compute bottlenecks. LLMs seek to balance low latency and high throughput, with the optimal parallelization technique depending on application requirements.
For instance, if lowest latency is the priority, tensor parallelism is critical, as the combined compute performance of multiple GPUs can be used to serve tokens to users more quickly. However, for use cases where peak throughput across all users is prioritized, pipeline parallelism can efficiently boost overall server throughput.
The table below shows that tensor parallelism can deliver over 5x more throughput in minimum latency scenarios, whereas pipeline parallelism brings 50% more performance for maximum throughput use cases.
For production deployments that seek to maximize throughput within a given latency budget, a platform needs to provide the ability to effectively combine both techniques like in TensorRT-LLM.
Read the technical blog on boosting Llama 3.1 405B throughput to learn more about these techniques.
Different scenarios have different requirements, and parallelism techniques bring optimal performance for each of these scenarios. The Virtuous Cycle Over the lifecycle of our architectures, we deliver significant performance gains from ongoing software tuning and optimization. These improvements translate into additional value for customers who train and deploy on our platforms. They're able to create more capable models and applications and deploy their existing models using less infrastructure, enhancing th
Most recent headlines
05/01/2027
Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...
06/09/2026
June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...
04/08/2026
Dalet, a leading technology and service provider for media-rich organizations, t...
08/07/2026
The end of the Big A' marks the demise of one more iconic sports venue and its signature sound
After 132 years as one of the preeminent horse-racing track...
08/07/2026
Memphis Athletics and Learfield have signed a long-term renewal to their 25-year relationship, which sees Learfield's Tiger Sports Properties continue repre...
08/07/2026
MLB All-Star Week returns to Philadelphia for the first time in 30 years as Amer...
08/07/2026
Former Wall Street Video event expanded to include all Enterprise sectors
SVG Enterprise has announced the inaugural Enterprise Video Summit, taking place Oct....
08/07/2026
An offseason project enhanced eight video displays around the venue in time for ...
08/07/2026
Kaseya Center has a ton of new features that weren't available to the indust...
08/07/2026
The International Federation of Film Critics (FIPRESCI) and the Sundance Film Festival are pleased to announce the establishment of a FIPRESCI jury beginning wi...
08/07/2026
Latest analogue-emulation plug-ins revealed
Universal Audio have just added two new analogue-emulation plug-ins to their DSP-powered UAD line-up. Developed ...
08/07/2026
Turn any audio loop into a playable kit
Pitch Innovations have recently introduced an interesting new tool that's capable of creating playable kits from...
08/07/2026
Adds new keyboard-split function
MonkeyC have recently launched an update that overhauls their powerful software sampler. Described as the sampler that mad...
08/07/2026
Four new delay algorithms & dual-engine functionality
Strymon have just announced that the much-anticipated MX version of their hugely popular delay pedal i...
08/07/2026
Well over half of Australia have tuned in to watch the FIFA World Cup on SBS
7 July, 2026
Media releases
Australia's Round of 32 match against Egypt a...
08/07/2026
SBS and Koorie Heritage Trust announce three-year partnership to celebrate First...
08/07/2026
Rohde & Schwarz and China Mobile Research Institute jointly demonstrate generati...
08/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
08/07/2026
Collaboration demonstrates how managed IP distribution and cloud-based syndication workflows can support live television at national scale
Fox Television Stati...
08/07/2026
New ARRI ALEXA 35 Live Xtreme camera seamlessly delivers up to 8x HFR via SDI fo...
08/07/2026
Did You Know? Redshift for Cinema 4D: The Power of the Distance Node
Marc Potocnik July 8, 2026
0 Comments
The Did You Know? Redshift for Cinema 4D ...
08/07/2026
Lightware is introducing the biggest update yet to the LARA Wizard Touchpanel configuration experience, designed to help integrators work faster while deliverin...
08/07/2026
DMC Production has turned to Calrec in its quest to redefine outside broadcasting with a flexible approach to live production that reverse engineers the convent...
08/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
08/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
08/07/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
08/07/2026
Foundry today announced that Reel FX Animation Studios has adopted Flix to standardize their pipeline and streamline processes across departments. Flix is a vis...
08/07/2026
The right team. The right time.' How The Killings at Parrish Station stuck t...
08/07/2026
Australians in Film and Screen Australia launch FUTURE VISION Forward to elevate...
08/07/2026
Film follows the courageous fight for justice by survivors of Ireland's Magdalene Laundries & Mother and Baby Homes
TESTIMONY airs Wednesday 22 July at 9...
08/07/2026
NVIDIA Nemotron 3 Ultra is offering leading performance at lower cost than top c...
07/07/2026
Applications are now open for the Validation Booster, a six-month acceleration programme supporting journalists, media organisations, content creators, media in...
07/07/2026
The Universal Talent Identifier (HAND) has issued HAND IDs to 50 top prospects ahead of the 2026 MLB Draft, including consensus top-ranked players Roch Cholowsk...
07/07/2026
Cisco and the McLaren Mastercard Formula 1 Team have announced an extension of their partnership, expanding the scope from networking and communications to incl...
07/07/2026
LucidLink has announced a workflow integration with Adobe Frame.io that gives production teams unified access to Frame.io projects alongside supporting producti...
07/07/2026
Chyron has announced the appointment of Ahmed Magd as Sales Director for the Middle East. Magd joins from previous senior commercial roles at Riedel Communicati...
07/07/2026
Leader Electronics of Europe has announced that its PHABRIX Qx rasterizer and Le...
07/07/2026
eCLUTCH has announced it has acquired broadcast rights to the Esports World Cup 2026 for EMEA and MENA, covering the Middle East, Africa, the Balkans, Sweden, a...
07/07/2026
Riedel Communications and ARRI supported ORF's host broadcast of the Eurovis...
07/07/2026
ARRI has announced the ALEXA 35 Live Xtreme, a camera that adds high frame rate ...
07/07/2026
LiveU will exhibit at IBC 2026 (Stand 7.C24), showcasing the LU900Q Intelligent Production Unit, expanded Sony collaboration, and additions to its open IP-video...
07/07/2026
Telemundo's coverage is packaged with Multiview, Catch Up with Key Plays, in...
07/07/2026
The Australian Rugby League Commission (ARLC) has announced a seven-year media r...
07/07/2026
Produced onsite at The Star in Frisco, TX, live coverage of the
two-day event airs on ESPN platforms
The season for Football Media Days is here, and the Big ...
07/07/2026
Rogers Communications Inc. has announced an agreement to acquire the remaining 25% ownership stake in Maple Leaf Sports and Entertainment (MLSE) from Kilmer Spo...
07/07/2026
End-to-end broadcast solution includes fleet of mobile units, Cloud Control, and...
07/07/2026
Sports-production-tech experts sound off on the state of the industry...
07/07/2026
New mixer aimed squarely at instrument-level sources
Heritage Audio have clearly been busy recently, as alongside their all-new Tubestrip, they've also ...
07/07/2026
New channel strip combines popular hardware designs
The latest arrival to the Heritage Audio line-up brings together a selection of their praised hardware d...
07/07/2026
eds3_5_jq(document).ready(function($) { $(#eds_sliderM519).chameleonSlider_2_1({ content_source:......