Large language models and the applications they power enable unprecedented opportunities for organizations to get deeper insights from their data reservoirs and to build entirely new classes of applications.
But with opportunities often come challenges.
Both on premises and in the cloud, applications that are expected to run in real time place significant demands on data center infrastructure to simultaneously deliver high throughput and low latency with one platform investment.
To drive continuous performance improvements and improve the return on infrastructure investments, NVIDIA regularly optimizes the state-of-the-art community models, including Meta's Llama, Google's Gemma, Microsoft's Phi and our own NVLM-D-72B, released just a few weeks ago.
Relentless Improvements Performance improvements let our customers and partners serve more complex models and reduce the needed infrastructure to host them. NVIDIA optimizes performance at every layer of the technology stack, including TensorRT-LLM, a purpose-built library to deliver state-of-the-art performance on the latest LLMs. With improvements to the open-source Llama 70B model, which delivers very high accuracy, we've already improved minimum latency performance by 3.5x in less than a year.
We're constantly improving our platform performance and regularly publish performance updates. Each week, improvements to NVIDIA software libraries are published, allowing customers to get more from the very same GPUs. For example, in just a few months' time, we've improved our low-latency Llama 70B performance by 3.5x.
NVIDIA has increased performance on the Llama 70B model by 3.5x. In the most recent round of MLPerf Inference 4.1, we made our first-ever submission with the Blackwell platform. It delivered 4x more performance than the previous generation.
This submission was also the first-ever MLPerf submission to use FP4 precision. Narrower precision formats, like FP4, reduces memory footprint and memory traffic, and also boost computational throughput. The process takes advantage of Blackwell's second-generation Transformer Engine, and with advanced quantization techniques that are part of TensorRT Model Optimizer, the Blackwell submission met the strict accuracy targets of the MLPerf benchmark.
Blackwell B200 delivers up to 4x more performance versus previous generation on MLPerf Inference v4.1's Llama 2 70B workload. Improvements in Blackwell haven't stopped the continued acceleration of Hopper. In the last year, Hopper performance has increased 3.4x in MLPerf on H100 thanks to regular software advancements. This means that NVIDIA's peak performance today, on Blackwell, is 10x faster than it was just one year ago on Hopper.
These results track progress on the MLPerf Inference Llama 2 70B Offline scenario over the past year. Our ongoing work is incorporated into TensorRT-LLM, a purpose-built library to accelerate LLMs that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM is built on top of the TensorRT Deep Learning Inference library and leverages much of TensorRT's deep learning optimizations with additional LLM-specific improvements.
Improving Llama in Leaps and Bounds More recently, we've continued optimizing variants of Meta's Llama models, including versions 3.1 and 3.2 as well as model sizes 70B and the biggest model, 405B. These optimizations include custom quantization recipes, as well as efficient use of parallelization techniques to more efficiently split the model across multiple GPUs, leveraging NVIDIA NVLink and NVSwitch interconnect technologies. Cutting-edge LLMs like Llama 3.1 405B are very demanding and require the combined performance of multiple state-of-the-art GPUs for fast responses.
Parallelism techniques require a hardware platform with a robust GPU-to-GPU interconnect fabric to get maximum performance and avoid communication bottlenecks. Each NVIDIA H200 Tensor Core GPU features fourth-generation NVLink, which provides a whopping 900GB/s of GPU-to-GPU bandwidth. Every eight-GPU HGX H200 platform also ships with four NVLink Switches, enabling every H200 GPU to communicate with any other H200 GPU at 900GB/s, simultaneously.
Many LLM deployments use parallelism over choosing to keep the workload on a single GPU, which can have compute bottlenecks. LLMs seek to balance low latency and high throughput, with the optimal parallelization technique depending on application requirements.
For instance, if lowest latency is the priority, tensor parallelism is critical, as the combined compute performance of multiple GPUs can be used to serve tokens to users more quickly. However, for use cases where peak throughput across all users is prioritized, pipeline parallelism can efficiently boost overall server throughput.
The table below shows that tensor parallelism can deliver over 5x more throughput in minimum latency scenarios, whereas pipeline parallelism brings 50% more performance for maximum throughput use cases.
For production deployments that seek to maximize throughput within a given latency budget, a platform needs to provide the ability to effectively combine both techniques like in TensorRT-LLM.
Read the technical blog on boosting Llama 3.1 405B throughput to learn more about these techniques.
Different scenarios have different requirements, and parallelism techniques bring optimal performance for each of these scenarios. The Virtuous Cycle Over the lifecycle of our architectures, we deliver significant performance gains from ongoing software tuning and optimization. These improvements translate into additional value for customers who train and deploy on our platforms. They're able to create more capable models and applications and deploy their existing models using less infrastructure, enhancing th
Most recent headlines
11/12/2025
Dalet, a leading provider of cloud-native, end-to-end media workflow solutions, ...
20/11/2025
LONDON The winners of the Rise Awards 2025, which recognize women and companies whose achievements have stood out in the media technology industry, have been an...
20/11/2025
SANTA MONICA & NEW YORK Lionsgate's Worldwide Television Distribution Group and Debmar-Mercury have launched MovieSphere Gold in more than 30 million homes....
20/11/2025
ALAMEDA, Calif. Clear-Com has introduced its four-channel HelixNet beltpack, a next-generation advancement of its widely used two-channel model, and anticipates...
20/11/2025
NEW PROVIDENCE, N.J. Streaming technology and services provider Kiswe has launched Kiswe Core, a next-generation cloud-based platform designed to simplify the s...
20/11/2025
MOUNTAIN VIEW, Calif. Dolby Laboratories has signed on as an official signature partner of the Bay Area Host Committee (BAHC) for the 2026 Super Bowl and the we...
20/11/2025
STUTTGART, Germany 3 Screen Solutions (3SS) has announced that the Canadian multiservice operator Telus has launched its Telus TV+ entertainment platform on Sam...
20/11/2025
Major League Baseball has announced deals with ESPN, NBCUniversal and Netflix covering 2026-2028 that make ESPN the exclusive rightsholder of MLB.TV, see regula...
20/11/2025
Berklee Presents Holiday Extravaganza: Duke Ellingtons The Nutcracker Suite and ...
19/11/2025
Sphere Sound Kicks Up Its Heels at Radio City Music HallUltra-immersive system from the singular Las Vegas venue is also ready for sportsBy Dan Daley, Audio Edi...
19/11/2025
The Future of Stadium and Arena Video Control Rooms: Tech Leaders Talk 4K, IP, a...
19/11/2025
One Year at Intuit Dome: Walking Through the Stunning Arena's Advanced Video...
19/11/2025
SVG Sit-Down: IMG's Francois Westcombe Goes Inside IMG's Annual Digital ...
19/11/2025
Inside Brighton & Hove Albion's fan-first media machine By George Bevir
Tuesday, November 18, 2025 - 10:10
Print This Story
Brighton & Hove Albion'...
19/11/2025
Rock Chalk Rebuild, Part 1: Kansas Athletics Brings New Production Power to Rebu...
19/11/2025
Rock Chalk Rebuild, Part 2: A Pro's Guide to Kansas Athletics' New Video...
19/11/2025
MLB Media Rights Shakeup: NBC Inks Three-Year Deal for Sunday Night Baseball, Pe...
19/11/2025
Rohan Parashuram Kanawade attends the premiere of Sabar Bonda (Cactus Pears) at the 2025 Sundance Film Festival at Egyptian Theatre on January 26, 2025, in Pa...
19/11/2025
Awards season is heating up, and the International Documentary Association just ...
19/11/2025
Aerojet Rocketdyne President Ken Bedingfield and Arkansas Gov. Sarah Huckabee Sanders were joined by state and local officials to celebrate the groundbreaking o...
19/11/2025
L3Harris and PentenAmio formalise their teaming agreement at MilCIS 2025, streng...
19/11/2025
Calrec gives Phoenix Broadcast Solutions its full support in strategic multi-year enterprise partnership New Singapore OB company invests in Calrec technology f...
19/11/2025
October brings a larger audience in front of TV screens. The cooler autumn weather and the continuation of new programming schedules brought increases in both t...
19/11/2025
IRVING, Texas Nexstar Media Group and Tegna filed applications on Nov. 18 with the Federal Communications Commission (FCC) seeking its consent to transfer broad...
19/11/2025
Vitec has acquired Datapath, a developer of real-time video processing for large-scale video walls, AVoIP content distribution and KVM control in mission-critic...
19/11/2025
NEVADA CITY, Calif. Telestream has introduced ARGUS Version 2.3, which introduces Live Look, a feature that lets operators inspect live and on-demand streams vi...
19/11/2025
STAMFORD, Conn. and NEW YORK Charter Communications, Inc. has announced a major deal with Amazon Web Services (AWS) that establishes AWS as one of Charters stra...
19/11/2025
TOKYO Atomos has announced a major firmware update that brings integrated camera control to the Ninja TX GO and Ninja TX its new CFexpress-based monitor-recor...
19/11/2025
Berklee's Signature Series Reimagines the Musical and Visual Artistry of Col...
19/11/2025
PlayBox Neo departed the NAB Show New York 2025 on a high note. The company's all-new PlayBox Neo Suite made a splash on the show floor, drawing strong inte...
19/11/2025
AAVBR (American Audio Visual Baton Rouge), a leading full-service event production company serving Louisiana and the Gulf Coast region, has enhanced its product...
19/11/2025
In its mission to deliver live event services combining advanced engineering with a creative approach to captivating live sports and entertainment experiences, ...
19/11/2025
NEW YORK Short-form vertical video has exploded across platforms like TikTok, Instagram and YouTube, but a new survey commissioned by Media.net, a provider of c...
19/11/2025
NEW YORK YES Network, the regional sports network home of the New York Yankees and Brooklyn Nets, and CAMB.AI have struck a partnership they said will leverage ...
19/11/2025
WASHINGTON The Federal Communications Commission has issued detailed guidance on filings that were disrupted by the government shutdown. While many of those fil...
19/11/2025
NEW YORK The popularity of NFL games is largely responsible for continued seasonal momentum in TV viewership into October (measured Sept. 29-Oct. 26), according...
19/11/2025
DirecTV has announced that its MyFree DirecTV streaming service has added seven new channels. Those include four sports services, NBA FAST, Red Bull TV, DAZN Ri...
19/11/2025
SURREY, U.K. Mark Roberts Motion Control (MRMC) has launched Flair Bridge, a compact device that transforms how operators control the company's motion-contr...
19/11/2025
WASHINGTON The Federal Communications Commission, which is continuing to rapidly ramp up its operations after the end of the federal government shutdown, has se...
19/11/2025
IRVING, Texas Nexstar Media Group and Tegna filed applications on Nov. 18 with the Federal Communications Commission (FCC) seeking its consent to transfer broad...
19/11/2025
Martin Freeman, Jessie Buckley, Wu-Tang Clan's Raekwon, Eddie the Eagle and ...
19/11/2025
Rohde & Schwarz collaborates with Broadcom to enable testing and validation of n...
19/11/2025
Back to All News
Innato, starring Imanol Arias and Elena Anaya, arrives on Netf...
19/11/2025
Back to All News
Guillermo del Toro Sits Down With Troll 2 Director Roar Uthaug...
19/11/2025
Some projects feel monumental from the moment the first phone call comes in. When BCE, the host broadcaster appointed to coordinate the national coverage, asked...
19/11/2025
November 19 2025, 08:00 (PST) BAY AREA HOST COMMITTEE AND DOLBY PARTNER TO DELIVER IMMERSIVE FAN EXPERIENCES
During the Week of Football's Biggest Game...
19/11/2025
Michael D. Higgins: Ireland's Ninth President features RT Archives footage and interviews across seven decades
Watch Wednesday 19 November at 9:35pm on RT...
19/11/2025
Scripps Research scientists receive $1.1 million to advance AI modeling for HIV vaccine development New AI system helps scientists rapidly pinpoint the most pro...
18/11/2025
Independent media across Central America are operating under intensifying financial pressure, yet there is a clear appetite for models that can sustain both ind...
Large language models and the applications they power enable unprecedented opportunities for organizations to get deeper insights from their data reservoirs and to build entirely new classes of applications.
