
In the latest MLPerf Inference V5.0 benchmarks, which reflect some of the most challenging inference scenarios, the NVIDIA Blackwell platform set records - and marked NVIDIA's first MLPerf submission using the NVIDIA GB200 NVL72 system, a rack-scale solution designed for AI reasoning.
Delivering on the promise of cutting-edge AI takes a new kind of compute infrastructure, called AI factories. Unlike traditional data centers, AI factories do more than store and process data - they manufacture intelligence at scale by transforming raw data into real-time insights. The goal for AI factories is simple: deliver accurate answers to queries quickly, at the lowest cost and to as many users as possible.
The complexity of pulling this off is significant and takes place behind the scenes. As AI models grow to billions and trillions of parameters to deliver smarter replies, the compute required to generate each token increases. This requirement reduces the number of tokens that an AI factory can generate and increases cost per token. Keeping inference throughput high and cost per token low requires rapid innovation across every layer of the technology stack, spanning silicon, network systems and software.
The latest updates to MLPerf Inference, a peer-reviewed industry benchmark of inference performance, include the addition of Llama 3.1 405B, one of the largest and most challenging-to-run open-weight models. The new Llama 2 70B Interactive benchmark features much stricter latency requirements compared with the original Llama 2 70B benchmark, better reflecting the constraints of production deployments in delivering the best possible user experiences.
In addition to the Blackwell platform, the NVIDIA Hopper platform demonstrated exceptional performance across the board, with performance increasing significantly over the last year on Llama 2 70B thanks to full-stack optimizations.
NVIDIA Blackwell Sets New Records The GB200 NVL72 system - connecting 72 NVIDIA Blackwell GPUs to act as a single, massive GPU - delivered up to 30x higher throughput on the Llama 3.1 405B benchmark over the NVIDIA H200 NVL8 submission this round. This feat was achieved through more than triple the performance per GPU and a 9x larger NVIDIA NVLink interconnect domain.
While many companies run MLPerf benchmarks on their hardware to gauge performance, only NVIDIA and its partners submitted and published results on the Llama 3.1 405B benchmark.
Production inference deployments often have latency constraints on two key metrics. The first is time to first token (TTFT), or how long it takes for a user to begin seeing a response to a query given to a large language model. The second is time per output token (TPOT), or how quickly tokens are delivered to the user.
The new Llama 2 70B Interactive benchmark has a 5x shorter TPOT and 4.4x lower TTFT - modeling a more responsive user experience. On this test, NVIDIA's submission using an NVIDIA DGX B200 system with eight Blackwell GPUs tripled performance over using eight NVIDIA H200 GPUs, setting a high bar for this more challenging version of the Llama 2 70B benchmark.
Combining the Blackwell architecture and its optimized software stack delivers new levels of inference performance, paving the way for AI factories to deliver higher intelligence, increased throughput and faster token rates.
NVIDIA Hopper AI Factory Value Continues Increasing The NVIDIA Hopper architecture, introduced in 2022, powers many of today's AI inference factories, and continues to power model training. Through ongoing software optimization, NVIDIA increases the throughput of Hopper-based AI factories, leading to greater value.
On the Llama 2 70B benchmark, first introduced a year ago in MLPerf Inference v4.0, H100 GPU throughput has increased by 1.5x. The H200 GPU, based on the same Hopper GPU architecture with larger and faster GPU memory, extends that increase to 1.6x.
Hopper also ran every benchmark, including the newly added Llama 3.1 405B, Llama 2 70B Interactive and graph neural network tests. This versatility means Hopper can run a wide range of workloads and keep pace as models and usage scenarios grow more challenging.
It Takes an Ecosystem This MLPerf round, 15 partners submitted stellar results on the NVIDIA platform, including ASUS, Cisco, CoreWeave, Dell Technologies, Fujitsu, Giga Computing, Google Cloud, Hewlett Packard Enterprise, Lambda, Lenovo, Oracle Cloud Infrastructure, Quanta Cloud Technology, Supermicro, Sustainable Metal Cloud and VMware.
The breadth of submissions reflects the reach of the NVIDIA platform, which is available across all cloud service providers and server makers worldwide.
MLCommons' work to continuously evolve the MLPerf Inference benchmark suite to keep pace with the latest AI developments and provide the ecosystem with rigorous, peer-reviewed performance data is vital to helping IT decision makers select optimal AI infrastructure.
Learn more about MLPerf.
Images and video taken at an Equinix data center in the Silicon Valley.
More from Nvidia
30/04/2025
The quick-service restaurant (QSR) industry is being reinvented by AI.
For exam...
30/04/2025
AI-powered image generation has progressed at a remarkable pace - from early exa...
28/04/2025
As enterprises increasingly adopt AI, securing AI factories - where complex, agentic workflows are executed - has never been more critical.
NVIDIA is bringing ...
28/04/2025
Agentic AI is redefining the cybersecurity landscape - introducing new opportunities that demand rethinking how to secure AI while offering the keys to addressi...
28/04/2025
Oracle has stood up and optimized its first wave of liquid-cooled NVIDIA GB200 N...
24/04/2025
Advancing AI requires a full-stack approach, with a powerful foundation of computing infrastructure - including accelerated processors and networking technologi...
24/04/2025
Get the controllers ready and clear the calendar - it's a jam-packed GFN Thu...
23/04/2025
As AI models evolve and adoption grows, enterprises must perform a delicate balancing act to achieve maximum value.
That's because inference - the process ...
23/04/2025
Financial services has long been at the forefront of adopting technological innovations. Today, generative AI and agentic systems are redefining the industry, f...
23/04/2025
AI is rapidly reshaping what's possible on a PC - whether for real-time image generation or voice-controlled workflows. As AI capabilities grow, so does the...
23/04/2025
An AI agent is only as accurate, relevant and timely as the data that powers it....
22/04/2025
Whether at sea, land or in the sky - even outer space - NVIDIA technology is helping research scientists and developers alike explore and understand oceans, wil...
22/04/2025
Traditionally, data centers have relied on air cooling - where mechanical chillers circulate chilled air to absorb heat from servers, helping them maintain opti...
22/04/2025
About 15% of the world's population - over a billion people - are affected by neurological disorders, from commonly known diseases like Alzheimer's and ...
17/04/2025
Gold prospecting in Venezuela has led to a malaria resurgence, but researchers h...
17/04/2025
As the days grow longer and the flowers bloom, GFN Thursday brings a fresh lineup of games to brighten the week.
Dive into thrilling hunts and dark fantasy adv...
16/04/2025
Isomorphic Labs is reimagining the drug discovery process with an AI-first approach. At the heart of this work is a new way of thinking about biology.
Max Jade...
16/04/2025
Editor's note: This post is part of Into the Omniverse, a series focused on how developers, 3D practitioners, and enterprises can transform their workflows ...
15/04/2025
The final days of the AI Mathematical Olympiad's latest competition were a transcontinental relay for team NVIDIA.
Every evening, two team members on oppos...
15/04/2025
Every company and country wants to grow and create economic opportunity - but they need virtually limitless intelligence to do so. Working with its ecosystem pa...
15/04/2025
CoreWeave today became one of the first cloud providers to bring NVIDIA GB200 NV...
14/04/2025
NVIDIA is working with its manufacturing partners to design and build factories that, for the first time, will produce NVIDIA AI supercomputers entirely in the ...
11/04/2025
As a teenager, Bradley Rothenberg was obsessed with CAD: computer-aided design software.
Before he turned 30, Rothenberg channeled that interest into building ...
10/04/2025
NVIDIA this week recognized the contributions of partners in Europe, the Middle ...
10/04/2025
As AI-powered tools continue to evolve, NVIDIA GeForce RTX 50 Series and NVIDIA ...
10/04/2025
Get ready to explore the Deep South. South of Midnight, the action-adventure gam...
09/04/2025
Necessity is the mother of invention. And sometimes, what a person really needs ...
09/04/2025
NVIDIA is collaborating with Google Cloud to bring agentic AI to enterprises seeking to locally harness the Google Gemini family of AI models using the NVIDIA B...
06/04/2025
This National Robotics Week, NVIDIA highlighted the pioneering technologies that...
03/04/2025
The Nintendo Switch 2, unveiled April 2, takes performance to the next level, powered by a custom NVIDIA processor featuring an NVIDIA GPU with dedicated RT Cor...
03/04/2025
Real-time AI is unlocking new possibilities in media and entertainment, improving viewer engagement and advancing intelligent content creation.
At NAB Show, a...
03/04/2025
Editor's note: This post is part of the AI On blog series, which explores the latest techniques and real-world applications of agentic AI, chatbots and copi...
03/04/2025
GeForce NOW isn't fooling around.
This month, 21 games are joining the cloud gaming library of over 2,000 titles. Whether chasing epic adventures, testing ...
02/04/2025
In the latest MLPerf Inference V5.0 benchmarks, which reflect some of the most c...
02/04/2025
AI is rapidly transforming how organizations solve complex challenges.
The early stages of enterprise AI adoption focused on using large language models to cre...
02/04/2025
Video editing workflows are getting a lot more colorful.
Adobe recently announc...
31/03/2025
Advances in physical AI are enabling organizations to embrace embodied AI across...
27/03/2025
A new resident is moving into the cloud - KRAFTON's inZOI joins the 2,000+ games in the GeForce NOW cloud gaming library.
Plus, members can get ready for a...
26/03/2025
The reliability of the electric grid is critical.
From handling demand surges and evolving power needs to preventing infrastructure failures that can cause wil...
25/03/2025
Generative AI is unlocking new capabilities for PCs and workstations, including ...
20/03/2025
The power and utilities sector keeps the lights on for the world's populatio...
20/03/2025
Time to sharpen the blade. GeForce NOW brings a legendary addition to the cloud: Ubisoft's highly anticipated Assassin's Creed Shadows is now available ...
19/03/2025
The roots of many of NVIDIA's landmark innovations - the foundational techno...
19/03/2025
AI has been shaping the media and entertainment industry for decades, from early recommendation engines to AI-driven editing and visual effects automation. Real...
19/03/2025
NVIDIA this week recognized 14 partners leading the way across the Americas for their work advancing agentic and physical AI across industries.
The 2025 Americ...
18/03/2025
The quick-service restaurant industry is a marvel of modern logistics, where spe...
18/03/2025
Global telecommunications networks can support millions of user connections per day, generating more than 3,800 terabytes of data per minute on average.
That m...
18/03/2025
The telecom industry is increasingly embracing AI to deliver seamless connections - even in conditions of poor signal strength - while maximizing sustainability...
18/03/2025
AI agents are transforming work, delivering time and cost savings by helping peo...
18/03/2025
AI deployments thrive on speed, data and scale. That's why NVIDIA is expandi...