Editor's note: This post is part of Think SMART, a series focused on how leading AI service providers, developers and enterprises can boost their inference performance and return on investment with the latest advancements from NVIDIA's full-stack inference platform.
AI models are becoming increasingly complex and collaborative through multi-agent workflows. To keep up, AI inference must now scale across entire clusters to serve millions of concurrent users and deliver faster responses.
Much like it did for large-scale AI training, Kubernetes - the industry standard for containerized application management - is well-positioned to manage the multi-node inference needed to support advanced models.
The NVIDIA Dynamo platform works together with Kubernetes to streamline the management of both single- and multi-node AI inference. Read on to learn how the shift to multi-node inference is driving performance, as well as how cloud platforms are putting these technologies to work.
Tapping Disaggregated Inference for Optimized Performance For AI models that fit on a single GPU or server, developers often run many identical replicas of the model in parallel across multiple nodes to deliver high throughput. In a recent paper, Russ Fellows, principal analyst at Signal65, showed that this approach achieved an industry-first record aggregate throughput of 1.1 million tokens per second with 72 NVIDIA Blackwell Ultra GPUs.
When scaling AI models to serve many concurrent users in real time, or when managing demanding workloads with long input sequences, using a technique called disaggregated serving unlocks further performance and efficiency gains.
Serving AI models involves two phases: processing the input prompt (prefill) and generating the output (decode). Traditionally, both phases run on the same GPUs, which can create inefficiencies and resource bottlenecks.
Disaggregated serving solves this by intelligently assigning these tasks to independently optimized GPUs. This approach ensures that each part of the workload runs with the optimization techniques best suited for it, maximizing overall performance. For today's large AI reasoning models, such as DeepSeek-R1, disaggregated serving is essential.
NVIDIA Dynamo seamlessly brings multi-node inference optimization features such as disaggregated serving to production scale across GPU clusters.
It's already delivering value.
Baseten, for example, used NVIDIA Dynamo to speed up inference serving for long-context code generation by 2x and increase throughput by 1.6x, all without incremental hardware costs. Such software-driven performance boosts enable AI providers to significantly reduce the costs to manufacture intelligence.
In addition, recent SemiAnalysis InferenceMAX benchmarks demonstrated that disaggregated serving with Dynamo on NVIDIA GB200 NVL72 systems delivers the lowest cost per million tokens for mixture-of-experts reasoning models like DeepSeek-R1, among platforms tested.
Scaling Disaggregated Inference in the Cloud As disaggregated serving scales across dozens or even hundreds of nodes for enterprise-scale AI deployments, Kubernetes provides the critical orchestration layer. With NVIDIA Dynamo now integrated into managed Kubernetes services from all major cloud providers, customers can scale multi-node inference across NVIDIA Blackwell systems, including GB200 and GB300 NVL72, with the performance, flexibility and reliability that enterprise AI deployments demand.
Amazon Web Services is accelerating generative AI inference for its customers with NVIDIA Dynamo and integrated with Amazon EKS.
Google Cloud is providing a NVIDIA Dynamo recipe to optimize large language model (LLM) inference at enterprise scale on its AI Hypercomputer.
OCI is enabling multi-node LLM inferencing with OCI Superclusters and NVIDIA Dynamo.
The push towards enabling large-scale, multi-node inference extends beyond hyperscalers.
Nebius, for example, is designing its cloud to serve inference workloads at scale, built on NVIDIA accelerated computing infrastructure and working with NVIDIA Dynamo as an ecosystem partner.
Simplifying Inference on Kubernetes With NVIDIA Grove in NVIDIA Dynamo Disaggregated AI inference requires coordinating a team of specialized components - prefill, decode, routing and more - each with different needs. The challenge for Kubernetes is no longer about running more parallel copies of a model, but rather about masterfully conducting these distinct components as one cohesive, high-performance system.
NVIDIA Grove, an application programming interface now available within NVIDIA Dynamo, allows users to provide a single, high-level specification that describes their entire inference system.
For example, in that single specification, a user could simply declare their requirements: I need three GPU nodes for prefill and six GPU nodes for decode, and I require all nodes for a single model replica to be placed on the same high-speed interconnect for the quickest possible response.
From that specification, Grove automatically handles all the intricate coordination: scaling related components together while maintaining correct ratios and dependencies, starting them in the right order and placing them strategically across the cluster for fast, efficient communication. Learn more about how to get started with NVIDIA Grove in this technical deep dive.
As AI inference becomes increasingly distributed, the combination of Kubernetes and NVIDIA Dynamo with NVIDIA Grove simplifies how developers build and scale intelligent applications.
Explore how these technologies come together to make cluster-scale AI easy and production-ready by joining NVIDIA at KubeCon, running through Thursday, Nov. 13, in Atlanta.
Most recent headlines
05/01/2027
Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...
01/06/2026
January 6 2026, 05:30 (PST) Dolby Sets the New Standard for Premium Entertainment at CES 2026
Throughout the week, Dolby brings to life the latest innovatio...
02/05/2026
Dalet, a leading technology and service provider for media-rich organizations, t...
01/05/2026
January 5 2026, 18:30 (PST) NBCUniversal's Peacock to Be First Streamer to ...
01/04/2026
January 4 2026, 18:00 (PST) DOLBY AND DOUYIN EMPOWER THE NEXT GENERATON OF CREATORS WITH DOLBY VISION
Douyin Users Can Now Create And Share Videos With Stun...
02/03/2026
NBC Sports selected Imaginary Forces to create the main title sequence for the M...
02/03/2026
The NFL announces the launch of the NFL Draft Innovation Challenge, a new crowds...
02/03/2026
The World Teleport Association (WTA) has named Skyline Communications' DataMiner SatOps solution as a finalist for the 2026 Teleport Technology of the Year ...
02/03/2026
Ventum Tech announces a strategic partnership with Pixotope, a real-time graphic...
02/03/2026
Victory , a free, ad-supported sports streaming service, announces the launch of...
02/03/2026
Global powerhouse Delta Goodrem to represent Australia at this year's Eurovi...
02/03/2026
SBS brings together prominent voices to discuss rising social division and seek ...
02/03/2026
NITV Unveils Wednesday Night Sport Night
Media releases
A Midweek Hub for Blak Sport
Flagship NRL program Over The Black Dot moves to Wednesdays at 9:30pm
...
02/03/2026
Rohde & Schwarz demonstrates FR1-FR3 carrier aggregation, advancing 6G readiness Rohde & Schwarz and Qualcomm Technologies, Inc. have reached another pivotal ...
02/03/2026
MELBOURNE, Fla., March 2, 2026 - L3Harris Technologies (NYSE: LHX) today announc...
02/03/2026
Harvey Norman tops New Zealand's biggest ad spenders as retail leads and telcos and beverages sure Auckland March 3, 2026 - Nielsen's New Zealand'...
02/03/2026
aconnic AG (ISIN: DE000A0LBKW6), Munich, and Arqit Quantum Inc. (Nasdaq: ARQQ, A...
02/03/2026
Advanced Systems Group, LLC (ASG), a technology and services provider for media creatives and content owners, has appointed Jody Boatwright as Chief Strategy Of...
02/03/2026
Lightware, an industry leader in signal management, is reinforcing its commitment to simplifying complex AV environments with the continued evolution of its TPN...
02/03/2026
ZEISS Cinema opens its doors to an exclusive screening, filmmakers Q&A and hands-on showcase of the new ZEISS Aatma lens family on Tuesday, March 10, 6:00-9:00 ...
02/03/2026
To expand the creative potential of the compact Astera QuikBeam, DoPchoice introduces the SNAPBAG Round. The new light-shaping accessory transforms the powerfu...
02/03/2026
Douglas Dubler
On a rooftop in New York City, veteran photographer Douglas Dubler watched the sun's angle shift across the urban landscape, his internal cl...
02/03/2026
ZEISS is expanding its Otus ML lens family with the introduction of the new ZEISS Otus ML 1.4/35. This manual-focus lens is designed for photographers who live ...
02/03/2026
Matthews Studio Equipment now offers the new Dinkum Systems White FlexiMount and Phone Mount Kit, a compact, highly adaptable setup for mobile content creation...
02/03/2026
Luxembourg, February 26, 2026 - SES and Africa Mobile Network (AMN) have expande...
02/03/2026
02 03 2026 - Media release Ausfilm and Screen Australia launch joint UK market initiative
Ausfilm and Screen Australia have announced Partner with Australia (...
02/03/2026
Monday 2 March 2026
Saturday Night Live UK announces writing team
L-R: Jonno Johnson; Charlie Skelton; Celya AB; Omar Badawy; Gr inne Maguire; Laura Claxton; ...
02/03/2026
Back to All News
Netflix Unveils the Trailer for 53 Sundays, the New Film by Cesc Gay
Entertainment
02 March 2026
GlobalSpain
Link copied to clipboard
Dis...
28/02/2026
With two features seen in Formula 1 coverage, the broadcaster aims to bring view...
28/02/2026
Secretary of War Pete Hegseth addresses a crowd of approximately 1,500 L3Harris employees in Camden, Arkansas, as part of his Arsenal of Freedom tour....
28/02/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
28/02/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
28/02/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
28/02/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
28/02/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
28/02/2026
Berklee Presents Mambo Mania: Eguie Castrillo and the Berklee All-Stars Big Band...
28/02/2026
Berklee Announces Two New Summer Programs in Los Angeles The Berklee Music Business Program and Electronic Music Production and Sound Design Workshop bring imme...
28/02/2026
AI-RAN is moving from lab to field, showing that a software-defined approach is ...
28/02/2026
Autonomous networks - intelligent, self-managing telecommunications operations -...
28/02/2026
Back to All News
Final Trailer for BEASTARS Final Season Part 2' Roars Tow...
28/02/2026
New way to intentionally discover molecular glues could expand drug discovery Scripps Research scientists and colleagues show how drugs that eliminate certain d...
27/02/2026
The E.W. Scripps Company names Oliver Gray as Vice President, Network Sports and...
27/02/2026
The Gotham Sports App, the exclusive direct-to-consumer streaming home of MSG Networks and the YES Network, is now available for purchase through Prime Video fo...
27/02/2026
ESPN and the Horizon League announce a new multi-year, multi-platform media rights agreement, continuing a 38-year collaboration that began with the 1988 Midwes...
27/02/2026
At the 2026 NAB Show in Las Vegas, NETGEAR will highlight its new switch models and major updates to its Engage Controller software. The company's network d...
27/02/2026
Riedel Communications announces that Fondazione Teatro alla Scala has deployed a...
27/02/2026
Lyuno specializes in media localization, including translation, dubbing, subtitling, and voice-over services for a wide array of entertainment content. The comp...
27/02/2026
Chyron Weather 2.3, the latest edition of Chyron's weather visualization suite for broadcasters and meteorologists, recently launched.
The release includes...
27/02/2026
Telestream, which concentrates in media workflow technologies, announces expanded practical AI enhancements across its Vantage, Vantage Cloud, EDC, Stanza, and ...
Editor's note: This post is part of Think SMART, a series focused on how leading AI service providers, developers and enterprises can boost their inference performance and return on investment with the latest advancements from NVIDIA's full-stack inference platform.
