Editor's note: This post is part of Think SMART, a series focused on how leading AI service providers, developers and enterprises can boost their inference performance and return on investment with the latest advancements from NVIDIA's full-stack inference platform.
AI models are becoming increasingly complex and collaborative through multi-agent workflows. To keep up, AI inference must now scale across entire clusters to serve millions of concurrent users and deliver faster responses.
Much like it did for large-scale AI training, Kubernetes - the industry standard for containerized application management - is well-positioned to manage the multi-node inference needed to support advanced models.
The NVIDIA Dynamo platform works together with Kubernetes to streamline the management of both single- and multi-node AI inference. Read on to learn how the shift to multi-node inference is driving performance, as well as how cloud platforms are putting these technologies to work.
Tapping Disaggregated Inference for Optimized Performance For AI models that fit on a single GPU or server, developers often run many identical replicas of the model in parallel across multiple nodes to deliver high throughput. In a recent paper, Russ Fellows, principal analyst at Signal65, showed that this approach achieved an industry-first record aggregate throughput of 1.1 million tokens per second with 72 NVIDIA Blackwell Ultra GPUs.
When scaling AI models to serve many concurrent users in real time, or when managing demanding workloads with long input sequences, using a technique called disaggregated serving unlocks further performance and efficiency gains.
Serving AI models involves two phases: processing the input prompt (prefill) and generating the output (decode). Traditionally, both phases run on the same GPUs, which can create inefficiencies and resource bottlenecks.
Disaggregated serving solves this by intelligently assigning these tasks to independently optimized GPUs. This approach ensures that each part of the workload runs with the optimization techniques best suited for it, maximizing overall performance. For today's large AI reasoning models, such as DeepSeek-R1, disaggregated serving is essential.
NVIDIA Dynamo seamlessly brings multi-node inference optimization features such as disaggregated serving to production scale across GPU clusters.
It's already delivering value.
Baseten, for example, used NVIDIA Dynamo to speed up inference serving for long-context code generation by 2x and increase throughput by 1.6x, all without incremental hardware costs. Such software-driven performance boosts enable AI providers to significantly reduce the costs to manufacture intelligence.
In addition, recent SemiAnalysis InferenceMAX benchmarks demonstrated that disaggregated serving with Dynamo on NVIDIA GB200 NVL72 systems delivers the lowest cost per million tokens for mixture-of-experts reasoning models like DeepSeek-R1, among platforms tested.
Scaling Disaggregated Inference in the Cloud As disaggregated serving scales across dozens or even hundreds of nodes for enterprise-scale AI deployments, Kubernetes provides the critical orchestration layer. With NVIDIA Dynamo now integrated into managed Kubernetes services from all major cloud providers, customers can scale multi-node inference across NVIDIA Blackwell systems, including GB200 and GB300 NVL72, with the performance, flexibility and reliability that enterprise AI deployments demand.
Amazon Web Services is accelerating generative AI inference for its customers with NVIDIA Dynamo and integrated with Amazon EKS.
Google Cloud is providing a NVIDIA Dynamo recipe to optimize large language model (LLM) inference at enterprise scale on its AI Hypercomputer.
OCI is enabling multi-node LLM inferencing with OCI Superclusters and NVIDIA Dynamo.
The push towards enabling large-scale, multi-node inference extends beyond hyperscalers.
Nebius, for example, is designing its cloud to serve inference workloads at scale, built on NVIDIA accelerated computing infrastructure and working with NVIDIA Dynamo as an ecosystem partner.
Simplifying Inference on Kubernetes With NVIDIA Grove in NVIDIA Dynamo Disaggregated AI inference requires coordinating a team of specialized components - prefill, decode, routing and more - each with different needs. The challenge for Kubernetes is no longer about running more parallel copies of a model, but rather about masterfully conducting these distinct components as one cohesive, high-performance system.
NVIDIA Grove, an application programming interface now available within NVIDIA Dynamo, allows users to provide a single, high-level specification that describes their entire inference system.
For example, in that single specification, a user could simply declare their requirements: I need three GPU nodes for prefill and six GPU nodes for decode, and I require all nodes for a single model replica to be placed on the same high-speed interconnect for the quickest possible response.
From that specification, Grove automatically handles all the intricate coordination: scaling related components together while maintaining correct ratios and dependencies, starting them in the right order and placing them strategically across the cluster for fast, efficient communication. Learn more about how to get started with NVIDIA Grove in this technical deep dive.
As AI inference becomes increasingly distributed, the combination of Kubernetes and NVIDIA Dynamo with NVIDIA Grove simplifies how developers build and scale intelligent applications.
Explore how these technologies come together to make cluster-scale AI easy and production-ready by joining NVIDIA at KubeCon, running through Thursday, Nov. 13, in Atlanta.
Most recent headlines
05/01/2027
Worlds first 802.15.4ab-UWB chip verified by Calterah and Rohde & Schwarz to be ...
06/09/2026
June 9 2026, 23:00 (PDT) Dolby and MagentaTV Bring Fans Closer to the FIFA Worl...
04/08/2026
Dalet, a leading technology and service provider for media-rich organizations, t...
04/07/2026
April 7 2026, 19:00 (PDT) Detective Conan: Fallen Angel of the Highway Opens in...
15/06/2026
Detach from Direct-Attached: How Remote Editing with EVO Keeps Creative Teams Mo...
14/06/2026
Library captures 1960s R&B/pop drum sound
Following on from their recent wave of plug-in effects, Iconic Instruments have just launched an all-new virtual d...
14/06/2026
HBO Comedy Rooster Shot with URSA Cine 17K 65
Brie Clayton June 14, 2026
0 Comments
Large format brings viewers intimately close to characters.
Black...
13/06/2026
Latest expansion pack includes 252 presets
Devious Machines have recently introduced another expansion for their powerful multi-effects plug-in, Infiltrator...
13/06/2026
Create custom DAW/plug-in controllers using prompts
MetaGrid have recently introduced an all-new AI Builder function to their touchscreen-based control surf...
13/06/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
13/06/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
12/06/2026
YES Network and The Gotham Sports App will air seven Athletes Unlimited Softball...
12/06/2026
The United Football League will host its FAST Innovation Suite at the 2026 United Bowl presented by Credit One Bank on Saturday, June 13 at 3:00 p.m. ET at Audi...
12/06/2026
PTZOptics and LayerJot will present live demonstrations at InfoComm 2026 showing how natural-language AI prompting, robotic camera control, and on-device comput...
12/06/2026
MultiDyne Video and Fiber Optic Systems will exhibit at InfoComm 2026, featuring...
12/06/2026
Ateme has announced that Eurovision Services is using Ateme's software-based frame-rate conversion technology for international live event workflows. The de...
12/06/2026
Bitmovin and Simplestream have announced a partnership with Xperi to simplify the launch of OTT streaming services on TiVo OS smart TVs and devices. The collabo...
12/06/2026
Net Insight has announced that a multinational technology company is deploying a...
12/06/2026
MLB Players Inc., the business arm of the MLB Players Association, has announced a partnership with Athletes First to develop and sell brand partnerships across...
12/06/2026
Guntermann and Drunck (G&D) and VuWall have announced the CommandKeyboard-Advanc...
12/06/2026
Comcast Smart Solutions announces a new smart technology deployment with Major L...
12/06/2026
Elevation Worship completed the initial leg of its Elevation Nights 2026 tour ...
12/06/2026
AJA Video Systems has announced KONA IP25 support for Colorfront Transkoder and ...
12/06/2026
Audinate Group Limited (ASX: AD8) will exhibit at InfoComm 2026 (Booth C7321, Ce...
12/06/2026
Pac-12 Commissioner Teresa Gould has announced the appointment of Scott Adametz as Chief Technology Officer. The Pac-12 describes the hire as the first CTO appo...
12/06/2026
Grass Valley has announced AMPP Edge Live, a production system combining Grass Valley hardware, NVIDIA Blackwell GPU acceleration, and AMPP OS in a single platf...
12/06/2026
At one time a trailblazer with the launch of the Longhorn Network, the Universit...
12/06/2026
Ratings Roundup is a rundown of recent rating news and is derived from press rel...
12/06/2026
Chyron has announced PAINT 10.4, an update to its illustrated replay and sports ...
12/06/2026
SVP, Production, Mark Gross: With the new schedule, with not having every Sunday night, it has given us an opportunity to take a step back and reimagine what o...
12/06/2026
For Televisa Technical Engineering Manager Roberto N nez Ibarra and the small team of 12 technicians and two production personnel at the IBC things are already ...
12/06/2026
Simple Steps to Better Acoustics - Taming The Small Room
Most of us mix in spare rooms and small spaces, where the acoustics fight us at every turn. At Gear...
12/06/2026
Latest addition expands vintage-inspired effects palette
Meris' Ottobit pedal range draws its inspiration from vintage gaming consoles, and the latest a...
12/06/2026
Soundbox-based chamber strings series expanded
Sonora Cinematic have just announced the launch of the second instalment in their Soundbox-based chamber stri...
12/06/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
12/06/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
12/06/2026
AJA Announces KONA IP25 Integration with Colorfront Software
Brie Clayton June 12, 2026
0 Comments
Collaboration enables uncompressed SMPTE ST 2110 I/O ...
12/06/2026
URSA Cine 12K LF Used to Create Visuals for STUTS' K-Arena Concert
Brie Clayton June 12, 2026
0 Comments
Organic visuals projected on a giant scre...
12/06/2026
MTI FILM Acquires Mango New Edit, Expanding its Global Post-Production Services ...
12/06/2026
AI Point Tracking Speeds Up Complex VFX Tracks in Mocha Pro
Jessie Electa Petrov June 12, 2026
0 Comments
The 2026.5 release adds automatic point trac...
12/06/2026
Bitmovin, a provider of video streaming solutions, has partnered with Simplestream, a provider of OTT and broadcast solutions, and technology provider Xperi, to...
12/06/2026
Leostream Corporation, creator of the world-leading Leostream Remote Desktop Access Platform, today announced Jigsaw24, a leading B2B IT solutions provider wit...
12/06/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
12/06/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
12/06/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
12/06/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
12/06/2026
Share
Copy link
Facebook
X
Linkedin
Bluesky
Email...
12/06/2026
How Aussie indie games and screen are levelling up with IP 11 June 2026
Ari Harrison, Pro Jank Footy
Head of Games Joey Egger and Ari Harrison of Umbrella sha...
12/06/2026
AgentPerf from Artificial Analysis, the industry's first agentic AI benchmark, gives developers, enterprises and infrastructure providers a clear way to com...
Editor's note: This post is part of Think SMART, a series focused on how leading AI service providers, developers and enterprises can boost their inference performance and return on investment with the latest advancements from NVIDIA's full-stack inference platform.
