More Performance From Every Byte: How Full-Stack Codesign Solves AI's Memory Challenge

Larger models, longer contexts and agentic applications are increasing GPU memory pressure while supply tightens and prices rise - a trend expected to continue through the near term.

NVIDIA works closely with memory partners to rapidly adopt the latest memory technologies. NVIDIA was first to adopt HBM3e and HBM4 technologies into an accelerated computing stack.

But the approach isn't to simply pack more memory onto the GPU. It's full-stack codesign. NVIDIA intentionally designs GPUs, CPUs, networking, storage and software together to get more useful work out of memory at every layer of the AI factory.

That matters because an AI factory runs on a memory and storage hierarchy. GPU high-bandwidth memory (HBM) is the fastest, most expensive and most contested tier - with model weights and KV cache competing for limited capacity. Below it sit CPU memory, local SSDs and shared storage, each trading speed for more capacity at a lower cost per bit.

How much useful work a byte does depends on how well the whole hierarchy is used: shrinking what must be stored, placing it on the right tier and bringing it back to the GPU before it's needed.

Byte for byte, NVIDIA makes the most efficient use of memory at every layer - and the results show it. From NVIDIA Blackwell Ultra to the Vera Rubin platform, HBM capacity per GPU rose roughly 3%, while agentic throughput per dollar rose by up to 30x on SemiAnalysis AgentX.

Agentic Inference Is Putting More Pressure on HBM Frontier models now have trillions of parameters - the values that hold what an AI model learned in training.

KV cache, which holds the context the model needs to understand each active request without reprocessing the same information, is also expanding. Agentic trajectories carry prior context across repeated model and tool calls, in some cases processing up to 3 million tokens, so each request's working memory grows over time.

Retrieval-augmented generation (RAG) adds relevant enterprise information to prompts, creating even more context to process and retain.

During inference, HBM primarily holds both model parameters and KV cache. The result: greater pressure on HBM.

Why AI Factories Need a Memory Hierarchy Adding more HBM capacity per GPU is the obvious response to that pressure, but capacity delivers only what bandwidth can move.

Think of HBM as a warehouse and memory bandwidth as its loading docks. A larger warehouse holds more inventory, but the docks set how much can move in and out at a time. Once the docks are at capacity, moving more inventory takes longer and slows response times. If delays exceed the required service level, the added space can't be used effectively, even as it adds cost and draws power.

That's also why no operation keeps all its inventory in the closest, fastest space. What's in motion stays near the line. What's waiting moves to a larger, cheaper space nearby and returns before it's needed.

An AI factory faces the same limit.

More HBM can hold more KV cache and support more concurrent users. Every added user shares the same compute and memory bandwidth. But when each user expects fast responses, eventually memory bandwidth is unable to supply data as fast as compute can use it: a limit engineers call the memory wall.

Past that point, extra HBM capacity can't be put to work, yet it still adds costs and draws power. The data still needs a place to live, but not all of it needs the fastest, most expensive tier. This is why an AI factory manages data across a memory and storage hierarchy.

Maximizing the Value of Every Byte With Full-Stack Codesign NVIDIA's full-stack codesign makes every byte of HBM count by relieving pressure from model weights, KV cache and RAG data. It does this with techniques that each target a different source of inefficiency:

Shrink model weights and distribute them across the NVL72 domain.

The NVFP4 format reduces the space required for weights compared to FP8, the lowest precision possible on the NVIDIA Hopper platform. Recent models, including FLUX.2 [klein] from Black Forest Labs, Inkling from Thinking Machines Lab, Beam 1.0 from Reflection, Meta Muse Spark from Meta, Step 5 Preview from StepFun and NVIDIA Nemotron 3.5 Super VL have been released with NVFP4 checkpoints, demonstrating growing model-builder adoption of 4-bit precision for faster, more memory-efficient inference.

Within an NVL72 system, model weights can be distributed across up to 72 GPUs, so each GPU holds a smaller share of the model - leaving more HBM for KV cache and concurrent users.

Shrink KV-cache and offload inactive context to lower-cost storage tiers.

Applying NVFP4 to the KV cache reduces its footprint compared with FP8, so more context can stay resident in HBM.

NVIDIA Vera CPU helps in two ways. Faster tool execution shortens the time KV cache sits inactive during tool calls, and its large LPDDR memory, connected to NVIDIA Rubin GPUs over NVIDIA NVLink-C2C interconnect, gives paused context a nearby place to wait and return quickly.

When context isn't actively in use, NVIDIA Dynamo inference software moves it to NVIDIA CMX context memory storage - a tier built for capacity at far lower cost per bit than HBM - and returns it before reuse. Unlike local SSDs, CMX is shared across servers, and unlike enterprise storage, it skips the overhead of long-term data protection that short-lived context doesn't need. NVIDIA CMX context memory storage partners include Cloudian, DDN, Dell Technologies, Everpure (previously Pure Storage), HPE, IBM, NetApp, VAST and WEKA.

Dynamo's KV Cache Reuse ties the tiers together, managing KV cache as one system-wide resource across GPU memory, CPU memory and storage, so context computed once can be reused across the system.

Compact RAG embeddings and keeping the index outside HBM.

RAG tur

LINK:	https://blogs.nvidia.com/blog/memory-full-stack-codesign/...
	See more stories from nvidia

Most recent headlines