Disaggregated inference infrastructure: Paving the way for AI's next leap

Inference is rapidly becoming a focal point in the evolution of artificial intelligence (AI). As systems become more agentic, they now boast capabilities such as multi-step reasoning, persistent memory, and comprehensive context understanding, positioning them to handle intricate tasks in fields like software development, video production, and research.

These enhanced functionalities significantly demand the underlying infrastructure, prompting a re-think of how inference can be scaled efficiently. A primary challenge in this evolving landscape includes processing immense contextual data streams vital for certain workloads. For instance, AI-enhanced software development necessitates understanding entire codebases, while video editing needs sustained coherence across millions of data points.

The introduction of the NVIDIA SMART framework redefines this domain. Emphasising a disaggregated approach, this framework optimises inference at scale. It utilises a full-stack infrastructure that ensures efficient resource allocation for computation and memory tasks. Key components of this infrastructure are tools like NVIDIA Blackwell, NVIDIA GB200 NVL72, NVFP4, and open-source software, such as NVIDIA TensorRT-LLM and NVIDIA Dynamo.

NVIDIA Rubin CPX further exemplifies this shift in AI infrastructure, offering a GPU tailored for long-context AI workloads. By enhancing the compute-intensive context phase, Rubin CPX transforms existing systems, delivering remarkable breakthroughs with its powerful specifications, such as 30 petaFLOPs of NVFP4 compute and a formidable 128 GB of GDDR7 memory.

Moreover, the innovation extends beyond a single solution. The integrated Vera Rubin NVL144 CPX racks, equipped with Rubin CPX GPUs, and using cutting-edge solutions like NVIDIA Quantum-X800 InfiniBand and Spectrum-X Ethernet, together orchestrated through the Dynamo platform, are primed to handle enormous AI contexts. Such advances ensure not only increased responsiveness but also maximise Returns on Investment (ROI) for generative AI applications.

Enterprises adopting these scalable architectures will benefit from improved efficiency and reduced costs, with estimated returns of 30x to 50x on investments. The forward-thinking design of Rubin CPX and the encompassing NVIDIA framework heralds a new era of AI capabilities, stretching the possibilities for developers and businesses alike.