Build a Validated Kubernetes Runtime for NVIDIA Dynamo

PUBLISHER ARTICLE PREVIEW

From the original article

Teams building AI applications need a repeatable path from a model configuration to a working inference endpoint. NVIDIA Dynamo is a distributed inference framework that coordinates model serving across GPU workers. Deploying it on Kubernetes is only a first step. A successful model deployment requires more: GPU drivers, resource allocation, scheduling, networking, and monitoring must all work together.

Resolving these dependencies takes time away from testing model behavior, serving performance, and application integration.

The NVIDIA DSX AI Factory Platform brings together technologies to help the ecosystem design, build, and operate more efficient AI factories, and Dynamo is one of them. To make management of those infrastructure dependencies easier, the platform contains NVIDIA AI Cluster Runtime (AICR). It packages NVIDIA Dynamo and its supporting infrastructure into validated recipes. Each recipe defines a configuration for a specific environment and workload, including pinned component versions, compatibility constraints, and validation checks.

AICR gives platform teams a repeatable way to deploy and check the runtime, providing developers with a consistent foundation for serving models and testing applications.

This walkthrough covers the infrastructure that supports Dynamo on Kubernetes, compares two routing options, and shows how to deploy and validate the runtime with AICR.

Running a model with Dynamo on Kubernetes starts with a DynamoGraphDeployment that specifies how vLLM, SGLang, or NVIDIA TensorRT LLM should run in pods on the cluster. But the graph is only the top of the stack. A production inference cluster also needs a runtime layer below it.

An AICR Dynamo recipe packages both Dynamo and the cluster runtime components it depends on.

AICR ships the dynamo-platform chart itself (pinned to version 1.4.2) that includes the Dynamo operator, Dynamo runtime images, and, among others, the nvidia.com/v1beta1 DynamoGraphDeployment CustomResourceDefinition (CRD). v1beta1 replaces the former v1alpha1 API version.

In addition, the recipe pins component versions and records configuration values, node selectors, tolerations, checksums, and validation expectations to make deployment reproducible.

With Dynamo 1.4, frontends, workers, optional Gateway API Inference Extension (GAIE) / Endpoint Picker Plugin (EPP) components, and other Dynamo services discover each other through Kubernetes resources. AICR’s default path therefore no longer uses NATS, and the dynamo-platform chart no longer installs it by default. NATS remains available as an opt-in for deployments that need it.

Agentic workloads repeatedly extend long token prefixes, including system prompts, tool definitions, conversation history, and tool results. If a turn reaches a worker without that prefix in GPU memory, the system reprocesses intermediate tokens instead of reusing cached results. Dynamo routes requests to workers that already hold the relevant KV blocks, either through its frontend router or through GAIE/EPP integration at the Gateway API layer.

Dynamo can also separate prompt processing (prefill) from token generation (decode), an architecture known as disaggregated serving. On KV-cache misses, all or part of the prompt must be reprocessed by the prefill pool. Decode workers then take over token-by-token generation, which has different GPU, memory, latency, and batching characteristics. The cache miss rate depends on the workload.

Long-running agentic sessions behave differently from one-off chat requests. Disaggregated serving enables operators to scale and tune prefill and decode capacity independently for a given request mix.