NVIDIA Dynamo
Distributed LLM inference with KV cache-aware routing and disaggregated prefill/decode on HyperPod EKS
View on GitHubOverview
NVIDIA Dynamo is a distributed inference framework providing KV cache-aware routing, disaggregated prefill/decode, and NIXL-based KV transfer for LLM serving on Kubernetes. This test case serves models on SageMaker HyperPod EKS with the SGLang backend.
Scenarios
| Scenario | Model | Pattern | GPUs |
|---|---|---|---|
gpt-oss-agg | GPT-OSS-20B | Aggregated (1 worker = prefill+decode) | 1 |
gpt-oss-disagg | GPT-OSS-20B | Disaggregated (prefill + decode, NIXL) | 2 |
qwen3.6-agg | Qwen3.6-27B-FP8 | Aggregated | 1 |
qwen3.6-disagg | Qwen3.6-27B-FP8 | Disaggregated (prefill + decode, NIXL) | 2 |
Key Features
- KV cache-aware routing — Routes requests to workers with cached prefixes
- Disaggregated prefill/decode — Separate workers for compute-heavy prefill and latency-sensitive decode
- NIXL-based KV transfer — Efficient GPU-to-GPU KV cache migration
- DynamoGraphDeployment (DGD) CRD — Kubernetes-native operator