Inference Engines

Framework-centric inference serving examples on AWS, organized by engine (vLLM, SGLang, NVIDIA Dynamo)

Overview

This collection provides framework-centric inference examples on AWS, organized by serving engine. Each example deploys a modern LLM or VLM serving stack — with options for high-performance EFA networking, expert-parallel all-to-all communication, and disaggregated prefill/decode KV-cache transfer — across Amazon EKS, SageMaker HyperPod EKS, and EC2.

Available engines and test cases

More engines (TRT-LLM, NIM, Ray Serve, and others) are planned.

vLLM

vLLM is a high-throughput, OpenAI-API-compatible LLM serving engine. The samples deploy vLLM on AWS for inference workloads, with options for:

• High-performance EFA networking
• Expert-parallel all-to-all (UCCL-EP)
• Disaggregated prefill/decode KV-cache transfer (NIXL)

DeepSeek-V3 disaggregated inference (dsv3-uccl-nixl)

This sample serves DeepSeek-V3-0324 (671B parameters, 256 experts, top-8 routing, MLA) with disaggregated prefill / decode across multiple p5en.48xlarge (8×H200) nodes on Amazon EKS or SageMaker HyperPod EKS, using:

• vLLM 0.21.0 for OpenAI-compatible serving
• UCCL-EP (deepep_high_throughput, deepep_low_latency) for MoE expert-parallel all-to-all over EFA
• NIXL (NVIDIA Inference Xfer Library) over LIBFABRIC for KV-cache transfer between prefill and decode pods
• A simple Python proxy (toy_proxy_server.py shipped in vLLM) for routing

It benchmarks configurations from a single 8-GPU unified node up to a 6-node 1-prefill / 5-decode disaggregated topology, plus a UCCL-EP vs. AGRS (allgather_reducescatter) backend comparison.

Architecture

• Prefill runs UCCL-EP deepep_high_throughput (high token/s under bursts)
• Decode runs UCCL-EP deepep_low_latency (low TPOT, CUDA graphs)
• NIXL transfers KV blocks from prefill → decode over EFA via the LIBFABRIC backend
• All pods run with hostNetwork: true so EFA / NIXL / NCCL can use the full 16-EFA, 3,200 Gbps fabric

Indicative results

Caveat. These numbers come from single-seed runs at --num-prompts 50. They are smoke-benchmark numbers, not statistically tight measurements. Treat the rankings as directionally informative, not as research findings. Source: the dsv3-uccl-nixl README.

Directional observations:

1. Disaggregation pays off mostly in latency, not throughput. Unified has the best tok/s/GPU; disaggregation lowers burst TPOT roughly 44ms → 30ms but only modestly increases peak RPS.
2. The 1P:4D ratio appears to be the knee — beyond that, the single prefill node looks like the bottleneck.
3. UCCL-EP outperforms AGRS for this MLA + 256-expert MoE model in these runs.

Prerequisites

• Amazon EKS or SageMaker HyperPod EKS, version 1.33+ (1.33 added native EFA traffic in the default EKS security group)
• 1–6 × p5en.48xlarge GPU nodes (8×H200 141GB, 16×EFA, ~30 TB NVMe)
• NVIDIA device plugin and EFA device plugin installed
• (Recommended) FSx for Lustre PVC for the model cache, or local NVMe
• Hugging Face token with access to deepseek-ai/DeepSeek-V3-0324
• ~680 GB of fast storage for the FP8 weights cache

Key software versions: vLLM 0.21.0, UCCL commit 0dc87eb, NIXL v1.1.0, PyTorch 2.11.0 (cu130), Ray 2.55.1, EFA installer 1.48.0, NCCL v2.30.4-1. These pins, the EKS 1.33+ EFA note, and the results table are transcribed from the dsv3-uccl-nixl README; the Cosmos Reason image tags (vllm/vllm-openai:v0.21.0 for the plain-EKS path and the AWS DLC vllm:0.17-gpu-py312 for the HyperPod path — two distinct images) are from the cosmos-reason README. The upstream repo is forward-dated; these versions reflect the source as published.

NVIDIA Cosmos Reason (cosmos-reason)

Online inference reference for NVIDIA’s Cosmos Reason physical-reasoning Vision-Language Model, served by vLLM on Amazon EKS or SageMaker HyperPod EKS. Cosmos Reason generates chain-of-thought reasoning traces about safety, causality, object interactions, and spatiotemporal dynamics in videos and images — the building block for auto-labeling Synthetic Data Generation (SDG) pipelines, Reasoning VLA models, and video Q&A workloads.

This test case provides two parallel kubectl-only deployment paths, both serving the same model with the same vLLM CLI args:

Supported models are parameterized via ${MODEL_ID}:

Both paths expose an OpenAI-compatible /v1/chat/completions endpoint and ship example clients for image VQA, video VQA, and SDG auto-labeling. Video workloads require an expanded encoder cache; the kubernetes/ path (upstream vLLM v0.21.0) includes the necessary flags by default, while the HyperPod DLC path is recommended for image workloads.

SGLang

Multiple SGLang test cases cover DeepSeek-R1 on p5/p5en with DeepEP MoE all-to-all over NVSHMEM-libfabric/EFA, along with newer single-node and node-level prefill/decode disaggregation topologies on H200 and B300 hardware — spanning Qwen3.5-27B, Kimi2.6, DeepSeek V4 Pro/Flash, and GLM-5.2.

Each sample is self-contained: most are EKS / SageMaker HyperPod manifests applied with kubectl, while dsr1-deepep-efa runs directly on EC2 with Docker. Except for GLM-5.2 (which uses a digest-pinned dev-glm52-nvfp4 image for NVFP4 support), the manifest-applied samples serve on the upstream lmsysorg/sglang:v0.5.12.post1-cu130 image; Kimi adds an EFA layer on top for inter-node RDMA. See the dedicated SGLang inference examples for the full per-sample topology, images, and shared helpers (EFA image build, model pre-staging, metrics).

NVIDIA Dynamo

The nvidia-dynamo example serves models with NVIDIA Dynamo (SGLang backend) on SageMaker HyperPod EKS, supporting both aggregated and PD-disaggregated serving via the Dynamo operator. See examples/inference/nvidia-dynamo for deployment, benchmarks, and observability.

Source

All examples on this page are maintained in examples/inference in the awsome-distributed-ai repository.