SGLang Inference

SGLang LLM serving deployments on AWS EKS, SageMaker HyperPod, and EC2 — including prefill/decode disaggregation, MoE expert parallelism, and router-based scaling

Overview

SGLang deployments on AWS. Each sub-directory is a self-contained sample — most are EKS / SageMaker HyperPod manifests applied with kubectl; the dsr1-deepep-efa sample runs directly on EC2 with Docker.

Test cases

Among the kubectl-applied EKS/HyperPod samples, all except GLM-5.2 serve on the same upstream lmsysorg/sglang:v0.5.12.post1-cu130 image (Kimi adds only an EFA layer on top for inter-node RDMA); GLM-5.2 uses the dev-glm52-nvfp4 image (pinned by digest, since dev-* tags are mutable) until NVFP4 support for that model lands in a tagged release. The EC2/Docker sample dsr1-deepep-efa is separate — it uses its own custom build (SGLang 0.5.13.post1 + DeepEP).

Prefill/decode disaggregation shapes

The intra-node PD samples deliberately run several engine processes in one pod — not the usual one-process-per-pod shape. Intra-node KV transfer rides NVLink via CUDA IPC, which requires all engines to share an IPC namespace and see each other’s GPUs — impossible across separate pods or containers, so the engines share one container.

• Intra-node PD (qwen3.5-27b-b300-intra-pd, dsv4flash-b300-intra-3p1d) — all engines in one pod on one B300 node, KV cache moved prefill → decode over NIXL, staying intra-node over NVLink (UCX; the EFA-detected warning is expected/benign).
• Node-level 1P1D (kimi2.6-h200-1p1d) — separate prefill and decode StatefulSets on two ml.p5en.48xlarge (8×H200) nodes, KV cache transferred over NIXL (LIBFABRIC over EFA), fronted by the SGLang router.
• Unified baseline (dsv4pro-b300-single-node) — one engine spans all 8 GPUs (tp=8, dp=8, --enable-dp-attention, MXFP4 MoE, EAGLE speculative decoding).
• Router-fronted replicas (glm5.2-b300-tp2-dp4) — four independent tp=2 engines behind an SGLang router load-balancing with a cache-aware policy, for elastic scaling and failure isolation.

Shared helpers

Reusable across all the samples:

EFA-enabled image (multi-node only)

Multi-node samples (e.g. kimi2.6-h200-1p1d) move the KV cache between nodes over EFA, which the stock lmsysorg/sglang image can’t do on its own. Dockerfile.efa layers the AWS EFA installer onto the same upstream image; build-image.sh builds it and pushes it to ECR, printing the image URI on the last line:

./build-image.sh   # -> <account>.dkr.ecr.<region>.amazonaws.com/sgl-dev-cu13:<tag>

Set that URI as <YOUR_ECR_IMAGE> in the sample’s manifest. Single-node samples run the upstream image directly and don’t need this build.

Pre-stage model weights

download-model.sh renders and applies a DaemonSet that downloads a Hugging Face repo to every matching node’s local NVMe (in HF cache layout under <nvme>/huggingface), so serving pods read weights from fast local disk instead of pulling them at startup:

./download-model.sh moonshotai/Kimi-K2.6        ml.p5en.48xlarge
./download-model.sh deepseek-ai/DeepSeek-V4-Pro ml.p6-b300.48xlarge
# watch: kubectl logs -f -l app=model-downloader
# then:  kubectl delete daemonset model-downloader

The NVMe hostPath defaults to HyperPod’s /opt/dlami/nvme; on self-managed EKS change it to /mnt/k8s-disks/0 so it stages to the disk the serving pods mount.

Metrics

Every serving pod exposes SGLang metrics on :30000/metrics (engines start with --enable-metrics) and carries the sglang-metrics=true label plus prometheus.io/{scrape,port,path} annotations. Point any Prometheus-compatible scraper at those pods — e.g. an in-cluster Prometheus agent remote-writing to Amazon Managed Prometheus (AMP) with Amazon Managed Grafana reading from AMP.

For the kernel-level DeepEP-on-EFA dispatch/combine benchmarks, see micro-benchmarks/expert-parallelism/deepep-benchmark. For a PD-disaggregated MoE deployment on vLLM (UCCL-EP, KV over NIXL, on EKS), see examples/inference/vllm/dsv3-uccl-nixl.

Source

The test-case table (models, hardware, topologies, image tags including SGLang 0.5.13.post1 and lmsysorg/sglang:v0.5.12.post1-cu130), the ml.p5en.48xlarge instance type for the Kimi 1P1D sample, the shared helpers (build-image.sh, download-model.sh), and the metrics annotations are transcribed verbatim from the upstream examples/inference/sglang/README.md. The upstream repo is forward-dated; the model/version names reflect the source as published.