miles

GRPO reinforcement learning post-training with miles on Amazon SageMaker HyperPod EKS

Overview

This test case runs GRPO post-training with miles on Amazon SageMaker HyperPod with Amazon EKS orchestration, or on a plain Amazon EKS cluster. It follows the same shape as the sibling examples/training/slime/ test case: build a container image, prepare data, deploy a multi-node Ray cluster, convert model checkpoints, and launch GRPO training on NVIDIA GPUs interconnected with Elastic Fabric Adapter (EFA) networking.

The miles container image bundles a matched PyTorch / CUDA stack, the sglang-miles fork, and the radixark Megatron-LM fork. This test case’s Dockerfile takes that image as-is and adds only the AWS EFA stack on top; see miles.Dockerfile, which is the source of truth for the exact base image, its pinned digest, and the CUDA/PyTorch versions it provides. The Dockerfile also documents which NVIDIA GPU architectures the base image’s CUDA stack targets; consult it for the authoritative values, and see Hardware requirements for what this test case is actually exercised on.

Introduction

miles is a fork of SLIME, an LLM post-training framework for RL scaling. It keeps SLIME’s design of two backends under Ray orchestration:

• SGLang for high-throughput rollout generation, providing RadixAttention, continuous batching, and tensor parallelism.
• Megatron-LM (a radixark fork) for scalable distributed training with TP, PP, CP, EP, and ZeRO-style sharding.

This page is classified by its training backend. miles is a dual-backend RL post-training framework: SGLang serves the rollout/generation side, while the radixark Megatron-LM fork performs the weight updates. Because Megatron-LM is the backend that actually trains the model, this entry carries framework: megatron — one of the collection schema’s allowed framework enum values (pytorch, megatron, jax, neuron). The dual-backend design is documented here so the label is read as “the training backend,” not the whole story. The coverage agent resolves entries by their githubPath frontmatter field regardless of which Astro collection the file lives in (scripts/orchestration/coverage.mjs line 115: byPath.has(item.githubPath)). This is the same pattern used by many existing frameworks/ entries with examples/training/ paths: ddp, deepspeed, fsdp, jax, megatron-bridge, nemo-rl, and nemo all live under src/content/frameworks/ with githubPath: examples/training/... and are correctly counted as covered by the daily coverage run.

Ray manages resource orchestration and supports two deployment topologies:

• Colocated mode (COLOCATE=true) — training actors and rollout engines share one GPU pool.
• Disaggregated mode (COLOCATE=false) — training actors and rollout engines run on separate GPU pools and synchronize weights over NCCL/EFA.

A third, independent choice moves reward scoring to a lightweight CPU-only service (a separate python:3.12-slim reward-service image) and can be combined with either GPU topology.

How miles differs from slime

miles keeps a structure close to the sibling slime case, so both can be used the same way; it is not a recommendation of one over the other. Key differences:

The train.py CLI and GRPO flags are compatible with SLIME’s, so the recipes here are drop-in.

Note on version pins: the exact base-image tag and digest, the EFA installer version, and the NCCL / aws-ofi-nccl reference values live in miles.Dockerfile. Because radixark publishes mutable snapshot tags (and has deleted a tag this file previously pinned), the base is pinned by digest in the Dockerfile rather than by a tag alone. Consult the Dockerfile for the authoritative, current values.

Architecture

The reference layout runs on 2x p5en.48xlarge nodes with EKS orchestration (HyperPod-compatible):

+---------------------------------------------------------------------+
|             Amazon EKS Cluster, HyperPod-compatible                 |
|                 2x p5en.48xlarge, EKS orchestration                 |
|                                                                     |
|  +----------------------------+  +----------------------------+     |
|  |  Node 1: p5en.48xlarge     |  |  Node 2: p5en.48xlarge     |     |
|  |  8x H200 GPU     EFA       |  |  8x H200 GPU     EFA       |     |
|  +----------------------------+  +----------------------------+     |
|            |                                  |                     |
|            +---------- EFA, GPU<->GPU RDMA ----------+              |
|                                                                     |
|  +---------------------------------------------------------+        |
|  |          FSx for Lustre, RWX, mounted at /fsx           |        |
|  +---------------------------------------------------------+        |
|                                                                     |
|  Kubernetes Resources: KubeRay operator, EFA device plugin,         |
|  FSx CSI driver, NVIDIA device plugin                               |
+---------------------------------------------------------------------+

A single node is also supported for the dense 4B case; the multi-node layout adds the second node for MoE and disaggregated runs. For exact per-instance RAM, vCPU, GPU memory, and EFA device/bandwidth figures, refer to the AWS EC2 P5 instance documentation.

Hardware requirements

This test case is exercised on the following configuration:

The recipes here are validated on p5en.48xlarge (H200). The miles base image’s CUDA stack also targets a Blackwell compute capability in addition to H200 (see the target architectures documented in miles.Dockerfile), but no Blackwell instance type has been verified end-to-end in this test case.

Quick Start

The default path is Qwen3-4B dense, colocated on one node: build the image, prepare the model and data, deploy a Ray cluster, and run a short GRPO loop.

cd examples/training/miles
cp env_vars.colocated.example env_vars
# Edit env_vars for your cluster (this file defines FULL_IMAGE and the
# COLOCATE / node / GPU topology variables the recipes read), then:
source env_vars

FULL_IMAGE is your target image reference (registry, repository, and tag) and is defined in env_vars — set it there before building. Build and push the image (the radixark/miles base, pinned by digest, plus the AWS EFA layer), then launch training:

docker build -t "${FULL_IMAGE}" -f miles.Dockerfile .
docker push "${FULL_IMAGE}"

# Qwen3-4B, colocated:
bash recipe/run_grpo_qwen3_4b.sh

Alternatively, you can run a recipe directly from the Ray head pod (no local ray CLI or port-forward required) once the RayCluster is deployed and its workers are Running and Ready:

./run-on-cluster.sh --recipe run_grpo_qwen3_4b.sh --env ./env_vars

See the upstream examples/training/miles/README.md for the full walkthrough (KubeRay install, model download, checkpoint conversion, MoE recipe, and troubleshooting).

Source

This page summarizes the upstream test case at examples/training/miles. The authoritative version pins, image digests, EFA installer version, and reward-service details live in the upstream files — chiefly miles.Dockerfile and the README.md. Where those files pin a base image by digest (radixark publishes mutable snapshot tags), always consult the Dockerfile for the current, resolvable value. Per-instance hardware specifications (RAM, vCPU, GPU memory, EFA device counts, and bandwidth) are documented by AWS in the EC2 P5 instance documentation; the miles project source is radixark/miles.