Expert-Parallelism Backend Comparison on EFA

Compares three expert-parallel dispatch/combine backends (UCCL, DeepEP V1 NVSHMEM, DeepEP V2 NCCL GIN) under common Decode-like and Prefill-like workloads on B200 nodes over EFA.

Overview

This micro-benchmark compares three expert-parallel dispatch/combine backends through common Decode-like and Prefill-like communication workloads over EFA:

Each workload uses one external CUDA timing boundary and one logical payload definition. Backend-native latency and bandwidth fields remain diagnostics because they do not share one timing boundary or byte numerator.

EP-size definitions

The harness runs 8 GPU ranks per B200 node, so each EP size maps to a node count as follows. These definitions are the single source of truth used throughout this page:

EP32 means 32 GPU ranks on 4 B200 nodes, not 32 B200 nodes.

Workload profiles

The Prefill-like timing boundary starts with the dispatch input and exact route ready and includes the dispatch layout required by the normal UCCL and DeepEP V1 APIs, dispatch, and combine completion. The Decode-like boundary starts with the dispatch input ready and includes dispatch and combine completion. Host-side FP8 conversion runs once before either boundary, so no arm is charged for its Python quantizer. In the Decode-like FP8 cell the low-latency UCCL and DeepEP V1 kernels still quantize internally, inside the boundary; each backend is measured from its own API’s dispatch-ready entry point.

Controls held constant

Both profiles hold the following constant across backends:

The model shape (hidden size 7,168, 256 experts, top-k 8 experts/token) is the “Model shape” control in the upstream README and matches the --hidden=7168, --experts=256, and --top-k=8 arguments passed by the per-rank launcher. The token counts (128 tokens/rank for Decode-like, 4,096 tokens/rank for Prefill-like) and the iteration counts (20 warmup and 100 measured) are the values recorded in the upstream README’s workload-profile and controls tables, and the runner gates the scored matrix on exactly 20 warmup and 100 measured iterations.

The default of 20 independent starts is the harness setting; the two reported B200 campaigns used lower replication levels on purpose. As stated in the upstream RESULTS.md, the primary campaign covers EP16 with 16 independent process starts, “which is the replication level at which the bootstrap intervals below carry information,” and the companion campaign covers EP16 and EP32 with 4 starts to show the EP-size axis, with its intervals described as weak and its EP32 rows read as a direction rather than a measurement.

Each process start contributes its median of 100 slowest-rank iteration measurements, and the report then takes the median across the independent process starts. Iterations within one process are not treated as independent replicates.

Common logical throughput

Each valid expert assignment contributes the dispatch tensor, FP8 scales when FP8 dispatch is selected, and the BF16 combine tensor. Backend metadata is excluded. Scale-out logical bytes include only assignments whose destination expert is on another node. These are secondary logical efficiency metrics, not observed wire bandwidth: within one profile, EP size, and dtype cell the byte numerator is a shared constant, so they rank backends identically to the primary latency metric.

Running the comparison

The harness runs on p6-b200.48xlarge nodes in a single EKS cluster. The upstream README requires 8 allocatable GPUs and 8 allocatable EFA devices on every selected node. EP_WORLD_SIZES defaults to 16 32, using the EP-size definitions above (8 ranks/node), and the largest configured EP size determines the required node count (4 nodes for EP32; a 2-node cluster can run EP_WORLD_SIZES=16). KUBECTL_CONTEXT is required explicitly so a concurrent process changing the default context cannot redirect the campaign.

Launch an exclusive run against a named node set with a unique namespace and durable artifact directory:

campaign_id=ep-b200-$(date -u +%Y%m%d%H%M%S)
CAMPAIGN_ID="${campaign_id}" \
EP_BENCHMARK_NODES=node-a,node-b,node-c,node-d \
PROTECTED_NODES_CSV="" \
ARTIFACT_ROOT="/shared/artifacts/${campaign_id}" \
KUBECTL_CONTEXT=aps1-shared \
LOCK_MODE=exclusive \
./run_ep_comparison.sh

LOCK_MODE=exclusive claims the configured shared Lease only when its holder is empty, and the runner releases only a Lease that it still owns. To run beside a coordinated campaign, set LOCK_MODE=observe with EXPECTED_LOCK_HOLDER and a disjoint PROTECTED_NODES_CSV; observe mode never mutates the shared Lease and verifies the exact holder before every arm and again before aggregation.

Scope limits

These profiles measure synthetic dispatch-plus-combine communication. Prefill-like does not measure TTFT, and Decode-like does not measure TPOT. Neither profile measures expert compute, communication/computation overlap, end-to-end training, serving throughput, or end-to-end latency. Results apply only to the reported profile, EP size, routing distribution, hardware, and runtime stack.

Source

The model shape (hidden size 7,168, 256 experts, top-k 8), token counts (128 decode / 4,096 prefill per rank), iteration counts (20 warmup / 100 measured), the 16- and 4-start campaigns, the p6-b200.48xlarge node/device counts, and the run/lock commands are transcribed from the upstream README.md and RESULTS.md. The --hidden, --experts, and --top-k values are also visible in the per-rank launcher run_ep_rank.sh, and the 20-warmup/100-measured gate and the 8 GPU / 8 EFA per-node checks are enforced in run_ep_comparison.sh.