PPLX Garden Expert Parallelism Benchmark

All-to-all dispatch and combine latency benchmarks for MoE expert parallelism using pplx-garden and UCCL-EP over EFA

Overview

This benchmark measures all-to-all dispatch and combine latency for Mixture-of-Experts (MoE) expert parallelism using Perplexity’s pplx-garden kernels. It compares expert-parallel (EP) communication performance across EFA-based and ConnectX-7 (CX7) transports, including comparisons against DeepEP and the UCCL-EP backend.

This page is filed under the communication benchmark category because it measures all-to-all collective communication latency; the upstream material lives under the micro-benchmarks/expert-parallelism family, which is preserved in the githubPath above.

PPLX Garden Benchmark

Clone the pplx-garden repository:

git clone https://github.com/perplexityai/pplx-garden.git

Build the images, then convert the benchmark image to an enroot squash file for Slurm. Run these commands from the directory containing pplx-garden.Dockerfile, with the pplx-garden clone alongside it:

docker build -t pplx-garden-dev - < ./pplx-garden/docker/dev.Dockerfile && docker build -t pplx-garden -f pplx-garden.Dockerfile .

This command is transcribed verbatim from the upstream README. Note the build context of each stage: the first docker build passes dev.Dockerfile to Docker on stdin (- reads the Dockerfile from standard input) to produce the pplx-garden-dev base image, and the second builds pplx-garden from pplx-garden.Dockerfile in the current directory. Because the stdin build has no file context, the dev.Dockerfile stage must be self-contained (no COPY/ADD from a local context); the pplx-garden.Dockerfile stage, by contrast, uses FROM pplx-garden-dev:latest and copies the pplx-garden clone into /app from the current directory. If your copy of dev.Dockerfile requires local files, supply an appropriate build context instead of piping the Dockerfile on stdin.

enroot import -o pplx-garden.sqsh dockerd://pplx-garden

Run the benchmark with the upstream Slurm script:

sbatch pplx-garden.sbatch

The pplx-garden.sbatch script runs benchmarks.bench_all_to_all across 2 nodes with 8 GPUs per node. As written, it launches with --nets-per-gpu 2, --nvlink=8, and --max-num-tokens 128, which corresponds to the decode (128-token) configuration. The prefill (4096-token) results in the table below are reported upstream but are not produced by the token count hard-coded in the committed script; reproducing them requires adjusting --max-num-tokens to 4096. Consult the upstream script and README for the exact configuration used for each row.

All-to-all performance results

The tables below are transcribed verbatim from the upstream README. Blank cells were not measured; crash indicates the run did not complete. Columns prefixed with My are the benchmark author’s own runs on this setup; the remaining columns are the reference figures reported upstream. See the UCCL-EP variant for how the UCCL-EP numbers are produced.

Decode (128 tokens) dispatch and combine median latency (μs)

Prefill (4096 tokens) dispatch and combine median latency (μs)

Anomalies preserved from the source. Two prefill cells break the otherwise-monotonic trend and appear physically implausible relative to their neighbors: the EP8 pplx-CX7 dispatch value of 5071.1 is roughly 5x the surrounding CX7 dispatch values (which trend downward: 4665.2 → 4011.8 → 2734.8) and about 5x the EP8 pplx-EFA dispatch value of 1052.4; and the EP16 pplx-CX7 combine value of 1062.2 is far below its EP32 and EP64 neighbors (7526.8 and 8771.1). These values are reproduced exactly as reported in the upstream README, which does not annotate or explain them. They may reflect transcription artifacts or measurement outliers in the source; treat them with caution and verify against the current upstream README before relying on them.

UCCL-EP variant

A UCCL-EP build is provided for comparison. Clone uccl-project/uccl alongside the benchmark directory, then build pplx-garden_uccl-ep.Dockerfile (which uses FROM pplx-garden:latest and copies the uccl clone into /opt/uccl) and run its Slurm script:

git clone https://github.com/uccl-project/uccl.git

docker build -t pplx-garden_uccl-ep -f pplx-garden_uccl-ep.Dockerfile .

enroot import -o pplx-garden_uccl-ep.sqsh dockerd://pplx-garden_uccl-ep

sbatch uccl-pplx-garden.sbatch

The uccl-pplx-garden.sbatch script runs the UCCL-EP variant, benchmarks.uccl_bench_all_to_all, with the --use-uccl flag. Like the base script, it launches across 2 nodes with 8 GPUs per node and --max-num-tokens 128. Consult the upstream script for its exact runtime flags.

Source

The clone/build/run commands, the Dockerfile and .sbatch names, and every latency figure (including the crash cells and blank/not-measured cells) are transcribed verbatim from the upstream micro-benchmarks/expert-parallelism/pplx-garden-benchmark/README.md. pplx-garden.Dockerfile, pplx-garden_uccl-ep.Dockerfile, and the .sbatch scripts live in that upstream benchmark directory and are the source of truth for the exact launch configuration. Because these figures are a point-in-time snapshot, always consult the upstream README for the latest measurements.