NCCL Send/Recv Ring Test

A minimal, self-contained NCCL point-to-point benchmark that measures send/recv latency and bandwidth across GPUs in a ring topology

Overview

The NCCL Send/Recv Ring Test is a minimal NCCL point-to-point benchmark that measures send/recv latency and bandwidth across GPUs arranged in a ring topology. It is designed to run on AWS SageMaker HyperPod clusters with Slurm and EFA.

Why this exists

The repository already includes micro-benchmarks/nccl-tests/, which wraps NVIDIA’s nccl-tests suite (including sendrecv_perf). This standalone benchmark is a minimal, self-contained alternative — no nccl-tests build dependency, a single C file, and easy to modify. It is intended as a starting point for users who want to understand or extend NCCL send/recv patterns without pulling in the full suite.

What it does

Each MPI rank sends a 64 MB buffer to its right neighbour and receives from its left neighbour in a ring pattern using ncclSend/ncclRecv grouped calls. After a warm-up phase, it runs 100 timed iterations and reports average latency and bandwidth.

Prerequisites

• CUDA toolkit
• NCCL library
• MPI (OpenMPI is pre-installed on HyperPod at /opt/amazon/openmpi)
• At least 2 GPUs

Building

The Slurm script builds automatically before running. To build manually:

make

Override paths if needed:

make CUDA_HOME=/usr/local/cuda NCCL_HOME=/usr MPI_HOME=/opt/amazon/openmpi

The default paths and Makefile variable names are defined in the upstream Makefile, which is the source of truth for build configuration.

Running on SageMaker HyperPod (Slurm)

Submit the job:

sbatch run_nccl_test.sbatch

Monitor output:

squeue -u $USER
tail -f nccl_test_<jobid>.out

Customizing the Slurm job

Edit run_nccl_test.sbatch to adjust:

Running locally with mpirun

make
mpirun -np 2 ./nccl_sendrecv_test

Configuration

Compile-time constants in nccl_sendrecv_test.c:

Example output

The values below are illustrative and will vary based on instance type, message size, and network conditions. This block reproduces the sample output shown in the upstream README.md.

---------- NCCL Send/Recv Ring Test ----------
Ranks           : 16
Message size    : 67108864 bytes (64 MB)
Iterations      : 100
Avg latency     : 2.450 ms
Algo BW (1-dir) : 27.39 GB/s
Bidir throughput : 54.78 GB/s
-----------------------------------------------

Source

The filenames and defaults on this page match the upstream source directory micro-benchmarks/nccl-sendrecv:

• run_nccl_test.sbatch (--nodes=2, --ntasks-per-node=8, --gpus-per-node=8).
• Makefile (defines CUDA_HOME, NCCL_HOME, and MPI_HOME ?= /opt/amazon/openmpi).
• README.md, which documents the MSG_SIZE = 64 MB, ITERATIONS = 100, and WARMUP_ITERATIONS = 10 compile-time constants and provides the illustrative example output reproduced above.