NVRx

NVIDIA's resilient training toolkit for fault-tolerant distributed workloads

Overview

NVRx is NVIDIA’s resilient training toolkit that provides fault tolerance, automatic recovery, and checkpoint management for large-scale distributed training workloads. It ensures training jobs can survive node failures without full restart.

Key Features

• Automatic fault detection — Monitors GPU/node health during training
• Seamless recovery — Resumes from last checkpoint without manual intervention
• Efficient checkpointing — Asynchronous, distributed checkpoint strategies
• Node failure handling — Replaces failed nodes and continues training
• Integration — Works with Slurm and SageMaker HyperPod health monitoring

When to Use

• Large-scale training jobs (64+ GPUs) where node failures are likely
• Long-running pre-training runs (days/weeks) that need fault tolerance
• Production training pipelines that need to meet SLA requirements