Keep Distributed Training Running with Automated Workload Recovery

PUBLISHER ARTICLE PREVIEW

From the original article

As distributed AI training scales, failures across hardware and software dependencies increasingly disrupt training progress, making fast, reliable recovery essential. A stalled training rank can trigger an NVIDIA Collective Communications Library (NCCL) timeout, a GPU node can stop responding to the scheduler, or a straggler can slow the entire job. Each disruption is manageable on its own, but the cumulative impact and recovery overhead determine how efficiently the system converts wall-clock time into productive training progress or goodput.

Manual recovery becomes unsustainable when it must cover every training job across a cluster, around the clock, at scale. Each interruption demands log analysis, fault isolation, and a restart from a saved checkpoint, while affected GPUs remain idle during recovery. As the number and size of training jobs grow, overlapping failures increase the engineering burden and wasted GPU-hours.

Without continuous coverage, those losses can accumulate, increasing overall compute costs and extending time to train.

NVIDIA Recovery, part of NVIDIA DSX OS, the modular software portfolio with the NVIDIA DSX AI Factory Platform, is an open-source, workload-level service that automates recovery across a cluster’s training jobs, around the clock. It detects interruptions using application logs, GPU metrics, and scheduler signals, uses failure attribution to identify implicated resources, and coordinates isolation and checkpoint restarts from checkpoints on healthy capacity.

By automating this recovery loop across workloads, it handles concurrent interruptions, reduces manual intervention, and helps sustain training goodput. This post explains how the recovery loop works, how NVIDIA Recovery integrates with Slurm, and how to deploy it on your own cluster.

Our previous NVIDIA DGX Cloud resilience post, GTC talk and NVIDIA Mission Control 2.5 post describe a layered approach to keeping training productive. NVIDIA Recovery builds on that architecture by connecting infrastructure remediation to workload recovery.

Break-fix automation identifies faulty nodes, removes them from service, and routes hardware for remediation. Framework-level libraries such as NVRx provide recovery, hang detection, and straggler mitigation within individual training jobs, but each user must configure and manage these capabilities for their own workloads. Cluster workload managers such as Slurm manage resource allocation, queuing, and container execution.

Workload recovery is a distinct coordination problem: restoring productive training requires attributing failures to their causes, deciding whether and where affected jobs should restart, and acting consistently across workloads that share infrastructure. Addressing this problem requires a cluster-level control loop that integrates infrastructure remediation, framework-level recovery, and cluster workload management, with operational metrics that track recovery latency, recovery success rate, and time to restore productive training.

As shown in Figure 1, below, NVIDIA Recovery provides the dedicated workload recovery layer within the DSX OS Stack. Node health monitoring and isolation depend on the cluster’s orchestration and workload management stack. Kubernetes deployments use NVSentinel to monitor GPU health metrics and XID events and automatically cordon and drain faulty nodes. Many Slurm sites check node health with custom prolog and epilog scripts.