accelerate

Accelerate scales your native PyTorch code to any distributed environment (multi-GPU, TPU, DeepSpeed) with minimal changes: typically four lines of code.

This is Hugging Face Accelerate: the definitive library for simplifying large-scale PyTorch training. It abstracts away the complexities of distributed setups (like `torch.distributed`) and hardware (GPU, TPU, multi-node). You write standard PyTorch; Accelerate handles the heavy lifting. The library automatically supports mixed-precision training and integrates advanced techniques, including DeepSpeed and Fully Sharded Data Parallelism (FSDP). Engineers gain immediate, efficient scaling without rewriting their entire codebase, maximizing resource utilization across clusters.

https://huggingface.co/docs/accelerate/index