Oxen

A high-speed data version control system built in Rust to manage massive machine learning datasets with Git-like simplicity.

Oxen handles the heavy lifting of versioning massive datasets that break standard Git. It indexes files at 1,000+ per second, making it 10x to 100x faster than Git LFS for large-scale computer vision or NLP projects. The tool manages terabytes of data (images, videos, or 100GB+ CSVs) using a familiar command-line interface: add, commit, and push. By providing a Python client and a robust API, Oxen ensures that every model training run is backed by a precise, reproducible snapshot of the data.

https://www.oxen.ai