Apache Airflow

Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring complex data workflows.

Airflow is the industry-standard workflow orchestration tool: it defines pipelines as Directed Acyclic Graphs (DAGs) using pure Python code. This 'workflows as code' approach ensures version control and dynamic pipeline generation. The platform features a robust Scheduler, which triggers tasks based on time or external events, and a scalable architecture (e.g., using Celery or Kubernetes Executors) for handling massive workloads. Users leverage the rich web UI to visualize DAG dependencies, inspect logs, monitor real-time progress, and manage task retries, providing full operational control over ETL, MLOps, and infrastructure automation processes.

https://airflow.apache.org/