Apache Spark MLlib

Apache Spark MLlib is a high-performance distributed machine learning library designed to scale iterative algorithms across massive clusters.

MLlib delivers speed and scalability by leveraging Spark's in-memory computing architecture. It provides a robust suite of tools including classification (SVMs, Logistic Regression), regression, and clustering (K-Means). The library simplifies complex workflows with Pipelines for feature engineering and hyperparameter tuning (CrossValidator). By integrating directly with Spark SQL and DataFrames, MLlib allows data scientists to run large-scale training jobs on petabyte-scale datasets using Java, Scala, Python, or R.

https://spark.apache.org/mllib/