KServe

KServe is the standard Kubernetes custom resource for deploying high-performance, autoscaling inference services across frameworks like PyTorch, TensorFlow, and XGBoost.

Built on Knative and Istio, KServe simplifies production machine learning by abstracting complex networking and scaling logic. It provides advanced features out of the box: scale-to-zero for cost efficiency, canary rollouts for safe deployments, and standardized protocols like V2 (Open Inference Protocol). Whether you are running Large Language Models via vLLM or traditional Scikit-learn models, KServe handles the heavy lifting of model explainability, outlier detection, and multi-model serving on a unified control plane.

https://kserve.github.io/website/