TorchServe

TorchServe is the official, high-performance model serving framework for PyTorch, built by AWS and Meta to streamline production deployments.

TorchServe eliminates the friction of moving PyTorch models into production. It handles essential heavy lifting like multi-model serving, automated logging, and Prometheus metrics out of the box. You get native support for low-latency inference through batching and worker scaling, plus a robust management API for hot-swapping models without downtime. Whether you are deploying on Amazon SageMaker or a local Kubernetes cluster, TorchServe provides the standard interface needed to scale deep learning workloads efficiently.

https://pytorch.org/serve/