TensorRT

NVIDIA TensorRT is a high-performance deep learning inference optimizer and runtime that maximizes throughput and minimizes latency on NVIDIA GPUs.

TensorRT optimizes models from frameworks like PyTorch and TensorFlow for production deployment on NVIDIA hardware. It employs precision calibration (INT8 and FP16), layer fusion, and kernel auto-tuning to deliver up to 40x faster throughput than CPU-only platforms. Engineers use the ONNX workflow to deploy these engines across the NVIDIA ecosystem: from Jetson modules at the edge to H100 clusters in the data center. By maximizing CUDA core utilization and reducing memory overhead, TensorRT ensures real-time responsiveness for computer vision and generative AI applications.

https://developer.nvidia.com/tensorrt