VLLMs

vLLM: The high-performance inference engine, leveraging PagedAttention to deliver up to 24x higher throughput for LLM serving.

vLLM (Virtual Large Language Model) is the definitive open-source library for high-speed LLM inference and serving. Its core innovation is PagedAttention: an algorithm managing the Key-Value (KV) cache like operating system virtual memory, dramatically boosting throughput and reducing latency. Originating from the Sky Computing Lab at UC Berkeley, the project supports continuous batching and integrates seamlessly with popular models (e.g., Llama, Mixtral) and diverse hardware (NVIDIA, AMD). Deploy production-grade models with an OpenAI-compatible API server for maximum efficiency and scalability.

https://vllm.ai/