LlamaEdge

LlamaEdge is the easiest, fastest LLM runtime and API server for local or edge deployment.

This is LlamaEdge: the lightweight, high-performance solution for running customized LLMs on local or edge devices. We leverage the Rust and Wasm (WebAssembly) technology stacks, delivering a total runtime dependency under 30MB, which eliminates the 5GB-plus overhead of typical Python environments . LlamaEdge provides an OpenAI-compatible API service, allowing you to quickly host and interact with models like the Llama2 family in GGUF format . Deployment is cross-platform; you get a single, portable binary that runs at native speed across CPUs, GPUs, and NPUs (no complex toolchains required) .

https://llamaedge.com