vLLM

A high-performance LLM serving framework known for high throughput and ease of use.

vLLM manages KV cache via PagedAttention, achieving several times the throughput of naive serving and exposing an OpenAI-compatible API—now a de facto standard for self-hosted inference.