KV 缓存

Caching already-computed key-value vectors during inference to avoid recomputation.

Generating each new token requires attention over all previous ones; KV cache stores their keys and values, speeding up incremental decoding. Cache grows linearly with context length, a major part of long-context inference cost.