批处理

Merging multiple requests into one inference batch to raise throughput.

Batching improves GPU utilization, raising throughput and cutting per-request cost, at the cost of longer waits per request. Offline jobs (bulk generation, evals) fit well; serving frameworks use continuous batching to balance latency.