Static vs. Dynamic vs. Continuous Batching in LLM Inference

Static batching, and why waiting for a full batch is simple but costly under real traffic
Dynamic batching, and how a timeout window fixes the worst of that cost
Continuous batching, and why large language models need scheduling at the token level instead of the request level
Static vs. Dynamic vs. Continuous Batching in LLM Inference

Introduction
Most GPUs serving AI models spend most of their time doing nothing. A request comes in, the model runs it, and the GPU sits idle waiting for the next one while it could have handled several at once for close to the same cost. This gets worse with large language models specifically, since one request might finish in a few tokens and another might run for a thousand, so whatever handles the traffic has to deal with highly uneven work, not identical jobs arriving one after another.

Batching is how you fix this. Instead of running the model once per request, you group several requests together and run them through the same loaded weights in one pass, turning idle GPU cycles into throughput you’re already paying for. The part that actually matters is how you form those groups, because a batching scheme built for uniform workloads breaks down fast once request lengths stop being predictable