How continuous batching powers vLLM, TGI, and SGLang

Key Takeaways

• Continuous batching updates the batch every decoding step rather than once per batch, eliminating head-of-line blocking
• PagedAttention manages KV cache memory like an OS manages RAM, reducing fragmentation from 60-80% to near zero
• vLLM, TGI, and SGLang all share this foundation but differ in defaults and emphasis around chunked prefill and prefix caching

Serving a large language model is a scheduling problem dressed up as a hardware problem. GPUs want thousands of operations at once. Requests arrive one at a time, with unpredictable prompts and response lengths. The scheduler's job is to reconcile these two facts. Continuous batching, the technique behind vLLM, Text Generation Inference (TGI), and SGLang, does exactly that by updating the batch every single decode step instead of waiting for an entire group of requests to finish.

DigitalOcean published a detailed technical walkthrough of this scheduler design, explaining why static batching breaks under real traffic and how continuous batching fixes it. The core insight: if you let requests enter and exit the batch at every token generation step, you eliminate the GPU idle time that kills throughput.

Why static batching fails

Text generation happens in two phases. Prefill processes the entire prompt in one forward pass, building the key-value (KV) cache that stores attention data for every token. This phase is compute-bound. Decode generates the response one token at a time, streaming the model's weights through memory for each step. This phase is memory-bound.

A single request barely uses the GPU during decode. You move gigabytes of weights to produce one token's worth of arithmetic. The obvious fix is batching: run many requests together. Once you've paid to stream the weights, adding more sequences to the same step costs almost nothing.

Static batching collects N requests and runs them as a group. The problem: requests finish at different times. A short response completes in 50 tokens. A long one needs 500. Static batching forces the short request to wait in the batch, wasting its slot. Or it forces the system to wait until all requests finish before admitting new work. Either way, you get head-of-line blocking and GPU idle time.

Continuous batching updates every step

Continuous batching solves this by making admission and eviction decisions at every decode step. When a request finishes generating, its slot immediately opens for a new request. No waiting for the batch to complete. No wasted slots.

This freedom comes with complexity. Two mechanisms make it work:

• Preemption handles what happens when KV cache memory runs out mid-generation. The system can either recompute the evicted request's cache later or swap it to CPU memory.
• Paged KV-cache management (PagedAttention) eliminates memory fragmentation. Instead of allocating contiguous memory for the maximum possible sequence length, it allocates memory in pages, like an operating system manages RAM.

PagedAttention is the key innovation here. Traditional approaches pre-allocate memory assuming the worst case sequence length. This wastes 60-80% of GPU memory on padding. PagedAttention reduces that waste to near zero by allocating only what each request actually needs, when it needs it.

The prefill vs decode scheduling problem

Continuous batching creates a new challenge: how do you integrate a new request's compute-heavy prefill into the stream of ongoing decodes? Prefill wants to saturate the GPU's compute units. Decode wants to minimize memory latency. Running them simultaneously requires careful scheduling.

The main techniques for this are now common across all three systems:

• Chunked prefill breaks long prompts into smaller pieces, interleaving prefill chunks with decode steps so neither starves the other.
• Prefix caching stores the KV cache for common prompt prefixes, avoiding redundant computation when multiple requests share the same system prompt.
• Paged memory lets the system pack more concurrent requests into GPU memory, improving utilization.

How vLLM, TGI, and SGLang differ

vLLM, TGI, and SGLang all implement continuous batching with PagedAttention. The differences are mostly in defaults and emphasis rather than fundamental architecture.

vLLM pioneered PagedAttention and remains the reference implementation. It prioritizes throughput and has strong support for speculative decoding. TGI, from Hugging Face, emphasizes production readiness and integrates tightly with their model hub. SGLang focuses on structured generation and complex prompt patterns, making it attractive for applications that need constrained outputs.

All three face the same fundamental tradeoffs. More aggressive prefill scheduling improves throughput for new requests but increases latency for ongoing generations. More conservative approaches keep latency stable but leave throughput on the table. The right choice depends on your traffic patterns and latency requirements.

Memory is the constraint, not compute

The KV cache dominates every decision in LLM serving. Every token processed adds an entry to the cache, and that entry lives in GPU memory for the request's entire lifetime. Long conversations mean large caches. Many concurrent requests mean many caches competing for the same memory pool.

This is why continuous batching matters. Static batching wastes memory on finished requests that haven't left the batch yet. Continuous batching reclaims that memory immediately. With PagedAttention eliminating fragmentation, you can fit substantially more concurrent requests into the same GPU.

Frequently Asked Questions

What is the difference between static batching and continuous batching?

Static batching groups requests and processes them until all finish before admitting new work. Continuous batching admits and evicts requests at every decode step, eliminating idle slots when requests finish early.

Why does continuous batching improve GPU utilization?

It eliminates head-of-line blocking. Short requests no longer hold slots hostage while waiting for long requests to finish. The batch stays full because new requests enter immediately when slots open.

What is PagedAttention and why does it matter?

PagedAttention manages KV cache memory in pages rather than contiguous blocks. This eliminates fragmentation from pre-allocating worst-case sequence lengths, letting you fit more concurrent requests into GPU memory.

Which LLM inference engine should I use: vLLM, TGI, or SGLang?

vLLM is the reference implementation and safe default. TGI integrates well with Hugging Face workflows. SGLang excels at structured generation. All three share the same core scheduler design.

How much throughput improvement can I expect from continuous batching?

vLLM reports up to 23x throughput improvement over naive HuggingFace Transformers baseline. Real gains depend on your traffic patterns, but 2-4x latency reduction and near-maximum GPU utilization are achievable.