Why vLLM p99 latency spikes in production (and how to fix it)

Key Takeaways

• A single long prefill can push inter-token latency up by 28x, which is where your p99 lives
• Chunked prefill breaks large prompts into smaller pieces so they don't block active decodes
• PagedAttention solves memory fragmentation but is not a scheduler fix for tail latency

Your vLLM benchmark shows low p50 latency and high tokens per second. Then real traffic hits and p99 latency jumps 5x to 10x worse than anything you measured. Same model, same GPU. The culprit is almost always a long prefill blocking the decodes already running, and fixing it requires understanding how chunked prefill and scheduling policies interact.

This problem catches teams off guard because benchmarks are too clean. Most load tests send requests at a steady pace with similar-sized prompts. Production traffic comes in bursts, mixes tiny prompts with huge ones, and shares the GPU across many concurrent users. Under that load, one thing hurts your tail more than raw compute: a big prompt monopolizing the GPU while everyone else's token generation waits.

Why continuous batching didn't finish the job

If you serve LLMs at scale, you've moved past static request-level batching. With static batching, the server locks a group of requests together and nobody leaves until the slowest one finishes. Fast requests wait around, GPU time gets wasted, and latency suffers.

Continuous batching, the approach from the Orca paper (Yu et al., 2022) that vLLM uses, fixed much of this. Instead of locking in one group for the whole run, the scheduler re-decides the batch before every step. Finished requests leave immediately and free their memory. Waiting requests get pulled in as soon as there's room.

But continuous batching still tends to run a prefill as one big unit. When a long prompt lands in the batch, that iteration takes a while, and every decode already in flight has to wait. Users who were streaming tokens suddenly hit a pause. That pause is your p99.

Prefill vs decode: two different workloads

Prefill processes all input tokens at once. It's compute-heavy, parallelizable, and can dominate GPU time for long prompts. Decode generates one token at a time per sequence. It's memory-bound and latency-sensitive because users are watching tokens stream in.

The problem is mixing them badly. When a 4,000-token prompt starts prefilling, the GPU is busy for tens or hundreds of milliseconds. Every decode in the batch stalls. The users waiting for their next token experience that stall directly.

This is why benchmarks that test only throughput miss the real issue. Average latency can look fine while tail latency destroys user experience.

How chunked prefill works

Chunked prefill breaks the long prefill into smaller pieces. Instead of processing all 4,000 tokens at once, the scheduler processes them in chunks of, say, 512 tokens. Between chunks, decodes can run.

The result is that no single prefill blocks the batch for too long. Users streaming tokens experience smaller pauses. Your p99 drops because the worst-case wait time is bounded by chunk size, not by the longest prompt in the system.

There's a cost. Chunked prefill adds some overhead because you're running more scheduler iterations. For pure throughput on uniform workloads, it can hurt. For mixed production traffic where tail latency matters, it's usually worth it.

PagedAttention is memory, not scheduling

PagedAttention, the core innovation from the vLLM paper (Kwon et al., 2023), treats the KV cache like virtual memory. Instead of reserving one big block per sequence, it splits the cache into small fixed-size blocks that can sit anywhere in GPU memory.

This cuts fragmentation waste by about 55% in typical deployments. You can keep more sequences in memory at once, which helps throughput. But PagedAttention doesn't decide when to run prefill versus decode. It solves a memory problem, not a scheduling problem.

The confusion comes because memory and scheduling are linked. Before PagedAttention, memory was usually the first limit on batch size because each sequence had to over-reserve cache space for its worst-case length. Once memory stops being the bottleneck, scheduling becomes the thing that boxes you in.

Scheduling policies that shape your tail

vLLM offers scheduling knobs beyond chunked prefill. The max_num_seqs parameter limits how many sequences can be in a batch. Higher values improve throughput but can hurt latency. The max_prefill_tokens parameter caps how many tokens can be prefilled in one iteration.

The scheduler also decides priority. Should a new request start prefilling immediately, or should active decodes finish first? Different policies suit different use cases. A chatbot wants low inter-token latency. A batch processing pipeline cares more about overall throughput.

The Sarathi-Serve paper introduced a stall-free scheduling approach that prioritizes decodes over prefills when there's contention. The idea is simple: a user waiting for a token is more latency-sensitive than a user whose request just arrived.

How to diagnose what's blocking you

Start by measuring p99 separately from p50 and average. If p99 is way worse than p50, you have a tail problem. If they're close, you have a throughput problem.

• Profile with realistic traffic patterns, not uniform load
• Check time_to_first_token (TTFT) separately from inter-token latency
• Watch for correlation between p99 spikes and long input sequences
• Measure GPU utilization during different batch compositions

If you see p99 spikes that correlate with long prompts entering the batch, chunked prefill is likely your fix. If you're bound by GPU compute even on small batches, you need more GPU, not better scheduling.

What real traffic shows

Teams running vLLM in production report 2x to 4x latency reductions with chunked prefill for long-context requests. The gains are biggest when traffic is bursty and prompt lengths vary widely.

The tradeoff is small. Chunked prefill adds scheduling overhead, which can reduce peak throughput by a few percent on uniform workloads. For most production deployments where user experience matters, that's a trade worth making.

Frequently Asked Questions

What causes vLLM p99 latency to spike in production?

Long prefills blocking decodes. When a big prompt starts prefilling, token generation for all other requests in the batch stalls until the prefill completes.

Does PagedAttention fix latency problems?

No. PagedAttention solves KV cache memory fragmentation, which improves throughput by letting you batch more sequences. It doesn't change how prefill and decode operations are scheduled.

Should I always enable chunked prefill?

For mixed production traffic where tail latency matters, yes. For batch processing with uniform prompt sizes where only throughput counts, you might skip it to avoid the small overhead.

How do I configure chunked prefill in vLLM?

Set the --enable-chunked-prefill flag and optionally configure max_num_batched_tokens to control chunk size. Smaller chunks bound tail latency more tightly but add more scheduling overhead.

What's a good benchmark for realistic latency testing?

Use traffic that varies prompt length significantly, arrives in bursts, and includes concurrent users. Steady-state load with uniform prompts will miss the tail latency problems that chunked prefill solves.