Serving large language models at scale is expensive, a challenge that directly impacts the economics of AI margin collapse for API providers. When you send a prompt to an LLM, the GPU doesn't just calculate the next token and discard the state. It must retain the history of the conversation to compute subsequent tokens. This history lives in the Key-Value (KV) cache.
In early serving setups, managing this cache was highly inefficient. Systems regularly ran out of memory even when GPU compute utilization remained low. The engineering team behind vLLM identified memory fragmentation as the primary blocker, rather than raw compute limitations.
By introducing PagedAttention, a technique inspired by operating system virtual memory, they changed how models are served.
The KV Cache Bottleneck
To understand the architecture of vLLM, we have to look at the mechanics of text generation.
LLM inference runs in two phases: prefill and decode. During prefill, the model processes the entire input prompt at once. During decode, the model generates tokens sequentially, one by one. Each new token generation requires the keys and values of all previous tokens in the sequence to calculate the attention mechanism.
Storing these keys and values in GPU memory forms the KV cache.
The memory footprint of this cache is enormous. For a model with L layers, H attention heads, a head dimension of D, a batch size of B, and a sequence length of S, the KV cache size in bytes (using 16-bit float precision) is calculated as:
Size = 2 * B * S * L * H * D * 2
The first factor of 2 accounts for the separate Key and Value vectors. The final factor of 2 represents the bytes per parameter in FP16 or BF16 precision.
Let's apply this to a standard 70B parameter model. If the model has 80 layers, 64 attention heads, and a head dimension of 128, a single request with a sequence length of 2048 tokens requires about 2.6 GB of memory just for the KV cache. Multiply this by a batch size of 32, and the KV cache alone demands over 80 GB of memory. This exceeds the capacity of a standard Nvidia A100 GPU before even loading the model parameters.
In older frameworks, memory had to be allocated contiguously for the maximum possible sequence length. If a user sent a short prompt and got a quick answer, the unused allocated space sat empty. Other requests could not touch it.
This design led to massive internal fragmentation. External fragmentation also occurred as requests of varying lengths started and completed at different times, leaving small, unusable gaps in the memory space. On average, legacy serving engines wasted between 60% and 80% of their GPU memory on these empty allocations.
PagedAttention Mechanics
vLLM eliminates this waste by decoupling logical sequences from physical memory locations.
Instead of allocating a single contiguous block of memory for a request's KV cache, PagedAttention divides the KV cache of each sequence into fixed-size blocks. Each block holds the keys and values for a fixed number of tokens, typically 16. These blocks do not need to be contiguous in physical memory.
The system maintains a logical block table, mapping the logical sequence of tokens to physical blocks on the GPU.
Logical Blocks: [ Block 0 ] ──> [ Block 1 ] ──> [ Block 2 ]
│ │ │
▼ ▼ ▼
Physical Blocks: [ Phys 42 ] [ Phys 109 ] [ Phys 12 ] (Non-contiguous)
When a request arrives, the scheduler allocates logical blocks. As the model generates tokens, the engine maps these logical blocks to physical blocks drawn from a global pool of free memory.
Because the allocations are small and uniform, external fragmentation disappears. Internal fragmentation is confined to the final block of a sequence. For a block size of 16 tokens, the worst-case waste is 15 tokens worth of memory, which translates to less than 4% of a block.
This allows vLLM to utilize almost all available GPU memory for active cache storage. The engine can pack significantly more concurrent requests into the same hardware.
Continuous Batching
Traditional serving engines relied on static batching. Under static batching, the engine groups a set of requests and processes them together. The entire batch must wait for the longest request to finish generating before any new requests can start. If three requests in a batch finish in 10 tokens, but the fourth takes 300 tokens, the three completed slots remain idle.
Early batching-on-the-fly implementations improved on this by grouping requests dynamically, but they still suffered from the same batch-level synchronization bottleneck.
vLLM uses continuous batching, also known as iteration-level scheduling. The scheduler operates at the level of individual execution steps rather than entire requests.
At the end of each token generation step, the engine inspects the active batch. If a request has finished generating (by hitting a stop token or the length limit), the scheduler immediately removes it. The scheduler then pulls a new request from the waiting queue and inserts it into the active batch for the next iteration.
This approach keeps the GPU compute cores saturated. The execution loop never pauses to wait for a slow request to complete.
The System Architecture
The internal architecture of vLLM is structured into three primary components: the Scheduler, the Cache Engine, and the Worker.
Host CPU
┌───────────┐ ┌──────────────┐
│ Scheduler │ ─────> │ Cache Engine │
└─────┬─────┘ └──────────────┘
│ Control │ Memory
▼ ▼
Worker (GPU)
┌───────────────────────────────────┐
│ PagedAttention Kernels │
└───────────────────────────────────┘
The Scheduler runs on the host CPU. It manages the request queue, tracks the logical-to-physical block mappings, and decides which requests enter the execution batch for the next step.
The Cache Engine manages the actual memory pools. It allocates a large block of GPU memory at startup and divides it into physical blocks. It also allocates a pool on CPU memory to support swapping.
The Worker runs on the GPU. It executes the model's forward pass. It receives the input tokens and block tables from the Scheduler. During the attention calculation, the Worker uses the block table to locate the non-contiguous physical blocks in GPU memory.
This separation of concerns keeps the latency-sensitive GPU execution path clean. The CPU handles the complex bookkeeping of block tables, while the GPU focuses on parallel matrix multiplication and memory access.
For teams building their own custom serving wrappers, documenting these structural choices in architecture decision records helps prevent repeating past debates as the system scales.
Memory Pressure and Preemption
Even with efficient memory allocation, incoming traffic can exceed GPU capacity. If a batch of requests generates exceptionally long sequences, the pool of free physical blocks will eventually run dry.
The Scheduler handles this situation using a preemption policy.
When the GPU runs out of blocks, the Scheduler must free up space to allow existing requests to progress. It has two mechanisms to achieve this: swapping and recomputation.
Swapping moves physical blocks from the GPU memory to the CPU's system memory (DRAM) over the PCIe bus. When GPU memory becomes available again, the engine swaps the blocks back.
Recomputation takes a different approach. The engine drops the physical blocks for a request entirely, freeing the memory. When the request is scheduled again, the engine re-runs the prompt to reconstruct the KV cache.
Swapping is typically faster because transferring data over PCIe is faster than running the model forward pass again, especially for long prompts. If CPU memory also fills up, the engine defaults to recomputation.
The Scheduler uses a First-In-First-Out (FIFO) policy to select which requests to preempt. The oldest requests are prioritized to finish, while the newest requests are preempted first. This prevents starvation and ensures requests that have consumed the most compute resources reach completion.
Custom CUDA Kernels for PagedAttention
Standard attention implementations, such as FlashAttention, assume that the KV cache for a sequence is stored contiguously in memory. Because PagedAttention stores the KV cache in fragmented blocks, these standard kernels cannot read the data correctly.
The vLLM team wrote custom CUDA kernels to handle the fragmented layout.
During the decode phase, the attention computation is highly memory-bound. The GPU spends most of its time waiting for KV cache data to travel from High Bandwidth Memory (HBM) to the processor's SRAM.
The custom PagedAttention kernels read the block table directly within the GPU. They fetch the non-contiguous blocks in parallel, perform the attention math, and write the output back. This design minimizes memory access overhead and ensures that the fragmentation of the cache does not degrade execution speed.
Implications for Infrastructure Design
The architectural layout of vLLM alters how teams size and scale their inference infrastructure.
Because memory utilization is near 100%, throughput is no longer limited by memory waste. Under identical hardware configurations, vLLM regularly delivers two to four times the throughput of standard Hugging Face implementations. However, performance is only one factor; when selecting infrastructure, developers choose tools that encode trust and reliability over raw benchmarks.
This efficiency makes advanced generation techniques practical at scale. For example, parallel sampling (generating multiple responses for a single prompt) is common in code generation and reasoning tasks.
In vLLM, multiple output sequences can share the physical blocks of the prompt's KV cache. The engine only allocates new blocks when the generated paths diverge. This shared block architecture reduces the memory footprint of parallel sampling to a fraction of what traditional serving engines require.
