KV Cache Offloading

Overview

KV cache offloading stores key-value cache data in CPU memory when GPU HBM is exhausted, enabling:

• Longer context lengths beyond what fits in GPU VRAM
• More concurrent requests at a given context length
• Trading latency for capacity

When to Consider Offloading

Basic Usage

$ vllm serve <model> \
  --tensor-parallel-size <TP> \
  --cpu-offload-gb 32 \
  --trust-remote-code

The --cpu-offload-gb flag allocates a CPU memory buffer for KV cache overflow. When GPU KV cache is full, least-recently-used blocks are moved to CPU memory.

Configuration Options

Model Compatibility

KV cache offloading works differently depending on the attention architecture:

Sizing Guidelines

CPU Buffer Size

The NVIDIA HGX B200 node has ample system RAM (typically 1-2 TB), so large CPU buffers are feasible.

Memory Bandwidth Impact

CPU memory bandwidth (~200 GB/s DDR5) is 40x slower than NVIDIA HGX B200 HBM (8.0 TB/s). Offloading adds latency proportional to the cache blocks transferred:

The performance impact depends on how often the active working set exceeds GPU cache. If most requests fit in GPU VRAM and only overflow requests hit CPU, the average impact is small.

Alternatives to Offloading

Before enabling KV cache offloading, try these approaches first: they maintain full GPU-speed operation.

1. NVFP4 Quantization (NVIDIA HGX B200 Exclusive)

Halves model weight memory, freeing VRAM for KV cache:

$ vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 \
  --tensor-parallel-size 1 \
  --trust-remote-code

2. FP8 KV Cache

Halves per-token KV cache size independently of model quantization:

$ vllm serve <model> --kv-cache-dtype fp8

3. Reduce Context Length

--max-model-len 16384  # Instead of 32768

4. Increase GPU Memory Utilization

--gpu-memory-utilization 0.95  # Instead of 0.90

Adds ~9 GB KV capacity per GPU. Use with caution under peak load.

5. Use More GPUs (Increase TP)

Higher tensor parallelism distributes model weights across more GPUs, freeing VRAM per GPU for KV cache:

--tensor-parallel-size 4  # Instead of 2

Dynamo Tiered Caching

NVIDIA Dynamo extends KV cache offloading with a multi-tier hierarchy:

1. GPU HBM: Primary (179 GB per NVIDIA HGX B200, 8.0 TB/s)
2. CPU DRAM: Host memory (1-2 TB, ~200 GB/s)
3. Local NVMe SSD: Persistent cache (~7 GB/s)
4. Remote storage: NFS, S3-compatible

Dynamo's KV-aware router intelligently places requests on GPUs that already have relevant cache, minimizing transfers.

See Dynamo Overview for details. Dynamo 0.9.1 was successfully started on this node with vLLM 0.16.0, though tiered caching requires the full Dynamo stack (etcd, NATS) which was not deployed.