KV Cache Offloading

Overview

KV cache offloading stores key-value cache data in CPU memory when GPU HBM is exhausted, enabling:

• Longer context lengths
• More concurrent requests
• Better memory utilization

Basic Usage

docker run --rm \
  --device /dev/kfd \
  --device /dev/dri \
  --group-add=video \
  --ipc=host \
  -p 8000:8000 \
  vllm/vllm-openai-rocm:latest \
  --model MODEL_NAME \
  --kv-offloading-backend native \
  --kv-offloading-size 64 \
  --disable-hybrid-kv-cache-manager

Configuration Options

Backend Options

• native: vLLM's built-in offloading (recommended)
• lmcache: LMCache integration for advanced caching

Model Compatibility

DeepSeek V3.2 (685B) Error

KeyError: 'model.layers.0.self_attn.indexer.k_cache'

MLA (Multi-head Latent Attention) uses an indexer-based KV cache structure that the OffloadingConnector cannot handle.

Working Configurations

Llama-3.1-405B

docker run --rm \
  --device /dev/kfd \
  --device /dev/dri \
  --group-add=video \
  --ipc=host \
  -p 8000:8000 \
  vllm/vllm-openai-rocm:latest \
  --model meta-llama/Llama-3.1-405B-Instruct \
  --tensor-parallel-size 8 \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768

Qwen3-VL-235B

docker run --rm \
  --device /dev/kfd \
  --device /dev/dri \
  --group-add=video \
  --ipc=host \
  --env "VLLM_USE_TRITON_FLASH_ATTN=0" \
  -p 8000:8000 \
  vllm/vllm-openai-rocm:latest \
  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
  --tensor-parallel-size 8 \
  --max-model-len 32768 \
  --kv-offloading-backend native \
  --kv-offloading-size 64 \
  --disable-hybrid-kv-cache-manager

Sizing Guidelines

Offload Buffer Size

When to Use Offloading

1. 128K+ context with large batch sizes
2. 1000+ concurrent sequences
3. Large models (200B+) with long context

Alternatives to Offloading

Before enabling KV offloading, consider:

1. FP8 KV Cache

Reduces KV cache memory by 50%:

--kv-cache-dtype fp8

2. Reduce Context Length

--max-model-len 16384

3. Increase GPU Memory Utilization

--gpu-memory-utilization 0.95

Performance Impact

KV offloading adds latency due to CPU-GPU transfers:

Exact impact varies by model and workload. The trade-off is worthwhile when you need context lengths or concurrency that wouldn't otherwise fit.

Troubleshooting

GEMM Kernel Error (Llama-405B, MoE models)

RuntimeError: wrong! device_gemm with the specified compilation parameters does not support this GEMM problem

This occurs with large models when AITER GEMM kernels encounter incompatible configurations. Adding KV offloading flags resolves it, even if you don't need the extra memory:

--kv-offloading-backend native \n--kv-offloading-size 64 \n--disable-hybrid-kv-cache-manager

Affected models: Llama-3.1-405B, Qwen3-VL-235B, and other large MoE models.

OOM Despite Offloading

Increase the offload buffer:

--kv-offloading-size 128

Or reduce context length:

--max-model-len 16384