Inference Cookbook for ROCm

This cookbook provides tested, working configurations for deploying LLMs on AMD hardware:

• Environment Setup - ROCm, Docker, and system configuration
• Model Deployment - Step-by-step guides for DeepSeek V3.2, Qwen3, and more
• Performance Optimization - FP8 quantization, KV cache tuning, AITER configuration
• Benchmarks - Real throughput numbers from MI325X testing

Get a model running in under 5 minutes:

docker run --rm \
  --group-add=video \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --device /dev/kfd \
  --device /dev/dri \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai-rocm:latest \
  --model Qwen/Qwen3-0.6B \

Test the endpoint:

curl http://localhost:8000/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{"model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "Hello!"}]}'

From our comprehensive testing on MI325X:

• FP8 quantization enables running large models (405B+) that wouldn't fit in BF16
• Throughput scaling up to 50–100 concurrent requests per model, with stable performance through 1,000
• Architecture matters - GQA models (Qwen3) significantly outperform MLA models (DeepSeek) in throughput
• KV cache offloading works with GQA models but NOT with MLA (DeepSeek, Kimi)

Multi-run means (n=5). Peak throughput measured at optimal concurrency per model.