Latest Content

Optimization

Updated on 12 March, 2026

Improve LLM inference performance on AMD Instinct GPUs with FP8 quantization, KV cache optimization, and concurrency tuning.

FP8 Quantization
Reduce memory usage and improve throughput with FP8 quantization on AMD Instinct GPUs.
KV Cache Offloading
Extend effective memory by offloading KV cache to CPU memory.
Concurrency Tuning
Maximize throughput by tuning vLLM for high concurrent request loads.
AITER Configuration
Configure AMD's AI Tensor Engine for ROCm (AITER) to accelerate vLLM inference.