Concurrency Tuning

Key Parameters

max_num_seqs

Maximum concurrent sequences in a single batch.

--max-num-seqs 2048

max_num_batched_tokens

Maximum total tokens across all sequences in a batch.

--max-num-batched-tokens 32768

gpu_memory_utilization

Fraction of GPU memory allocated for KV cache.

--gpu-memory-utilization 0.95

Throughput Results (MI325X Verified)

All results below are multi-run means (n=5) using input=2048, output=512 tokens.

Qwen3-VL-235B (BF16)

Llama-3.1-405B (FP8)

DeepSeek V3.2 (685B, FP8+AITER)

Kimi-K2.5 (1T, INT4 QAT, TP=4)

Peak Throughput & Saturation

Multi-run means (n=5). 100% success rate at all concurrency levels up to 1,000.

Key Findings

• Throughput saturates early: All models reach peak throughput by 100 concurrent requests; additional concurrency provides no benefit
• FP8 enables large models like Llama-405B and DeepSeek V3.2 on 8x MI325X
• Architecture matters: MoE models with fewer active parameters (Qwen 22B active) achieve higher throughput than dense models (Llama 405B)
• Active parameters predict throughput better than total parameters
• 100% reliability at all tested concurrency levels (up to 1,000) across all models

Cross-Model Analysis

*Kimi-K2.5 uses only 4 GPUs (TP=4), so per-GPU throughput is calculated as 952/4.

Key insights:

• MoE advantage: Qwen's sparse activation (22B of 235B) delivers the highest per-GPU throughput despite being run in BF16
• Dense models scale more linearly: Llama achieves 6.1x scaling (10→200) vs DeepSeek's 1.9x
• MLA latency cost: DeepSeek V3.2 and Kimi-K2.5 (both MLA) show higher p99 latency than GQA models
• Kimi-K2.5 constraint: Limited to TP=4 due to AITER MLA head count requirements, using only half the GPU cluster

Concurrency Recommendations

Values based on Qwen3-VL-235B multi-run means (n=5, input=2048, output=512). Other models follow similar saturation patterns — see per-model tables above.

Configuration Profiles

High-Throughput Configuration

# Environment
export VLLM_ROCM_USE_AITER=1
export NCCL_MIN_NCHANNELS=112

# Serving
docker run --rm \
  --device /dev/kfd \
  --device /dev/dri \
  --group-add=video \
  --ipc=host \
  --env "VLLM_ROCM_USE_AITER=1" \
  --env "NCCL_MIN_NCHANNELS=112" \
  -p 8000:8000 \
  vllm/vllm-openai-rocm:latest \
  --model MODEL_NAME \
  --tensor-parallel-size 8 \
  --max-num-seqs 2048 \
  --max-num-batched-tokens 65536 \
  --gpu-memory-utilization 0.95 \
  --num-scheduler-steps 15

Low-Latency Configuration

docker run --rm \
  --device /dev/kfd \
  --device /dev/dri \
  --group-add=video \
  --ipc=host \
  --env "VLLM_ROCM_USE_AITER=1" \
  -p 8000:8000 \
  vllm/vllm-openai-rocm:latest \
  --model MODEL_NAME \
  --tensor-parallel-size 4 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.90

Chunked Prefill

In vLLM V1, chunked prefill is always enabled and cannot be disabled. Tune performance via --max-num-batched-tokens:

Multi-Step Scheduling

Reduce GPU idle time by batching scheduler steps:

--num-scheduler-steps 15

Test Matrix

Benchmarking Script

Test concurrency scaling:

#!/bin/bash
MODEL="Qwen/Qwen3-VL-235B-A22B-Instruct"
MAX_TOKENS=100

for CONCURRENT in 10 25 50 100 200; do
    REQUESTS=$((CONCURRENT * 2))

    echo "Testing: $CONCURRENT concurrent, $REQUESTS total"

    START=$(date +%s.%N)

    for i in $(seq 1 $REQUESTS); do
        curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{"model": "$MODEL", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": $MAX_TOKENS}" &

        while [ $(jobs -r | wc -l) -ge $CONCURRENT ]; do
            sleep 0.05
        done
    done
    wait

    END=$(date +%s.%N)
    ELAPSED=$(echo "$END - $START" | bc)

    echo "Completed in ${ELAPSED}s"
done