Llama 3.1 (405B) Stress Testing

Concurrency Scaling

Scaling Results

Observations:

• Throughput scales well up to 150 concurrent requests
• Performance peaks at 150 concurrent (6,927 tok/s) before slight degradation at 200
• p99 latency increases from 6.15s to 14.49s across the scaling range
• Dense architecture provides consistent, predictable performance

Stress Tests

Stress Test Results

Key findings:

• Long context prefill efficient: 8,240 tok/s with 4K context
• Very long context (8K): 6,794 tok/s total throughput
• Long output generation: 1,224 tok/s output throughput
• All tests passed with 100% success rate

Saturation Testing

Extreme Load Results

Observations:

• Peak throughput of 15,944 tok/s achieved at 500 concurrent
• Saturation begins at 750 concurrent (throughput plateau)
• 100% success rate maintained even under extreme load
• System remains stable at 1,000 concurrent requests

Recommendations

Test Configuration

Launch Command

docker run --rm \
  --group-add=video \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --device /dev/kfd \
  --device /dev/dri \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai-rocm:latest \
  --model meta-llama/Llama-3.1-405B-Instruct \
  --tensor-parallel-size 8 \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768

Test Environment