Llama 3.1 (405B) Validation Testing

Concurrency Scaling

Scaling Results

Observations:

• Good scaling from 5 to 50 concurrent requests
• Peak scaling throughput of 3,417 tok/s at 50 concurrent
• Performance plateaus at 50-100 concurrent (dense model limitation)
• Dense architecture provides consistent, predictable performance
• DEGRADED status indicates p99 latency >2x baseline (expected under concurrent load)

Stress Tests

Key findings:

• Long output generation (500 tokens): 600 tok/s with 15.5s p99 latency
• Long context (4K tokens): 2,727 tok/s with 6.7s p99 latency

Saturation Testing

Extreme Load Results

Observations:

• Peak throughput of 7,222 tok/s achieved at 150 concurrent
• Saturation begins at 200 concurrent requests
• 100% success rate maintained even under extreme load
• Consistent ~7,170 tok/s across 150-500 concurrent

Recommendations

Test Configuration

Launch Command

docker run --rm \
  --group-add=video \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --device /dev/kfd \
  --device /dev/dri \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  --env "VLLM_USE_TRITON_FLASH_ATTN=0" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai-rocm:latest \
  --model meta-llama/Llama-3.1-405B-Instruct \
  --tensor-parallel-size 8 \
  --max-model-len 32768 \
  --quantization fp8

Test Environment