Benchmark Methodology

Client Overhead Validation

Before attributing throughput bottlenecks to model inference, we validated that the benchmark client itself is not the limiting factor.

Client Throughput Capacity

Client Latency Profile

Detailed Results

The echo server test confirms the Python asyncio benchmark client can sustain 1,006 requests/second at concurrency 100, with sub-millisecond overhead scaling well to 2,000 concurrent connections. Since the highest model throughput observed is ~11,218 tok/s for Qwen3-VL (effective ~5.5 req/s at 2048 input tokens), the client is never the bottleneck.

Test Environment

Hardware

Software Stack

Container Images

Container digests (pinned for reproducibility):

Model Checkpoint Revisions

Benchmark Protocol

Multi-Run Statistical Testing (n=5)

Each model was tested across 5 independent runs (container restarts between runs) to capture variance:

Fine-Grained Saturation Sweep (n=3)

A separate high-resolution sweep around the saturation knee:

Statistical Analysis

• Mean and standard deviation computed across runs at each concurrency level
• 95% confidence intervals computed using the t-distribution at 95% with df = n − 1, appropriate for small sample sizes
• Coefficient of variation (CoV) reported for variance characterization
• All throughput values are total throughput (input + output tokens per second)

Per-Model Configuration

DeepSeek V3.2

Llama-3.1-405B

Qwen3-VL-235B

Kimi-K2.5