Concurrency Tuning

Key Parameters

max_num_seqs

Maximum concurrent sequences in a single batch.

--max-num-seqs 2048

max_num_batched_tokens

Maximum total tokens across all sequences in a batch.

--max-num-batched-tokens 32768

gpu_memory_utilization

Fraction of GPU memory allocated for model weights and KV cache.

--gpu-memory-utilization 0.95

Throughput Results (NVIDIA HGX B200 Verified)

All results use input=2048, output=512 tokens, random dataset.

Nemotron Nano 30B (FP8, TP=2, 2 GPUs)

Saturation: ~512 concurrent. Peak: 18,829 tok/s sustained on 2 GPUs.

Nemotron Super 49B (FP8, TP=1, 1 GPU)

Saturation: ~64 concurrent. Peak: 3,816 tok/s on 1 GPU. Throughput oscillates at c=256/1024 due to TP=1 batch scheduling effects.

MiniMax M2.5 (FP8, TP=4, 4 GPUs)

Saturation: ~512 concurrent. Peak: 8,838 tok/s sustained on 4 GPUs.

GLM-5 744B (FP8, TP=8, 8 GPUs)

Saturation: ~128 concurrent. Peak: 2,132 tok/s sustained on 8 GPUs.

DeepSeek V3.2 685B (FP8, TP=8, 8 GPUs)

Saturation: ~512 concurrent. Peak: 4,370 tok/s sustained on 8 GPUs.

Peak Throughput and Saturation

Key Findings

• Active parameters predict per-GPU throughput: Nemotron Nano (3B active) achieves 9,415 tok/s/GPU; MiniMax M2.5 (10B active) achieves 2,210 tok/s/GPU; DeepSeek V3.2 (37B active) achieves 546 tok/s/GPU. Fewer active parameters = higher per-GPU throughput.
• Saturation varies by KV cache pressure: Models with efficient KV caching (MLA, Mamba) saturate at ~512 concurrent. GLM-5 saturates earlier at ~128 due to higher KV cache consumption per token.
• MoE efficiency: Four of five models leverage MoE to activate only a fraction of total parameters per token, enabling high throughput despite large total parameter counts. Nemotron Super 49B is the exception: a dense transformer where all 49B parameters are active on every token.
• TTFT degrades linearly with concurrency: Expected behavior as prefill requests queue behind active decode operations.
• DeepSeek V3.2 scales best among TP=8 models: MLA's compressed KV cache provides 1.15M tokens capacity vs GLM-5's 691K, enabling 2x the sustained throughput at high concurrency.
• Zero failures: 100% success rate at all concurrency levels up to 1,024 for all five models.

TTFT vs TPOT Trade-Offs

Interactive applications should optimize for TTFT (time to first token): the user-perceived latency before streaming begins. Batch processing should optimize for throughput (tok/s).

Nemotron Nano TTFT Thresholds

MiniMax M2.5 TTFT Thresholds

Goodput Benchmarks (Nemotron Nano FP8)

Goodput measures requests per second that meet all latency SLOs simultaneously. Tested with TTFT < 500ms and TPOT < 50ms.

Peak goodput at c=64 (13.29 req/s). Beyond this, TTFT exceeds 500ms and most requests violate the SLO. TPOT stays well under the 50ms SLO at all concurrency levels: TTFT is the binding constraint.

Nemotron Nano NVFP4 (TP=1, 1 GPU)

NVFP4 peaks at 8.90 req/s (c=64): lower than FP8's 13.29 because TP=1 means all prefill runs on a single GPU, making TTFT the bottleneck sooner. However, NVFP4 uses half the GPUs (1 vs 2), so it delivers better per-GPU efficiency even under SLO constraints.

For production with strict SLOs, target c=32–64. For batch processing where latency doesn't matter, push to c=512+.

Configuration Profiles

High-Throughput (Batch Processing)

$ vllm serve <model> \
  --tensor-parallel-size <TP> \
  --max-model-len 32768 \
  --max-num-seqs 2048 \
  --max-num-batched-tokens 65536 \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code

Low-Latency (Interactive)

$ vllm serve <model> \
  --tensor-parallel-size <TP> \
  --max-model-len 8192 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code

Multi-Instance (Maximum Node Utilization)

For smaller models that don't need all 8 GPUs, run multiple independent instances:

# Instance 1: GPUs 0-1
$ CUDA_VISIBLE_DEVICES=0,1 vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
  --tensor-parallel-size 2 --port 8000 --trust-remote-code &

# Instance 2: GPUs 2-3
$ CUDA_VISIBLE_DEVICES=2,3 vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
  --tensor-parallel-size 2 --port 8001 --trust-remote-code &

# Instance 3: GPUs 4-5
$ CUDA_VISIBLE_DEVICES=4,5 vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
  --tensor-parallel-size 2 --port 8002 --trust-remote-code &

# Instance 4: GPUs 6-7
$ CUDA_VISIBLE_DEVICES=6,7 vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
  --tensor-parallel-size 2 --port 8003 --trust-remote-code &

This achieves ~75,000 tok/s aggregate throughput on a single 8-GPU node: 4x the single-instance peak.

Chunked Prefill

In vLLM V1 (0.12.0+), chunked prefill is always enabled. Tune the chunk size via --max-num-batched-tokens:

Multi-Step Scheduling

Reduce CPU-GPU synchronization overhead by batching scheduler steps:

--num-scheduler-steps 10

Benchmarking Your Configuration

Use vLLM's built-in benchmark tool to test your specific setup:

# Single concurrency level
$ vllm bench serve \
  --base-url http://localhost:8000 \
  --model <model_id> \
  --dataset-name random \
  --random-input-len 2048 \
  --random-output-len 512 \
  --num-prompts 400 \
  --max-concurrency 64 \
  --save-result \
  --result-filename results.json

# Full sweep (use the included script)
$ ./scripts/bench.sh <model_id> http://localhost:8000