Benchmark Results Overview

Test Configuration

Peak Throughput Summary

* Nemotron Super 49B exhibits throughput oscillation at c=256 and c=1024 (~1,587 tok/s, roughly half of peak) due to TP=1 batch scheduling effects. Peak throughput is stable at c=64–128 and c=512.

Throughput per Active Parameter

A useful metric for comparing architectural efficiency: how many tokens per second does each active billion parameters produce?

Nemotron Nano's extreme per-parameter efficiency comes from the Mamba hybrid architecture: SSM layers have no KV cache overhead and process tokens in O(n), keeping the memory bandwidth pipeline saturated. The NVFP4 variant achieves 5,192 tok/s per active-B on a single GPU, making it the most cost-efficient configuration tested.

Latency Summary (at 32 Concurrent)

Concurrency 32 represents a realistic interactive workload: enough load to keep GPUs busy without excessive queuing.

Key Observations

• Nemotron Nano FP8 has the lowest TTFT at c=32 (206 ms) among all models, with TPOT under 8ms
• Nemotron Nano NVFP4 achieves similar latency on a single GPU: slightly higher TTFT (280 ms) due to TP=1 vs TP=2
• MiniMax M2.5 has the lowest TTFT at c=32 (91 ms) despite being a larger model, likely because TP=4 provides more compute for prefill
• GLM-5 has the highest TTFT at c=32 (1,341 ms) due to DSA's attention computation overhead during prefill, but reasonable TPOT (33.56 ms)
• All models achieve ITL p99 under 41ms at c=32, suitable for real-time streaming

Scaling Curves

Output Throughput vs Concurrency

• Nemotron Nano FP8 (TP=2) leads at ~18,800 tok/s peak, with NVFP4 (TP=1) close behind at ~15,600 tok/s on a single GPU
• MiniMax M2.5 and both Nemotron variants plateau around c=512
• DeepSeek V3.2 scales well to c=512 (4,370 tok/s): MLA's compressed KV cache enables higher concurrency
• GLM-5 saturates earlier at ~c=128 (2,132 tok/s) due to its larger active parameter count (40B) and lower KV cache capacity

TPOT vs Concurrency

TPOT (decode latency) increases gradually with concurrency as batch sizes grow. Key observations:

• Nemotron Nano FP8/NVFP4 maintain the lowest TPOT across all concurrency levels (~3ms at c=1, ~15ms at c=512), reflecting the Mamba hybrid architecture's efficient decode phase
• GLM-5 and DeepSeek V3.2 show steeper TPOT growth at high concurrency: their larger active parameter counts (37-40B) consume more memory bandwidth per decode step
• Nemotron Super 49B stays remarkably flat (~12ms) up to c=128 before the oscillation pattern kicks in at c=256/1024

TTFT vs Concurrency

TTFT increases with concurrency as prefill requests queue behind active decode operations. MiniMax M2.5 maintains the lowest TTFT across the board due to having more GPUs (TP=4) handling prefill compute.

Per-GPU Throughput Efficiency

Per-GPU throughput normalizes for tensor parallelism, revealing how efficiently each model uses its allocated GPUs. Nemotron Nano NVFP4 achieves 15,575 tok/s on a single GPU: 1.65x more per-GPU throughput than the FP8 variant on 2 GPUs, and 58x more than GLM-5 per GPU. This metric drives deployment decisions: fewer GPUs per model instance means more instances per node and higher aggregate throughput.

Goodput (SLO-Constrained Throughput)

Raw throughput tells you the maximum output rate, but production deployments need to meet latency SLOs. Goodput measures how many requests per second meet all SLO targets simultaneously.

Tested on Nemotron Nano 30B FP8 (TP=2) with SLOs: TTFT < 500ms and TPOT < 50ms.

Peak goodput is at c=64 (13.29 req/s), while peak throughput is at c=512+ (18,800+ tok/s). Beyond c=64, TTFT exceeds the 500ms SLO and goodput drops sharply: at c=512, only ~6% of requests meet both SLOs despite maximum throughput.

This demonstrates the classic throughput-vs-latency trade-off: the goodput-optimal concurrency is 8x lower than the throughput-optimal concurrency. Production deployments should target c=32–64 for interactive workloads with strict SLOs, and c=256–512 only for batch processing where latency is not critical.

Goodput: FP8 (TP=2) vs NVFP4 (TP=1)

FP8 peaks at 13.29 req/s (c=64), NVFP4 peaks at 8.90 req/s (c=64). FP8 has higher peak goodput because TP=2 distributes prefill across 2 GPUs, keeping TTFT lower at moderate concurrency. However, NVFP4 maintains more consistent goodput across c=32–256 (7.21–8.87 req/s) while FP8 drops sharply past c=64. NVFP4 still wins on per-GPU efficiency: it achieves 8.90 goodput req/s on 1 GPU vs FP8's 13.29 on 2 GPUs, delivering better per-GPU throughput even under SLO constraints.

Startup Times (Model Already Cached)

Startup times measured with models already in HuggingFace cache. First-time downloads add 1–5 minutes depending on model size and network speed.

Why Mostly MoE?

Four of the five models in this cookbook use Mixture of Experts (MoE) architecture. This reflects the current state of open-source LLMs:

• No dense model above 100B has been released since Llama 3.1 405B (July 2024)
• Every major open-source model since mid-2024 uses MoE: DeepSeek V3, GLM-5, MiniMax M2.5, Qwen3, Llama 4
• MoE enables larger total parameter counts (more knowledge capacity) while keeping per-token compute manageable

The exception is Nemotron Super 49B, a dense NAS-optimized transformer based on Llama 3.3. It was included specifically because its standard attention architecture is compatible with NVIDIA Dynamo's NIXL KV transfer for disaggregated serving: a key testing requirement the MoE models could not satisfy.

Architecture Diversity

The five models represent five distinct attention mechanisms:

This diversity means each model has fundamentally different memory and compute characteristics during inference, making them interesting comparison points beyond raw throughput numbers.