FP8 and NVFP4 Quantization

Format Comparison

FP8 on NVIDIA HGX B200

FP8 is the default quantization for large model inference on the NVIDIA HGX B200. Most model providers now ship official FP8 checkpoints.

Pre-Quantized FP8 Models

All five models in this cookbook use FP8:

# Nemotron Nano: official FP8 checkpoint
$ vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
  --tensor-parallel-size 2 --trust-remote-code

# Nemotron Super 49B: official FP8 checkpoint
$ vllm serve nvidia/Llama-3_3-Nemotron-Super-49B-v1_5-FP8 \
  --tensor-parallel-size 1 --trust-remote-code

# GLM-5: official FP8 checkpoint
$ vllm serve zai-org/GLM-5-FP8 \
  --tensor-parallel-size 8 --trust-remote-code

# MiniMax M2.5: native FP8 support
$ vllm serve MiniMaxAI/MiniMax-M2.5 \
  --tensor-parallel-size 4 --trust-remote-code

# DeepSeek V3.2: on-the-fly FP8 quantization
$ vllm serve deepseek-ai/DeepSeek-V3-0324 \
  --tensor-parallel-size 8 --trust-remote-code --quantization fp8 --block-size 1

FP8 KV Cache

Separately from model weight quantization, you can quantize the KV cache to FP8. This reduces per-request memory usage and allows more concurrent requests:

$ vllm serve <model> --kv-cache-dtype fp8

NVFP4: NVIDIA HGX B200-Exclusive

NVFP4 is NVIDIA's 4-bit floating point format, supported only on Blackwell GPUs (NVIDIA HGX B200, B100, GB200). It halves memory compared to FP8 and doubles throughput for memory-bandwidth-bound workloads.

Available NVFP4 Models

Deploying with NVFP4

# Nemotron Nano in NVFP4: fits on a single GPU
$ vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code

NVFP4 enables deployment scenarios that aren't possible with FP8:

• Single-GPU deployment for models that would otherwise need TP=2
• Higher context lengths within the same VRAM budget
• More concurrent requests at the same context length

When to Use NVFP4

Memory Budget Calculator

For a given model and quantization format on the NVIDIA HGX B200 (179 GB VRAM per GPU):

Available VRAM per GPU = 179 GB × gpu_memory_utilization
Model memory per GPU = model_size_GB / tensor_parallel_size
KV cache per GPU = Available - Model memory
Max concurrent = KV cache per GPU / (per_request_kv_size × context_length)

Example: Nemotron Nano 30B

The biggest win from NVFP4 isn't more KV cache on the same GPU count: it's being able to drop TP entirely and run on fewer GPUs.

Benchmark: FP8 vs NVFP4 (Nemotron Nano 30B, NVIDIA HGX B200 Verified)

NVFP4 delivers 1.65x better cost efficiency than FP8 by eliminating the second GPU. On a full 8-GPU node, 8 NVFP4 instances produce ~124,000 tok/s aggregate vs ~75,000 tok/s from 4 FP8 instances. Zero failed requests across all concurrency levels for both formats.