GLM-5

Model Overview

Architecture

GLM-5 uses Differential Sparse Attention (DSA), a novel attention mechanism that:

• Selectively attends to important tokens, discarding low-relevance entries
• Reduces effective KV cache size by compressing unimportant positions
• Uses differential scoring to identify which tokens contribute meaningfully to generation

Combined with MoE routing (~40B active of 744B total), GLM-5 balances model capacity with inference efficiency. The architecture is designed for instruction following, reasoning, and code generation.

Implications for NVIDIA HGX B200 deployment:

1. Full node required: 744B total params in FP8 requires TP=8 (all 8 GPUs)
2. DSA reduces KV pressure: Sparse attention means less KV cache consumed per token than standard GQA
3. Memory-bound at decode: With ~89 GB per GPU in weights, only ~64 GB remains for KV cache

Quick Start

$ vllm serve zai-org/GLM-5-FP8 \
  --tensor-parallel-size 8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code

Or with Docker:

$ docker run --rm --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  vllm/vllm-openai:v0.16.0 \
  --model zai-org/GLM-5-FP8 \
  --tensor-parallel-size 8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code

Configuration

Memory Usage (NVIDIA HGX B200 Verified)

With TP=8 on FP8:

Performance (NVIDIA HGX B200 Verified)

Benchmark parameters: 2048 input tokens, 512 output tokens, random dataset. TP=8 on 8x NVIDIA HGX B200.

Concurrency Scaling

Peak Performance

Key Observations

• Early saturation at c=128 — GLM-5 peaks at 2,132 tok/s with 128 concurrent requests, well below MiniMax M2.5's ~512 saturation point. This is driven by the larger active parameter count (40B vs 10B) consuming more memory bandwidth per token.
• KV cache pressure — With ~89 GB model weights per GPU, only ~64 GB remains for KV cache (691K tokens total). At high concurrency, KV cache becomes the bottleneck.
• ITL p99 spike at c=64 — Inter-token latency jumps from 32ms to 475ms between c=32 and c=64, indicating the decode phase begins competing for memory bandwidth with prefill.
• TTFT degradation — TTFT grows dramatically beyond c=128 (3.6s → 57s → 175s), confirming prefill queuing under memory pressure.

Test Endpoints

Chat Completion

$ curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "zai-org/GLM-5-FP8",
    "messages": [{"role": "user", "content": "Explain Differential Sparse Attention"}],
    "max_tokens": 256
  }'

NVFP4 Variant

A community-quantized NVFP4 variant is available:

$ vllm serve lukealonso/GLM-5-NVFP4 \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code

NVFP4 could reduce the GPU requirement from TP=8 to TP=4, freeing 4 GPUs for another model. See FP8/NVFP4 Quantization.

Known Issues

• vLLM 0.16.0+ required — GLM-5's GlmMoeDsaForCausalLM architecture was added in vLLM 0.16.0 (PR #34124). Earlier versions will fail with an "unrecognized architecture" error. Also requires transformers from git main (5.x+).
• DeepGEMM JIT needs CUTLASS — The FP8 kernels use DeepGEMM, which JIT-compiles CUDA code at first run. Ensure nvcc is on PATH and CUTLASS headers are available in the DeepGEMM package (see Troubleshooting).
• Long first-run warmup — DeepGEMM warms up ~2,259 JIT kernels on first launch (~5 minutes). Subsequent launches use cached kernels.
• Large download — GLM-5 FP8 is ~705 GB. Pre-download before deployment.
• Custom code — Always pass --trust-remote-code for the DSA implementation.