DeepSeek V3.2

Model Overview

Architecture

DeepSeek V3.2 uses Multi-Latent Attention (MLA), which compresses KV projections into a lower-dimensional latent space before caching:

• Standard GQA: Stores full key and value tensors per head per token
• MLA: Projects KV into a shared latent vector, then reconstructs on the fly

This compression means:

• Much less KV cache per token than a standard 685B model
• Higher memory efficiency at long context lengths
• Standard KV cache offloading is not supported (different cache format)
• --block-size 1 is required

Combined with MoE routing (~37B active of 685B total), DeepSeek V3.2 is memory-efficient for its size.

Quick Start

$ vllm serve deepseek-ai/DeepSeek-V3-0324 \
  --tensor-parallel-size 8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code \
  --quantization fp8 \
  --block-size 1

Or with Docker:

$ docker run --rm --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  vllm/vllm-openai:v0.16.0 \
  --model deepseek-ai/DeepSeek-V3-0324 \
  --tensor-parallel-size 8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code \
  --quantization fp8 \
  --block-size 1

Configuration

Memory Usage (NVIDIA HGX B200 Verified)

With TP=8 on FP8 (on-the-fly quantization):

Performance (NVIDIA HGX B200 Verified)

Benchmark parameters: 2048 input tokens, 512 output tokens, random dataset. TP=8 on 8x NVIDIA HGX B200.

Concurrency Scaling

Peak Performance

Key Observations

• Strong scaling to c=512 — DeepSeek V3.2 scales nearly linearly up to 256 concurrent requests and continues gaining throughput at 512. The 4,370 tok/s at c=1024 is only marginally better than c=512 (4,321 tok/s), indicating saturation around 512.
• MLA efficiency — MLA's compressed KV cache provides 1,149,440 tokens of cache capacity (75.23 GiB per GPU), enabling high concurrency without running out of KV cache.
• Low single-request latency — TPOT of 9.28ms at c=1 is among the lowest of all models tested, demonstrating MLA's efficient decode phase.
• TTFT remains manageable — Even at c=256, TTFT stays under 3 seconds, making DeepSeek V3.2 suitable for interactive workloads at moderate concurrency.

Test Endpoints

Chat Completion

$ curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V3-0324",
    "messages": [{"role": "user", "content": "Explain Multi-Latent Attention"}],
    "max_tokens": 256
  }'

Text Completion

$ curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V3-0324",
    "prompt": "The advantage of MLA over standard GQA is",
    "max_tokens": 128
  }'

Known Issues

• vLLM 0.16.0+ recommended — While DeepSeek V3 is supported in earlier versions, vLLM 0.16.0 adds FlashInfer MLA backend and TRT-LLM ragged prefill optimizations that significantly improve performance.
• On-the-fly quantization: Unlike the other models which use pre-quantized checkpoints, DeepSeek V3.2 is quantized at load time via --quantization fp8. This adds to startup time but avoids needing a separate FP8 checkpoint.
• block-size 1 required: MLA's compressed KV format is incompatible with the default block size. Forgetting --block-size 1 will produce a clear error message.
• No KV cache offloading: MLA's custom cache format doesn't support standard vLLM KV cache offloading. Use Dynamo's tiered KV caching instead if you need to extend beyond GPU VRAM.
• Large download: The base model is ~642 GB in BF16 weights (quantized to FP8 at load time).