DeepSeek V3.2 (685B)

2 min readUpdated 17 March 2026

Deploy DeepSeek V3.2 (685B parameters) on AMD Instinct GPUs.

Model Overview

Property	Value
Model ID	`deepseek-ai/DeepSeek-V3.2`
Architecture	MoE with MLA (Multi-head Latent Attention)
Total Parameters	685B
Context Length	163,840 tokens
Download Size	~254 GB

Quick Start

BASH
docker run --rm \
  --group-add=video \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --device /dev/kfd \
  --device /dev/dri \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  --env "VLLM_ROCM_USE_AITER=1" \
  --env "AITER_ENABLE_VSKIP=0" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai-rocm:latest \
  --model deepseek-ai/DeepSeek-V3.2 \
  --tensor-parallel-size 8 \
  --block-size 1 \
  --quantization fp8
Powered by Vultr AgentBeta

Critical Configuration

Warning

These flags are mandatory for DeepSeek V3.2 on AMD:

--block-size 1 - Required for MLA architecture (will error without it)
AITER_ENABLE_VSKIP=0 - Prevents crashes on MI300X/MI325X
VLLM_ROCM_USE_AITER=1 - Enables optimized kernels

Full Features Configuration

Enable chat, tool calling, and reasoning mode:

BASH
docker run --rm \
  --group-add=video \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --device /dev/kfd \
  --device /dev/dri \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  --env "VLLM_ROCM_USE_AITER=1" \
  --env "AITER_ENABLE_VSKIP=0" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai-rocm:latest \
  --model deepseek-ai/DeepSeek-V3.2 \
  --tensor-parallel-size 8 \
  --block-size 1 \
  --quantization fp8 \
  --tokenizer-mode deepseek_v32 \
  --tool-call-parser deepseek_v32 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v3
Powered by Vultr AgentBeta

Additional Flags Explained

Flag	Purpose
`--tokenizer-mode deepseek_v32`	Enables chat completions endpoint
`--tool-call-parser deepseek_v32`	Parses tool/function calls
`--enable-auto-tool-choice`	Allows model to decide when to use tools
`--reasoning-parser deepseek_v3`	Extracts reasoning blocks

Memory Usage

Configuration	Total Memory	Per GPU (TP=8)
FP8	~83 GB	~10 GB
FP16	~180 GB	~22 GB

Metric	Value
Load Time	~348 seconds (~5.8 minutes)
FP8 BMM Warmup	~3 minutes (first run only)

MI325X (256GB) easily fits the model with room for KV cache.

Performance (MI325X Verified)

Concurrency Scaling

Concurrent	Throughput	p99 Latency
10	2,857 tok/s	22.76s
50	5,694 tok/s	23.49s
100	5,518 tok/s	24.22s
200	5,486 tok/s	24.14s
500	5,657 tok/s	23.46s

Multi-run means (n=5).

Peak Performance

Peak Throughput: 5,786 tok/s at 1,000 concurrent
Saturation Point: ~50 concurrent requests
100% Success Rate at all tested concurrency levels

See DeepSeek V3.2 Stress Testing for detailed results including saturation testing.

Test Endpoints

Chat Completion

BASH

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V3.2",
    "messages": [{"role": "user", "content": "Hello, who are you?"}]
  }'

Tool Calling

BASH

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V3.2",
    "messages": [{"role": "user", "content": "What is the weather in San Francisco?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get weather for a location",
        "parameters": {
          "type": "object",
          "properties": {"location": {"type": "string"}},
          "required": ["location"]
        }
      }
    }]
  }'

Reasoning Mode

BASH

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V3.2",
    "messages": [{"role": "user", "content": "Solve step by step: If a train travels 120km in 2 hours, and then 180km in 3 hours, what is the average speed?"}]
  }'

Known Limitations

KV Cache Offloading Not Supported

DeepSeek V3.2 uses MLA (Multi-head Latent Attention) which is incompatible with vLLM's KV cache offloading:

Error: KeyError: 'model.layers.0.self_attn.indexer.k_cache'

The MLA architecture uses an indexer-based KV cache that the OffloadingConnector cannot handle. Use the large HBM capacity instead (256GB per MI325X is sufficient).

FP8 KV Cache Not Supported

Do not use --kv-cache-dtype fp8:

Error: ROCMAiterMLASparseBackend doesn't support fp8 kv_cache_dtype

vLLM automatically uses the correct fp8_ds_mla format for DeepSeek models.

Troubleshooting

Crash on Startup

Ensure AITER_ENABLE_VSKIP=0 is set. If unset, it defaults to true which causes crashes on MI300X/MI325X.

Chat Endpoint Returns Error

Add --tokenizer-mode deepseek_v32 to enable the chat completions endpoint.

Long Warmup Time

FP8 BMM kernel pre-compilation takes ~3 minutes on first run. Subsequent starts use cached kernels.

0 Comments

Start the conversation

Be the first to comment and share your perspective with the community.

DeepSeek V3.2 (685B)

2 min readUpdated 17 March 2026

Deploy DeepSeek V3.2 (685B parameters) on AMD Instinct GPUs.

Model Overview

Property	Value
Model ID	`deepseek-ai/DeepSeek-V3.2`
Architecture	MoE with MLA (Multi-head Latent Attention)
Total Parameters	685B
Context Length	163,840 tokens
Download Size	~254 GB

Quick Start

BASH
docker run --rm \
  --group-add=video \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --device /dev/kfd \
  --device /dev/dri \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  --env "VLLM_ROCM_USE_AITER=1" \
  --env "AITER_ENABLE_VSKIP=0" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai-rocm:latest \
  --model deepseek-ai/DeepSeek-V3.2 \
  --tensor-parallel-size 8 \
  --block-size 1 \
  --quantization fp8
Powered by Vultr AgentBeta

Critical Configuration

Warning

These flags are mandatory for DeepSeek V3.2 on AMD:

--block-size 1 - Required for MLA architecture (will error without it)
AITER_ENABLE_VSKIP=0 - Prevents crashes on MI300X/MI325X
VLLM_ROCM_USE_AITER=1 - Enables optimized kernels

Full Features Configuration

Enable chat, tool calling, and reasoning mode:

BASH
docker run --rm \
  --group-add=video \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --device /dev/kfd \
  --device /dev/dri \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  --env "VLLM_ROCM_USE_AITER=1" \
  --env "AITER_ENABLE_VSKIP=0" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai-rocm:latest \
  --model deepseek-ai/DeepSeek-V3.2 \
  --tensor-parallel-size 8 \
  --block-size 1 \
  --quantization fp8 \
  --tokenizer-mode deepseek_v32 \
  --tool-call-parser deepseek_v32 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v3
Powered by Vultr AgentBeta

Additional Flags Explained

Flag	Purpose
`--tokenizer-mode deepseek_v32`	Enables chat completions endpoint
`--tool-call-parser deepseek_v32`	Parses tool/function calls
`--enable-auto-tool-choice`	Allows model to decide when to use tools
`--reasoning-parser deepseek_v3`	Extracts reasoning blocks

Memory Usage

Configuration	Total Memory	Per GPU (TP=8)
FP8	~83 GB	~10 GB
FP16	~180 GB	~22 GB

Metric	Value
Load Time	~348 seconds (~5.8 minutes)
FP8 BMM Warmup	~3 minutes (first run only)

MI325X (256GB) easily fits the model with room for KV cache.

Performance (MI325X Verified)

Concurrency Scaling

Concurrent	Throughput	p99 Latency
10	2,857 tok/s	22.76s
50	5,694 tok/s	23.49s
100	5,518 tok/s	24.22s
200	5,486 tok/s	24.14s
500	5,657 tok/s	23.46s

Multi-run means (n=5).

Peak Performance

Peak Throughput: 5,786 tok/s at 1,000 concurrent
Saturation Point: ~50 concurrent requests
100% Success Rate at all tested concurrency levels

See DeepSeek V3.2 Stress Testing for detailed results including saturation testing.

Test Endpoints

Chat Completion

BASH

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V3.2",
    "messages": [{"role": "user", "content": "Hello, who are you?"}]
  }'

Tool Calling

BASH

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V3.2",
    "messages": [{"role": "user", "content": "What is the weather in San Francisco?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get weather for a location",
        "parameters": {
          "type": "object",
          "properties": {"location": {"type": "string"}},
          "required": ["location"]
        }
      }
    }]
  }'

Reasoning Mode

BASH

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V3.2",
    "messages": [{"role": "user", "content": "Solve step by step: If a train travels 120km in 2 hours, and then 180km in 3 hours, what is the average speed?"}]
  }'

Known Limitations

KV Cache Offloading Not Supported

DeepSeek V3.2 uses MLA (Multi-head Latent Attention) which is incompatible with vLLM's KV cache offloading:

Error: KeyError: 'model.layers.0.self_attn.indexer.k_cache'

The MLA architecture uses an indexer-based KV cache that the OffloadingConnector cannot handle. Use the large HBM capacity instead (256GB per MI325X is sufficient).

FP8 KV Cache Not Supported

Do not use --kv-cache-dtype fp8:

Error: ROCMAiterMLASparseBackend doesn't support fp8 kv_cache_dtype

vLLM automatically uses the correct fp8_ds_mla format for DeepSeek models.

Troubleshooting

Crash on Startup

Ensure AITER_ENABLE_VSKIP=0 is set. If unset, it defaults to true which causes crashes on MI300X/MI325X.

Chat Endpoint Returns Error

Add --tokenizer-mode deepseek_v32 to enable the chat completions endpoint.

Long Warmup Time

FP8 BMM kernel pre-compilation takes ~3 minutes on first run. Subsequent starts use cached kernels.

0 Comments

Start the conversation

Be the first to comment and share your perspective with the community.