Kimi-K2.5 (1T)

3 min readUpdated 17 March 2026

Deploy Kimi-K2.5 (1 trillion parameters) on AMD Instinct GPUs.

Model Overview

Property	Value
Model ID	`moonshotai/Kimi-K2.5`
Architecture	MoE with MLA (384 experts, 8 selected per token)
Total Parameters	1T (1 trillion)
Active Parameters	32B per token
Context Length	256K tokens
Vision	MoonViT encoder (400M params)
Download Size	~400 GB (compressed-tensors)

Quick Start

Warning

Kimi-K2.5 requires vLLM nightly build (rocm/vllm-dev:nightly). The stable release does not yet support this model.

BASH
docker run --rm \
  --name vllm-kimi-k25 \
  --ipc=host \
  --network=host \
  --group-add video \
  --group-add render \
  --cap-add=SYS_PTRACE \
  --cap-add=CAP_SYS_ADMIN \
  --security-opt seccomp=unconfined \
  --device /dev/kfd \
  --device /dev/dri \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "VLLM_ROCM_USE_AITER=0" \
  --env "VLLM_USE_TRITON_FLASH_ATTN=0" \
  rocm/vllm-dev:nightly \
  vllm serve moonshotai/Kimi-K2.5 \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --trust-remote-code \
  --block-size 1
Powered by Vultr AgentBeta

Critical Configuration

Warning

These settings are mandatory for Kimi-K2.5 on AMD:

VLLM_ROCM_USE_AITER=0 - AITER disabled (MLA head count incompatibility with TP=4)
--tensor-parallel-size 4 - Not 8! Required for MLA attention head distribution (64/4=16 heads per GPU)
--block-size 1 - Required for MLA architecture
--trust-remote-code - Model uses custom code
VLLM_USE_TRITON_FLASH_ATTN=0 - Required for vision encoder

Note

Why AITER is Disabled?

Unlike other models, Kimi-K2.5 runs with AITER disabled. The AITER MLA backend requires specific head counts (16 or 128 per GPU). With TP=8, Kimi-K2.5's 64 heads would give 8 heads per GPU (unsupported). With TP=4, we get 16 heads per GPU, but AITER still has compatibility issues with this model's MLA implementation.

Full Features Configuration

Enable chat, tool calling, and reasoning mode:

BASH
docker run --rm \
  --name vllm-kimi-k25 \
  --ipc=host \
  --network=host \
  --group-add video \
  --group-add render \
  --cap-add=SYS_PTRACE \
  --cap-add=CAP_SYS_ADMIN \
  --security-opt seccomp=unconfined \
  --device /dev/kfd \
  --device /dev/dri \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "VLLM_ROCM_USE_AITER=0" \
  --env "VLLM_USE_TRITON_FLASH_ATTN=0" \
  rocm/vllm-dev:nightly \
  vllm serve moonshotai/Kimi-K2.5 \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --trust-remote-code \
  --block-size 1 \
  --mm-encoder-tp-mode data \
  --tool-call-parser kimi_k2 \
  --reasoning-parser kimi_k2
Powered by Vultr AgentBeta

Additional Flags Explained

Flag	Purpose
`--mm-encoder-tp-mode data`	Data parallel mode for MoonViT vision encoder
`--tool-call-parser kimi_k2`	Parses tool/function calls in Kimi format
`--reasoning-parser kimi_k2`	Extracts reasoning blocks

Memory Usage

Configuration	Total Memory	Per GPU (TP=4)
Compressed-tensors (default)	~160 GB	~40 GB

Metric	Value
Load Time	~5-8 minutes (first run)

MI325X (256GB) easily fits the model with TP=4, leaving significant room for KV cache.

Performance (MI325X Verified)

Concurrent	Throughput	p99 Latency	Status
10	225 tok/s	77.83s	OK
50	583 tok/s	149.39s	OK
100	948 tok/s	183.35s	OK
200	950 tok/s	182.96s	OK
500	948 tok/s	183.23s	OK

Multi-run means (n=5).

Note

100% success rate across all concurrency levels (no failures)
Peak throughput: 952 tok/s at 1,000 concurrent requests
Saturation point: ~100 concurrent requests

See Kimi-K2.5 Stress Testing for detailed benchmark results including saturation testing up to 1,000 concurrent requests.

Test Endpoints

Chat Completion

BASH

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "moonshotai/Kimi-K2.5",
    "messages": [{"role": "user", "content": "Hello, who are you?"}]
  }'

Vision Request

BASH

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "moonshotai/Kimi-K2.5",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "What do you see in this image?"},
        {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}
      ]
    }]
  }'

Tool Calling

BASH

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "moonshotai/Kimi-K2.5",
    "messages": [{"role": "user", "content": "What is the weather in San Francisco?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get weather for a location",
        "parameters": {
          "type": "object",
          "properties": {"location": {"type": "string"}},
          "required": ["location"]
        }
      }
    }]
  }'

Comparison with Similar Models

Model	Total Params	Active Params	Architecture	Vision
Kimi-K2.5	1T	32B	MoE + MLA	Yes (MoonViT)
DeepSeek V3.2	685B	~37B	MoE + MLA	No
Qwen3-VL-235B	235B	22B	MoE + GQA	Yes

Known Limitations

Tensor Parallel Size Must Be 4

Unlike other models, Kimi-K2.5 requires TP=4:

Error: AITER MLA requires 16 heads per GPU

The MLA architecture has 64 attention heads that must be evenly distributed with 16 heads per GPU.

KV Cache Offloading Not Supported

Like DeepSeek V3.2, Kimi-K2.5 uses MLA which is incompatible with vLLM's KV cache offloading. The 256GB per MI325X provides sufficient capacity without offloading.

FP8 KV Cache Not Supported

Do not use --kv-cache-dtype fp8:

Error: MLA doesn't support fp8 kv_cache_dtype

The model uses its own compressed-tensors quantization format.

Troubleshooting

Crash on Startup

Ensure AITER_ENABLE_VSKIP=0 is set. If unset, it defaults to true which causes crashes on MI300X/MI325X.

TP Size Error

If you see attention head distribution errors, verify you're using --tensor-parallel-size 4, not 8.

Model Not Found

Kimi-K2.5 requires the nightly vLLM build. Use rocm/vllm-dev:nightly instead of the stable release.

Vision Requests Fail

Ensure VLLM_USE_TRITON_FLASH_ATTN=0 is set for the MoonViT encoder to work correctly.

0 Comments

Start the conversation

Be the first to comment and share your perspective with the community.

Kimi-K2.5 (1T)

3 min readUpdated 17 March 2026

Deploy Kimi-K2.5 (1 trillion parameters) on AMD Instinct GPUs.

Model Overview

Property	Value
Model ID	`moonshotai/Kimi-K2.5`
Architecture	MoE with MLA (384 experts, 8 selected per token)
Total Parameters	1T (1 trillion)
Active Parameters	32B per token
Context Length	256K tokens
Vision	MoonViT encoder (400M params)
Download Size	~400 GB (compressed-tensors)

Quick Start

Warning

Kimi-K2.5 requires vLLM nightly build (rocm/vllm-dev:nightly). The stable release does not yet support this model.

BASH
docker run --rm \
  --name vllm-kimi-k25 \
  --ipc=host \
  --network=host \
  --group-add video \
  --group-add render \
  --cap-add=SYS_PTRACE \
  --cap-add=CAP_SYS_ADMIN \
  --security-opt seccomp=unconfined \
  --device /dev/kfd \
  --device /dev/dri \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "VLLM_ROCM_USE_AITER=0" \
  --env "VLLM_USE_TRITON_FLASH_ATTN=0" \
  rocm/vllm-dev:nightly \
  vllm serve moonshotai/Kimi-K2.5 \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --trust-remote-code \
  --block-size 1
Powered by Vultr AgentBeta

Critical Configuration

Warning

These settings are mandatory for Kimi-K2.5 on AMD:

VLLM_ROCM_USE_AITER=0 - AITER disabled (MLA head count incompatibility with TP=4)
--tensor-parallel-size 4 - Not 8! Required for MLA attention head distribution (64/4=16 heads per GPU)
--block-size 1 - Required for MLA architecture
--trust-remote-code - Model uses custom code
VLLM_USE_TRITON_FLASH_ATTN=0 - Required for vision encoder

Note

Why AITER is Disabled?

Full Features Configuration

Enable chat, tool calling, and reasoning mode:

BASH
docker run --rm \
  --name vllm-kimi-k25 \
  --ipc=host \
  --network=host \
  --group-add video \
  --group-add render \
  --cap-add=SYS_PTRACE \
  --cap-add=CAP_SYS_ADMIN \
  --security-opt seccomp=unconfined \
  --device /dev/kfd \
  --device /dev/dri \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "VLLM_ROCM_USE_AITER=0" \
  --env "VLLM_USE_TRITON_FLASH_ATTN=0" \
  rocm/vllm-dev:nightly \
  vllm serve moonshotai/Kimi-K2.5 \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --trust-remote-code \
  --block-size 1 \
  --mm-encoder-tp-mode data \
  --tool-call-parser kimi_k2 \
  --reasoning-parser kimi_k2
Powered by Vultr AgentBeta

Additional Flags Explained

Flag	Purpose
`--mm-encoder-tp-mode data`	Data parallel mode for MoonViT vision encoder
`--tool-call-parser kimi_k2`	Parses tool/function calls in Kimi format
`--reasoning-parser kimi_k2`	Extracts reasoning blocks

Memory Usage

Configuration	Total Memory	Per GPU (TP=4)
Compressed-tensors (default)	~160 GB	~40 GB

Metric	Value
Load Time	~5-8 minutes (first run)

MI325X (256GB) easily fits the model with TP=4, leaving significant room for KV cache.

Performance (MI325X Verified)

Concurrent	Throughput	p99 Latency	Status
10	225 tok/s	77.83s	OK
50	583 tok/s	149.39s	OK
100	948 tok/s	183.35s	OK
200	950 tok/s	182.96s	OK
500	948 tok/s	183.23s	OK

Multi-run means (n=5).

Note

100% success rate across all concurrency levels (no failures)
Peak throughput: 952 tok/s at 1,000 concurrent requests
Saturation point: ~100 concurrent requests

See Kimi-K2.5 Stress Testing for detailed benchmark results including saturation testing up to 1,000 concurrent requests.

Test Endpoints

Chat Completion

BASH

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "moonshotai/Kimi-K2.5",
    "messages": [{"role": "user", "content": "Hello, who are you?"}]
  }'

Vision Request

BASH

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "moonshotai/Kimi-K2.5",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "What do you see in this image?"},
        {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}
      ]
    }]
  }'

Tool Calling

BASH

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "moonshotai/Kimi-K2.5",
    "messages": [{"role": "user", "content": "What is the weather in San Francisco?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get weather for a location",
        "parameters": {
          "type": "object",
          "properties": {"location": {"type": "string"}},
          "required": ["location"]
        }
      }
    }]
  }'

Comparison with Similar Models

Model	Total Params	Active Params	Architecture	Vision
Kimi-K2.5	1T	32B	MoE + MLA	Yes (MoonViT)
DeepSeek V3.2	685B	~37B	MoE + MLA	No
Qwen3-VL-235B	235B	22B	MoE + GQA	Yes

Known Limitations

Tensor Parallel Size Must Be 4

Unlike other models, Kimi-K2.5 requires TP=4:

Error: AITER MLA requires 16 heads per GPU

The MLA architecture has 64 attention heads that must be evenly distributed with 16 heads per GPU.

KV Cache Offloading Not Supported

Like DeepSeek V3.2, Kimi-K2.5 uses MLA which is incompatible with vLLM's KV cache offloading. The 256GB per MI325X provides sufficient capacity without offloading.

FP8 KV Cache Not Supported

Do not use --kv-cache-dtype fp8:

Error: MLA doesn't support fp8 kv_cache_dtype

The model uses its own compressed-tensors quantization format.

Troubleshooting

Crash on Startup

Ensure AITER_ENABLE_VSKIP=0 is set. If unset, it defaults to true which causes crashes on MI300X/MI325X.

TP Size Error

If you see attention head distribution errors, verify you're using --tensor-parallel-size 4, not 8.

Model Not Found

Kimi-K2.5 requires the nightly vLLM build. Use rocm/vllm-dev:nightly instead of the stable release.

Vision Requests Fail

Ensure VLLM_USE_TRITON_FLASH_ATTN=0 is set for the MoonViT encoder to work correctly.

0 Comments

Start the conversation

Be the first to comment and share your perspective with the community.