Params: 600B
Step 5 Preview is a multimodal sparse Mixture-of-Experts model built for agentic software engineering, long-horizon tool use, and finance-focused knowledge work. The model carries 600B total parameters and 27B active per token across 92 layers, with a 4,096 hidden size, 64 attention heads, and 4 KV heads. Sliding-window and full-attention layers alternate in a 3:1 pattern, while each MoE layer activates 8 of 352 routed experts plus a shared expert. Supporting a 1M-token context, it adds a Perception Encoder for image input and three Multi-Token Prediction layers.
Text GenerationInstruction Following+8