Tencent Hy4 Preview

Hy4 preview is a flagship Mixture-of-Experts model designed for advanced reasoning, coding, agentic workflows, and efficient long-context processing. It features 770B total parameters with 49B activated per token, using 78 layers with 6,144 hidden size and 64 attention heads. The architecture combines Gated DeepSeek Sparse Attention with IndexCache, 256 routed experts and 1 shared expert, activating 8 routed experts per token, while iHC expands residual information flow across 4 streams. It includes a native Multi-Token Prediction layer for speculative decoding and supports up to a 1M-token context window.