GPU Power and Thermal Monitoring

Power Efficiency

Per-GPU Power Consumption

GPUs 0-3 (active for TP=4 inference) draw significantly more power than GPUs 4-7 (idle). The active GPUs average 640-680W each, well within the MI325X TDP of 750W.

Per-GPU Temperature

Junction and HBM3e memory temperatures track closely. Active GPUs (0-3) reach 60-80°C junction, while idle GPUs stay below 40°C.

Power Draw Over Time

Total system power fluctuates as benchmark phases transition between concurrency levels. Peaks correspond to high-concurrency stress phases.

Temperature Over Time

Temperature ramps correlate with sustained compute phases. The MI325X's cooling solution keeps junction temperatures well below throttling thresholds (100°C+).

Monitoring Summary

Key Observations

MI325X Thermal Headroom

• Peak junction temperature: ~76°C (well below throttle point)
• Active GPU mean power: ~660W (88% of 750W TDP)
• Idle GPU baseline: ~120W per GPU

Power Efficiency

With Kimi-K2.5 achieving ~952 tok/s peak throughput using 4 active GPUs at ~660W each, the effective power efficiency is approximately 0.36 tok/s per watt for the active GPUs. The remaining 4 idle GPUs draw baseline power but provide no inference throughput, which is a consideration for TP=4 models on 8-GPU systems.

Monitoring Methodology

All monitoring data was collected using:

rocm-smi --showpower --showtemp --showuse

at 1-second intervals as a background process during benchmark execution. This introduces negligible overhead (<0.1% CPU).