The NVIDIA HGX B200 is a Blackwell-architecture data center GPU designed for large-scale AI inference and training. NVIDIA HGX B200 servers provide 8 GPUs per node connected via NVSwitch 5.0.
LLM inference is primarily memory-bandwidth-bound during the decode phase (generating tokens one at a time). The NVIDIA HGX B200's 8.0 TB/s memory bandwidth is among the highest available, directly translating to faster token generation for memory-bound workloads.
Key NVIDIA HGX B200 advantages for inference:
The NVIDIA HGX B200 uses an 8-GPU configuration with all-to-all NVSwitch connectivity:
All GPU pairs connect through NVSwitch 5.0, providing 1.8 TB/s bidirectional bandwidth between any two GPUs in the node. This is critical for tensor parallelism (weight sharding across GPUs) and for NVIDIA Dynamo's disaggregated serving (KV cache transfer between prefill and decode pools).
Verified topology (abbreviated):
Every GPU pair shows NV18 — 18 bonded NVLinks via NVSwitch 5.0. GPUs 0-3 are on NUMA node 0 (CPUs 0-63, 128-191) and GPUs 4-7 are on NUMA node 1 (CPUs 64-127, 192-255). The node has 14x Mellanox ConnectX NICs (mlx5_0 through mlx5_13) for network connectivity.
These specifications were verified on the NVIDIA HGX B200 instance used for this cookbook:
Each NVIDIA HGX B200 draws up to 1000W under load. At idle, GPUs draw approximately 140W each. During inference benchmarks, power consumption varies with load:
Power efficiency (tok/s per watt) is a useful metric for production deployments but is not the primary focus of this cookbook.
0 Comments
Be the first to comment and share your perspective with the community.