How to Build a vLLM Container Image
Introduction
vLLM is a fast inference and serving library for Large Language Models (LLMs). It offers several functionalities such as integration with popular hugging face models, optimized CUDA kernels for NVIDIA GPUs, tensor parallelism support and fast model execution.
This article explains how to build a vLLM container image using the Vultr Container Registry.
Prerequisites
Before you begin:
Deploy an instance using Vultr's GPU Marketplace App
Access the server using SSH.
Start the Docker service.
CONSOLE
Set Up the Server
Create a new directory to store your vLLM project files.
CONSOLESwitch to the directory.
CONSOLEClone the vLLM project repository using Git.
CONSOLEList files and verify that a new
vllmdirectory is available.CONSOLESwitch to the
vllmproject directory.CONSOLEList the directory files and verify that the necessary Dockerfile resources are available.
CONSOLEOutput:
The vLLM project directory includes the following Dockerfile resources:
Dockerfile: Contains the main vLLM library build context with support for NVIDIA GPU systems.Dockerfile.cpu: Contains the vLLM build context for CPU systems.Dockerfile.rocm: Contains the build context for AMD GPU systems.
Use the above resources in the next sections to build a CPU or GPU system container image.
Build a vLLM Container Image for CPU Systems
Follow the steps below to build a new vLLM container image using Dockerfile.cpu that contains the build context with all necessary packages and dependencies for CPU-based systems.
Build a new container image using
Dockerfile.cpuwith all files in the project working directory. Replacevllm-imagewith your desired image name.CONSOLEView all Docker images on the server and verify that your new vLLM image is available.
CONSOLEOutput:
Build a vLLM Container Image for GPU Systems
The vLLM project directory contains two Dockerfile resources for building container images for GPU-powered systems. Follow the steps below to use the main Dockerfile resource to build a new container image for GPU systems.
Build a new container image
vllm-gpu-imageusingDockerfilewith all files in the project directory.CONSOLEView all Docker images on the server and verify that the new
vllm-gpu-imageis available.CONSOLEOutput:
To run the vLLM GPU container image, verify that your target host runs the minimum or higher CUDA version referenced in the Dockerfile and use the
--gpus alloption when starting the container. Run the following command to verify the minimum CUDA version.CONSOLEOutput:
Upload the vLLM Container Image to the Vultr Container Registry
Open the Vultr Console.
Click Products and select Container Registry on the main navigation menu.
Click your target Vultr Container Registry to open the management panel and view the registry access credentials.
Copy the Registry URL value, Username, and API Key to use when accessing the registry.
Switch to your server terminal session and log in to your Vultr Container Registry. Replace
exampleregistry,exampleuser,registry-passwordwith your actual registry details.CONSOLETag the vLLM container image with your desired Vultr Container Registry tag. For example,
sjc.vultrcr.com/exampleregistry/vllm-gpu-image.CONSOLEView all Docker images on the server and verify that the new tagged image is available.
CONSOLEOutput:
Push the tagged image to your Vultr Container Registry.
CONSOLEOpen your Vultr Container Registry management panel and click Repositories on the top navigation bar to verify that the new repository is available.
0 Comments
Be the first to comment and share your perspective with the community.