How to Build a Llama.cpp Container Image
Introduction
Llama.cpp is a high-performance inference platform designed for Large Language Models (LLMs) like Llama, Falcon, and Mistral. It provides a streamlined development environment compatible with both CPU and GPU systems. This article explains how to set up and run Llama.cpp in Docker using the Vultr Container Registry.
Prerequisites
Before you begin:
Deploy an instance using Vultr's GPU Marketplace App
Access the server using SSH.
Start the Docker service.
CONSOLEAdd the non-root user to the Docker group. For example,
linuxuser.CONSOLESwitch to the user:
CONSOLE
Set Up the Server
Create a new directory to store your Llama.cpp project files.
CONSOLESwitch to the directory.
CONSOLEClone the Llama.cpp project repository using Git.
CONSOLEList files and verify that a new
llama.cppdirectory is available.CONSOLEOutput:
Switch to the
llama.cppproject directory.CONSOLEList all hidden files and verify that a new
.devopsdirectory is available.CONSOLEOutput:
The
.devopsdirectory contains the following Dockerfile resources:main.Dockerfile: Contains the build context for CPU systems.main-cuda.Dockerfile: Contains the build context for GPU systems.
Use the above resources in the next sections to build a CPU or GPU system container image.
Build a Llama.cpp Container Image for CPU Systems
Follow the steps below to build a new Llama.cpp container image using main.Dockerfile that contains the required build context and installs all necessary dependency packages for CPU-based systems.
Copy the
main.Dockerfilefrom the.devopsdirectory to the mainllama.cppproject directory.CONSOLEBuild a new container image using
main.Dockerfilewith all files in the project directory. Replacellama-imagewith your desired image name.CONSOLEView all Docker images on the server and verify that a new
llama-imageis available.CONSOLEOutput:
Build a Llama.cpp Container Image for GPU Systems
The Llama.cpp main-cuda.Dockerfile resource contains the build context for NVIDIA GPU systems that run the latest CUDA driver packages. Follow the steps below to build a Llama container image compatible with GPU systems.
Copy
main-cuda.Dockerfileto the Llama.cpp project directory.CONSOLEBuild a new container image
llama-gpu-imageusing themain-cuda.Dockerfilewith all files in the working project directory.CONSOLEView all Docker images on the server and verify that the new
llama-gpu-imageis available.CONSOLEOutput:
To run the Llama.cpp GPU container image, verify that your target host includes the minimum or higher CUDA version referenced by the
ARG CUDA_VERSION=directive withinmain-cuda.Dockerfile. Run the following command to verify the target CUDA version.CONSOLEOutput:
Upload the Llama.cpp Container Image to the Vultr Container Registry
Open the Vultr Console.
Click Products and select Container Registry on the main navigation menu.
Click your target Vultr Container Registry to open the management panel and view the registry access credentials.
Copy the Registry URL value, Username, and API Key to use when accessing the registry.
Switch to your server terminal session and log in to your Vultr Container Registry. Replace
exampleregistry,exampleuser,registry-passwordwith your actual registry details.CONSOLETag the Llama.cpp container image with your desired Vultr Container Registry tag. For example,
sjc.vultrcr.com/exampleregistry/llama-gpu-image.CONSOLEView all Docker images on the server and verify that the new tagged image is available.
CONSOLEOutput:
Push the tagged image to your Vultr Container Registry repository.
CONSOLEOpen your Vultr Container Registry management panel and click Repositories on the top navigation bar to verify that the new repository is available.
0 Comments
Be the first to comment and share your perspective with the community.