How to Build a Llama.cpp Container Image

Introduction

Llama.cpp is a high-performance inference platform designed for Large Language Models (LLMs) like Llama, Falcon, and Mistral. It provides a streamlined development environment compatible with both CPU and GPU systems. This article explains how to set up and run Llama.cpp in Docker using the Vultr Container Registry.

Prerequisites

Before you begin:

• Deploy an instance using Vultr's GPU Marketplace App

• Access the server using SSH.

• Start the Docker service.

  console Copy

  $ sudo systemctl start docker
  

  Explain Code

• Add the non-root user to the Docker group. For example, linuxuser.

  console Copy

  $ sudo usermod -aG docker linuxuser
  

  Explain Code

• Switch to the user:

  console Copy

  $ su - linuxuser
  

  Explain Code

Set Up the Server

1. Create a new directory to store your Llama.cpp project files.

   console Copy

   $ mkdir llama-project
   

   Explain Code

2. Switch to the directory.

   console Copy

   $ cd llama-project
   

   Explain Code

3. Clone the Llama.cpp project repository using Git.

   console Copy

   $ git clone https://github.com/ggerganov/llama.cpp/
   

   Explain Code

4. List files and verify that a new llama.cpp directory is available.

   console Copy

   $ ls
   

   Explain Code

   Output:

   llama.cpp

5. Switch to the llama.cpp project directory.

   console Copy

   $ cd llama.cpp
   

   Explain Code

6. List all hidden files and verify that a new .devops directory is available.

   console Copy

   $ ls -a
   

   Explain Code

   Output:

   ..                 cmake           convert.py                     flake.lock     ggml-common.h        ggml-metal.h      ggml-quantsh     .git                         llama.cpp        
   CMakeLists.txt  .devops    

   The .devops directory contains the following Dockerfile resources:

   • main.Dockerfile: Contains the build context for CPU systems.
   • main-cuda.Dockerfile: Contains the build context for GPU systems.

   Use the above resources in the next sections to build a CPU or GPU system container image.

Build a Llama.cpp Container Image for CPU Systems

Follow the steps below to build a new Llama.cpp container image using main.Dockerfile that contains the required build context and installs all necessary dependency packages for CPU-based systems.

1. Copy the main.Dockerfile from the .devops directory to the main llama.cpp project directory.

   console Copy

   $ cp .devops/main.Dockerfile .
   

   Explain Code

2. Build a new container image using main.Dockerfile with all files in the project directory. Replace llama-image with your desired image name.

   console Copy

   $ docker build -f main.Dockerfile -t llama-image .
   

   Explain Code

3. View all Docker images on the server and verify that a new llama-image is available.

   console Copy

   $ docker images
   

   Explain Code

   Output:

   REPOSITORY                                  TAG       IMAGE ID       CREATED              SIZE
   llama-image                                 latest    f4fcb571a2f7   About a minute ago   80.1MB

Build a Llama.cpp Container Image for GPU Systems

The Llama.cpp main-cuda.Dockerfile resource contains the build context for NVIDIA GPU systems that run the latest CUDA driver packages. Follow the steps below to build a Llama container image compatible with GPU systems.

1. Copy main-cuda.Dockerfile to the Llama.cpp project directory.

   console Copy

   $ cp .devops/main-cuda.Dockerfile .
   

   Explain Code

2. Build a new container image llama-gpu-image using the main-cuda.Dockerfile with all files in the working project directory.

   console Copy

   $ docker build -f main-cuda.Dockerfile -t llama-gpu-image .
   

   Explain Code

3. View all Docker images on the server and verify that the new llama-gpu-image is available.

   console Copy

   $ docker images
   

   Explain Code

   Output:

   REPOSITORY                                  TAG       IMAGE ID       CREATED          SIZE
   llama-gpu-image                             latest    40421f1469cf   2 minutes ago    2.08GB

   To run the Llama.cpp GPU container image, verify that your target host includes the minimum or higher CUDA version referenced by the ARG CUDA_VERSION= directive within main-cuda.Dockerfile. Run the following command to verify the target CUDA version.

   console Copy

   $ cat main-cuda.Dockerfile | grep CUDA_VERSION=
   

   Explain Code

   Output:

   ARG CUDA_VERSION=12.3.1

Upload the Llama.cpp Container Image to the Vultr Container Registry

1. Open the Vultr Customer Portal.

2. Click Products and select Container Registry on the main navigation menu.

   

3. Click your target Vultr Container Registry to open the management panel and view the registry access credentials.

4. Copy the Registry URL value, Username, and API Key to use when accessing the registry.

   

5. Switch to your server terminal session and log in to your Vultr Container Registry. Replace exampleregistry, exampleuser, registry-password with your actual registry details.

   console Copy

   $ docker login https://sjc.vultrcr.com/exampleregistry -u exampleuser -p registry-password
   

   Explain Code

6. Tag the Llama.cpp container image with your desired Vultr Container Registry tag. For example, sjc.vultrcr.com/exampleregistry/llama-gpu-image.

   console Copy

   $ docker tag llama-gpu-image sjc.vultrcr.com/exampleregistry/llama-gpu-image
   

   Explain Code

7. View all Docker images on the server and verify that the new tagged image is available.

   console Copy

   $ docker images
   

   Explain Code

   Output:

   REPOSITORY                                  TAG       IMAGE ID       CREATED          SIZE
   sjc.vultrcr.com/exampleregistry/llama-gpu-image     latest    40421f1469cf   3 minutes ago    2.08GB
   llama-gpu-image                             latest    40421f1469cf   3 minutes ago    2.08GB

8. Push the tagged image to your Vultr Container Registry repository.

   console Copy

   $ docker push sjc.vultrcr.com/exampleregistry/llama-gpu-image
   

   Explain Code

9. Open your Vultr Container Registry management panel and click Repositories on the top navigation bar to verify that the new repository is available.