How to Deploy LocalAI – Self-Hosted AI Model Management Platform

LocalAI is an open-source platform that enables running Large Language Models (LLMs) and other AI models locally without relying on external APIs. It provides an OpenAI-compatible API, allowing integration of AI capabilities into existing applications while maintaining full control over infrastructure, data privacy, and operational costs.

This article explains how to deploy LocalAI on a Linux server using Docker Compose and secure the API endpoint using a Traefik reverse proxy with HTTPS.

Prerequisites

Before you begin, you need to:

• Have access to a Linux-based server as a non-root user with sudo privileges.
• Install Docker and Docker Compose.
• Create a DNS A record, such as localai.example.com, pointing to your server's IP address.

Set Up the Directory Structure and Environment Variables

LocalAI stores model files and cache data on disk and relies on environment variables for domain and TLS configuration. This section prepares the required directory structure for Docker Compose.

1. Create the project folders.

   CONSOLE

   Copy

   $ mkdir -p ~/localai/{models,cache}

   Explain Code

   Powered by Vultr AgentBeta
   • models: Stores downloaded AI model files used by LocalAI for inference.
   • cache: Persists temporary data and intermediate files across container restarts.

2. Navigate to the project directory.

   CONSOLE

   Copy

   $ cd ~/localai

   Explain Code

   Powered by Vultr AgentBeta

3. Create an .env file to store your domain and certificate configurations.

   CONSOLE

   Copy

   $ nano .env

   Explain Code

   Powered by Vultr AgentBeta

4. Add the following values:

   INI

   Copy

   DOMAIN=localai.example.com
   LETSENCRYPT_EMAIL=admin@example.com

   Explain Code

   Powered by Vultr AgentBeta

   Replace the placeholders with your own values:

   • localai.example.com: Domain for the LocalAI API endpoint.
   • admin@example.com: Email address for Let's Encrypt notifications.

   Save and close the file.

Deploy with Docker Compose

The deployment stack exposes the LocalAI API securely over HTTPS on the configured domain, with Traefik handling TLS termination and automatic certificate provisioning through Let's Encrypt.

1. Add your user account to the Docker user group.

   CONSOLE

   Copy

   $ sudo usermod -aG docker $USER

   Explain Code

   Powered by Vultr AgentBeta

2. Apply the new group membership.

   CONSOLE

   Copy

   $ newgrp docker

   Explain Code

   Powered by Vultr AgentBeta

3. Create the Docker Compose manifest file.

   CONSOLE

   Copy

   $ nano docker-compose.yaml

   Explain Code

   Powered by Vultr AgentBeta

4. Add the following contents:

   YAML

   Copy

   services:
     traefik:
       image: traefik:v3.6
       container_name: traefik
       restart: unless-stopped
       environment:
         DOCKER_API_VERSION: "1.44"
       command:
         - "--providers.docker=true"
         - "--providers.docker.exposedbydefault=false"
         - "--entrypoints.web.address=:80"
         - "--entrypoints.websecure.address=:443"
         - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
         - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
         - "--certificatesresolvers.le.acme.httpchallenge=true"
         - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"
         - "--certificatesresolvers.le.acme.email=${LETSENCRYPT_EMAIL}"
         - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
       ports:
         - "80:80"
         - "443:443"
       volumes:
         - /var/run/docker.sock:/var/run/docker.sock:ro
         - ./letsencrypt:/letsencrypt
   
     localai:
       image: localai/localai:latest-aio-cpu
       container_name: localai
       restart: unless-stopped
       volumes:
         - ./models:/models:cached
         - ./cache:/cache:cached
       healthcheck:
         test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
         interval: 1m
         timeout: 20s
         retries: 5
       labels:
         - "traefik.enable=true"
         - "traefik.http.routers.localai.rule=Host(`${DOMAIN}`)"
         - "traefik.http.routers.localai.entrypoints=websecure"
         - "traefik.http.routers.localai.tls=true"
         - "traefik.http.routers.localai.tls.certresolver=le"
         - "traefik.http.services.localai.loadbalancer.server.port=8080"

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   Note

   The configuration above uses the latest-aio-cpu image for minimal CPU-based serving, suitable for any CPU-only environment. If GPU acceleration is available, replace the container image with the appropriate GPU-enabled variant and add the required runtime values for your GPU. For details, see the LocalAI GPU acceleration documentation.

   This Docker Compose configuration deploys LocalAI and exposes it securely over HTTPS using Traefik. The setup routes all traffic through a single domain while keeping TLS management centralized in Traefik.

   localai service

   • Runs the official localai/localai image with the latest-aio-cpu tag, which includes all built-in models optimized for CPU inference.
   • Exposes an internal port 8080 for the OpenAI-compatible API endpoint.
   • Uses two persistent volume mounts:
     • ./models:/models: Stores downloaded AI model files across container restarts.
     • ./cache:/cache: Persists temporary data and intermediate files. The :cached flag optimizes file system performance for read-heavy workloads.
   • Includes a health check that monitors the /readyz endpoint, allowing Docker to track service availability.
   • Traefik discovers LocalAI automatically using Docker labels. A single HTTPS router is defined:
     • Routes traffic for ${DOMAIN} to LocalAI's API on port 8080.
     • Listens on the websecure entrypoint.
     • Uses Let's Encrypt (le) to automatically obtain and renew TLS certificates.

   traefik service

   • Acts as the reverse proxy and TLS terminator for LocalAI.
   • Listens on ports 80 and 443 on the host.
   • Automatically redirects HTTP traffic to HTTPS.
   • Stores Let's Encrypt ACME certificates in the ./letsencrypt directory.
   • Uses the Docker socket (read-only) to discover services and apply routing rules dynamically.

5. Set read permissions on the models directory so all processes inside the container can access downloaded model files.

   CONSOLE

   Copy

   $ sudo chmod -R 755 ~/localai/models

   Explain Code

   Powered by Vultr AgentBeta

   When downloading models through the dashboard or API, the container writes model files as root. The above permissions ensures the directory and its contents remain readable and executable by all processes, while restricting write access to the owner.

6. Start the services.

   CONSOLE

   Copy

   $ docker compose up -d

   Explain Code

   Powered by Vultr AgentBeta

7. Verify the containers are running.

   CONSOLE

   Copy

   $ docker compose ps

   Explain Code

   Powered by Vultr AgentBeta

   The output displays two running containers with Traefik listening on ports 80 and 443.

   Note

   For more information on managing a Docker Compose stack, see the How to Use Docker Compose article.

Access and Verify LocalAI

LocalAI exposes a readiness endpoint to confirm the API server is operational. Traefik routes all HTTPS traffic on the configured domain to the LocalAI container.

1. Check the readiness endpoint. Replace localai.example.com with your configured domain name.

   CONSOLE

   Copy

   $ curl -i https://localai.example.com/readyz

   Explain Code

   Powered by Vultr AgentBeta

   The 200 OK response confirms that Traefik is routing traffic correctly and the LocalAI service is ready to accept requests.

2. Open your web browser and navigate to https://localai.example.com, replacing localai.example.com with your configured domain name.

   

   The LocalAI dashboard provides access to the model gallery, installed models, and the inference interface.

Load a Model and Run a Test Inference

The LocalAI All-in-One (AIO) image includes pre-loaded models. Verify model availability and run a test inference to confirm the pipeline is operational.

1. List available models. Replace localai.example.com with your configured domain name.

   CONSOLE

   Copy

   $ curl https://localai.example.com/v1/models

   Explain Code

   Powered by Vultr AgentBeta

   The output lists all models loaded and available to the LocalAI instance.

2. Run a test inference using the chat completions endpoint. Replace localai.example.com with your configured domain name.

   CONSOLE

   Copy

   $ curl -X POST https://localai.example.com/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "gpt-4",
       "messages": [
         {"role": "user", "content": "Explain what LocalAI does in one sentence."}
       ],
       "max_tokens": 60
     }'

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   ...
       "message": {
         "role": "assistant",
         "content": "LocalAI refers to artificial intelligence algorithms and techniques that enable machines to process and analyze data and information within a specific, localized environment or system, rather than relying on data from a larger, centralized source."
       }
   ...

   The response confirms that a model is loaded, the LocalAI API is operational, and the server processes inference requests successfully over HTTPS.

Conclusion

You have successfully deployed LocalAI on a Linux server using Docker Compose with Traefik providing automatic HTTPS through Let's Encrypt. The deployment provides a fully functional, OpenAI-compatible API accessible securely over a custom domain with persistent storage for models and cache data. For more information, refer to the official LocalAI documentation.