How to Deploy LiteLLM - Open-Source AI Gateway

LiteLLM is an open-source AI gateway that provides a unified, OpenAI-compatible API for over 100 large language model (LLM) providers, removing the need to integrate with each provider's Software Development Kit (SDK) and authentication scheme. In production, teams use LiteLLM to issue virtual keys with budget limits, track token usage and spend across providers, and configure centralized access control and routing.

This article explains how to deploy LiteLLM on a Linux server using Docker Compose with PostgreSQL for persistent storage, Prometheus for metrics collection, and Traefik as the reverse proxy for HTTPS access.

Prerequisites

Before you begin, you need to:

• Have access to a Linux-based server with at least 4 CPU cores and 8 GB of RAM as a non-root user with sudo privileges.
• Install Docker and Docker Compose.
• Create a DNS A record, such as litellm.example.com, pointing to your server's IP address.
• Have an API key from at least one supported LLM provider.

Set Up the Directory Structure, Configuration, and Environment Variables

LiteLLM requires a configuration file to define model providers and routing rules, a Prometheus configuration file for metrics scraping, and environment variables for secrets and database credentials. This section prepares all three before deploying the stack.

1. Create the project directory with subdirectories for persistent data.

   CONSOLE

   Copy

   $ mkdir -p ~/litellm/{letsencrypt,postgres,prometheus}

   Explain Code

   Powered by Vultr AgentBeta
   • letsencrypt: Stores SSL/TLS certificates.
   • postgres: Persists PostgreSQL database files.
   • prometheus: Persists Prometheus metrics data.

2. Navigate to the project directory.

   CONSOLE

   Copy

   $ cd ~/litellm

   Explain Code

   Powered by Vultr AgentBeta

3. Prometheus runs as UID 65534 inside the container. Set matching ownership on its host directory so the container can write to it.

   CONSOLE

   Copy

   $ sudo chown -R 65534:65534 prometheus

   Explain Code

   Powered by Vultr AgentBeta

4. Generate a master key and salt key for LiteLLM. Run this command twice to produce two separate values.

   CONSOLE

   Copy

   $ openssl rand -hex 32

   Explain Code

   Powered by Vultr AgentBeta

   Save both values for the LITELLM_MASTER_KEY and LITELLM_SALT_KEY fields in the next step.

5. Create the .env file to store credentials and secrets.

   CONSOLE

   Copy

   $ nano .env

   Explain Code

   Powered by Vultr AgentBeta

6. Add the following values:

   INI

   Copy

   DOMAIN=litellm.example.com
   LETSENCRYPT_EMAIL=admin@example.com
   LITELLM_MASTER_KEY=sk-YOUR_MASTER_KEY
   LITELLM_SALT_KEY=sk-YOUR_SALT_KEY
   LLM_PROVIDER_API_KEY=YOUR_LLM_PROVIDER_API_KEY
   POSTGRES_PASSWORD=STRONG_DATABASE_PASSWORD
   DATABASE_URL=postgresql://llmproxy:${POSTGRES_PASSWORD}@db:5432/litellm
   UI_USERNAME=admin
   UI_PASSWORD=YOUR_UI_PASSWORD

   Explain Code

   Powered by Vultr AgentBeta

   Replace the placeholders with your own values:

   • litellm.example.com: Domain pointing to your server's IP address.
   • admin@example.com: Email address for Let's Encrypt certificate notifications.
   • sk-YOUR_MASTER_KEY: Admin key used to authenticate with the LiteLLM API. Replace YOUR_MASTER_KEY with the first value you generated, and keep the sk- prefix.
   • sk-YOUR_SALT_KEY: Key used to encrypt provider credentials stored in PostgreSQL. Replace YOUR_SALT_KEY with the second value you generated, and keep the sk- prefix. This value cannot be changed after the first model is added.
   • YOUR_LLM_PROVIDER_API_KEY: API key for the LLM provider configured in config.yaml.
   • STRONG_DATABASE_PASSWORD: PostgreSQL password used by both the database container and the connection string. Replace with a strong password of your choice.
   • admin: Username for the LiteLLM dashboard. Change to your preferred username.
   • YOUR_UI_PASSWORD: Password for the LiteLLM dashboard. Use a strong password distinct from your master key.

   Save and close the file.

7. Create the LiteLLM configuration file.

   CONSOLE

   Copy

   $ nano config.yaml

   Explain Code

   Powered by Vultr AgentBeta

8. Add the following contents:

   YAML

   Copy

   model_list:
     - model_name: my-model
       litellm_params:
         model: provider/model
         api_key: os.environ/LLM_PROVIDER_API_KEY
   
   litellm_settings:
     callbacks:
       - prometheus
     require_auth_for_metrics_endpoint: true
   
   general_settings:
     master_key: os.environ/LITELLM_MASTER_KEY

   Explain Code

   Powered by Vultr AgentBeta

   Replace my-model with a name of your choice to identify this model within LiteLLM. Replace provider/model with the LiteLLM provider prefix followed by the model identifier, in the format provider/model-name. For example, to expose Anthropic's Claude Haiku, use anthropic/claude-haiku-4-5. The os.environ/ prefix tells LiteLLM to read the value from an environment variable at runtime rather than hardcoding it in the file. For the provider prefix and exact model identifiers for each provider, see the LiteLLM providers documentation.

   The litellm_settings block enables Prometheus metrics collection via the callbacks field. Setting require_auth_for_metrics_endpoint to true restricts the /metrics endpoint to authenticated requests only, so only Prometheus, which sends the master key as a Bearer token, can scrape metrics successfully.

   The general_settings block defines the master key LiteLLM uses to authenticate admin API requests and virtual key management operations. Reading it from an environment variable keeps the actual key value out of the configuration file.

   Save and close the file.

9. Create the Prometheus configuration file.

   CONSOLE

   Copy

   $ nano prometheus.yml

   Explain Code

   Powered by Vultr AgentBeta

10. Add the following contents:

    YAML

    Copy

    global:
      scrape_interval: 15s
      evaluation_interval: 15s
    
    scrape_configs:
      - job_name: "litellm"
        static_configs:
          - targets: ["litellm:4000"]
        bearer_token: "sk-YOUR_MASTER_KEY"

    Explain Code

    Powered by Vultr AgentBeta

    Replace sk-YOUR_MASTER_KEY with the value of LITELLM_MASTER_KEY from your .env file. Prometheus uses this token to authenticate its scrape requests against the protected /metrics endpoint.

    Save and close the file.

Deploy with Docker Compose

The deployment stack runs LiteLLM behind Traefik, which handles Transport Layer Security (TLS) termination and automatic certificate provisioning through Let's Encrypt. PostgreSQL provides persistent storage for virtual keys, spend data, and usage logs. Prometheus collects gateway metrics by scraping the LiteLLM /metrics endpoint every 15 seconds over the internal Docker network.

1. Add your user account to the Docker group.

   CONSOLE

   Copy

   $ sudo usermod -aG docker $USER

   Explain Code

   Powered by Vultr AgentBeta

2. Apply the new group membership.

   CONSOLE

   Copy

   $ newgrp docker

   Explain Code

   Powered by Vultr AgentBeta

3. Create the Docker Compose manifest file.

   CONSOLE

   Copy

   $ nano docker-compose.yaml

   Explain Code

   Powered by Vultr AgentBeta

4. Add the following contents:

   YAML

   Copy

   services:
     traefik:
       image: traefik:v3.6
       container_name: traefik
       restart: always
       command:
         - "--providers.docker=true"
         - "--providers.docker.exposedbydefault=false"
         - "--entrypoints.web.address=:80"
         - "--entrypoints.websecure.address=:443"
         - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
         - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
         - "--certificatesresolvers.le.acme.httpchallenge=true"
         - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"
         - "--certificatesresolvers.le.acme.email=${LETSENCRYPT_EMAIL}"
         - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
       ports:
         - "80:80"
         - "443:443"
       volumes:
         - /var/run/docker.sock:/var/run/docker.sock:ro
         - ./letsencrypt:/letsencrypt
   
     litellm:
       image: docker.litellm.ai/berriai/litellm:v1.89.1
       container_name: litellm
       restart: always
       volumes:
         - ./config.yaml:/app/config.yaml
       command:
         - "--config=/app/config.yaml"
       environment:
         DATABASE_URL: ${DATABASE_URL}
         STORE_MODEL_IN_DB: "True"
       env_file:
         - .env
       expose:
         - "4000"
       depends_on:
         db:
           condition: service_healthy
       healthcheck:
         test:
           - CMD-SHELL
           - python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"
         interval: 30s
         timeout: 10s
         retries: 3
         start_period: 40s
       labels:
         - "traefik.enable=true"
         - "traefik.http.routers.litellm.rule=Host(`${DOMAIN}`)"
         - "traefik.http.routers.litellm.entrypoints=websecure"
         - "traefik.http.routers.litellm.tls=true"
         - "traefik.http.routers.litellm.tls.certresolver=le"
         - "traefik.http.services.litellm.loadbalancer.server.port=4000"
   
     db:
       image: postgres:16
       container_name: litellm_db
       restart: always
       environment:
         POSTGRES_DB: litellm
         POSTGRES_USER: llmproxy
         POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
       volumes:
         - ./postgres:/var/lib/postgresql/data
       expose:
         - "5432"
       healthcheck:
         test: ["CMD-SHELL", "pg_isready -d litellm -U llmproxy"]
         interval: 1s
         timeout: 5s
         retries: 10
   
     prometheus:
       image: prom/prometheus
       container_name: litellm_prometheus
       restart: always
       volumes:
         - ./prometheus:/prometheus
         - ./prometheus.yml:/etc/prometheus/prometheus.yml
       expose:
         - "9090"
       command:
         - "--config.file=/etc/prometheus/prometheus.yml"
         - "--storage.tsdb.path=/prometheus"
         - "--storage.tsdb.retention.time=15d"
   

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   This Docker Compose configuration deploys four services behind a single HTTPS endpoint:

   • traefik: Acts as the reverse proxy and TLS terminator, listening on ports 80 and 443 and automatically redirecting HTTP traffic to HTTPS using Let's Encrypt.
   • litellm: Runs the LiteLLM proxy image pinned to a specific release tag for reproducible deployments, mounting config.yaml for model configuration and waiting for PostgreSQL to become healthy before starting. Traefik routes HTTPS traffic for ${DOMAIN} to port 4000.
   • db: Runs PostgreSQL 16 as the persistent backend for virtual keys, spend data, and usage logs, using a named volume so data survives container restarts.
   • prometheus: Scrapes LiteLLM metrics from port 4000 every 15 seconds and stores time-series data in a named volume with a 15-day retention window.

   The volumes block declares the named volumes used by db and prometheus, ensuring their data persists independently of the container lifecycle and survives container removal or recreation.

Verify the Docker Image with Cosign

LiteLLM Docker images are signed with Cosign. Verifying the image signature before deployment confirms the image has not been tampered with since it was published by the LiteLLM team.

1. Download and install the Cosign binary.

   CONSOLE

   Copy

   $ curl -O -L "https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64"

   Explain Code

   Powered by Vultr AgentBeta

2. Move the binary into your system path.

   CONSOLE

   Copy

   $ sudo mv cosign-linux-amd64 /usr/local/bin/cosign

   Explain Code

   Powered by Vultr AgentBeta

3. Make the binary executable.

   CONSOLE

   Copy

   $ sudo chmod +x /usr/local/bin/cosign

   Explain Code

   Powered by Vultr AgentBeta

4. Verify the installation.

   CONSOLE

   Copy

   $ cosign version

   Explain Code

   Powered by Vultr AgentBeta

   The output displays the installed Cosign version, confirming the binary is available in your system path.

5. Verify the LiteLLM image signature using the pinned public key. This checks the same release tag deployed in the Docker Compose file in the next step. LiteLLM publishes signatures for the ghcr.io registry specifically, so this confirms the v1.89.1 release itself, not the docker.litellm.ai mirror byte-for-byte.

   CONSOLE

   Copy

   $ cosign verify \
       --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
       ghcr.io/berriai/litellm:v1.89.1

   Explain Code

   Powered by Vultr AgentBeta

   A successful verification outputs a JSON payload confirming the image was signed with the LiteLLM public key. If verification fails, confirm you copied the key URL and image tag, then run the command again.

Start the Stack

With the configuration files and images in place, bring up the full deployment.

1. Start the services.

   CONSOLE

   Copy

   $ docker compose up -d

   Explain Code

   Powered by Vultr AgentBeta

2. Verify all containers are running.

   CONSOLE

   Copy

   $ docker compose ps

   Explain Code

   Powered by Vultr AgentBeta

   Verify that all four containers show an Up status, with litellm and db marked (healthy) and traefik listing ports 80 and 443 under PORTS.

   Note

   For more information on managing a Docker Compose stack, see the How to Use Docker Compose article.

Access and Configure LiteLLM

LiteLLM exposes a web-based admin dashboard at the /ui path of your configured domain. The dashboard provides visibility into model configuration, virtual keys, usage, and spend tracking.

1. Open your web browser and navigate to https://litellm.example.com/ui, replacing litellm.example.com with your configured domain.

   

2. Log in using the credentials from your .env file. Enter the value of UI_USERNAME in the Username field and the value of UI_PASSWORD in the Password field, then click Login. Verify that the dashboard loads and displays the main navigation panels.

   

3. Click Models + Endpoints in the left sidebar to verify the configured model providers. The model defined in config.yaml appears in the list with its alias and underlying provider model.

   

   The Model Management page lists all configured model aliases and their underlying providers. The left sidebar also provides access to Virtual Keys for managing scoped access credentials, Usage for per-model and per-key request tracking, and Logs for spend and cost breakdowns by model, key, and team.

Test the Gateway and Create a Virtual Key

LiteLLM exposes an OpenAI-compatible API, so any application built for the OpenAI SDK works with LiteLLM by pointing base_url at the gateway. Virtual keys provide scoped, credential-isolated access without exposing the master key. Each key carries its own model access list, budget limit, and rate limit.

1. Install the Python virtual environment package.

   CONSOLE

   Copy

   $ sudo apt install -y python3-venv

   Explain Code

   Powered by Vultr AgentBeta

2. Create a Python virtual environment.

   CONSOLE

   Copy

   $ python3 -m venv litellm-env

   Explain Code

   Powered by Vultr AgentBeta

3. Activate the virtual environment.

   CONSOLE

   Copy

   $ source litellm-env/bin/activate

   Explain Code

   Powered by Vultr AgentBeta

4. Install the OpenAI Python SDK.

   CONSOLE

   Copy

   $ pip install openai

   Explain Code

   Powered by Vultr AgentBeta

5. Export your master key as an environment variable. Replace sk-YOUR_MASTER_KEY with the LITELLM_MASTER_KEY value from your .env file.

   CONSOLE

   Copy

   $ export LITELLM_MASTER_KEY="sk-YOUR_MASTER_KEY"

   Explain Code

   Powered by Vultr AgentBeta

6. Export your gateway domain as an environment variable. Replace litellm.example.com with your configured domain.

   CONSOLE

   Copy

   $ export LITELLM_DOMAIN="litellm.example.com"

   Explain Code

   Powered by Vultr AgentBeta

7. Create the test script.

   CONSOLE

   Copy

   $ nano test_litellm.py

   Explain Code

   Powered by Vultr AgentBeta

8. Add the following contents, replacing my-model with the model name you set in config.yaml:

   PYTHON

   Copy

   import os
   from openai import OpenAI
   
   client = OpenAI(
       api_key=os.environ["LITELLM_MASTER_KEY"],
       base_url=f"https://{os.environ['LITELLM_DOMAIN']}",
   )
   
   response = client.chat.completions.create(
       model="my-model",
       messages=[{"role": "user", "content": "What is an AI gateway?"}],
   )
   
   print(response.choices[0].message.content)

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

9. Run the script.

   CONSOLE

   Copy

   $ python3 test_litellm.py

   Explain Code

   Powered by Vultr AgentBeta

   The script returns a response from the configured LLM provider, confirming the gateway is routing requests correctly.

10. Create a virtual key using the LiteLLM API. Replace my-model with the model name you set in config.yaml.

    CONSOLE

    Copy

    $ curl -X POST "https://${LITELLM_DOMAIN}/key/generate" \
        -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
        -H "Content-Type: application/json" \
        -d '{"key_alias": "test-app-key", "models": ["my-model"], "max_budget": 5.00}' \
        | python3 -c "import sys, json; print(json.load(sys.stdin)['key'])"

    Explain Code

    Powered by Vultr AgentBeta

    The command prints the generated virtual key, similar to sk-M9M0_aUDLi7AuhXtOAw6uw. Virtual keys let you issue scoped credentials to individual users or applications without exposing the master key. Each key can have its own model access list, budget limit, and rate limit.

11. Export the virtual key returned in the previous step. Replace sk-your-virtual-key with the full key string printed by the previous command.

    CONSOLE

    Copy

    $ export LITELLM_VIRTUAL_KEY="sk-your-virtual-key"

    Explain Code

    Powered by Vultr AgentBeta

12. Send a request using the virtual key. Replace my-model with the model name you set in config.yaml.

    CONSOLE

    Copy

    $ curl -X POST "https://${LITELLM_DOMAIN}/v1/chat/completions" \
        -H "Authorization: Bearer ${LITELLM_VIRTUAL_KEY}" \
        -H "Content-Type: application/json" \
        -d '{
          "model": "my-model",
          "messages": [{"role": "user", "content": "Hello from a virtual key!"}]
        }'

    Explain Code

    Powered by Vultr AgentBeta

    The gateway authenticates the virtual key, routes the request to the configured provider, and returns the model response.

13. Return to the dashboard at https://litellm.example.com/ui and click Logs in the left sidebar.

    

    Each entry shows the model alias, key alias, token count, and estimated cost for the requests sent through both the master key and the virtual key.

Conclusion

You have deployed LiteLLM on a Linux server using Docker Compose with PostgreSQL for persistent storage, Prometheus for authenticated metrics collection, and Traefik for automatic HTTPS through Let's Encrypt. The deployment provides a fully functional, OpenAI-compatible AI gateway accessible over a custom domain, with virtual key management, spend tracking, and provider routing configured. For more information, refer to the official LiteLLM documentation.