How to Deploy Kestra – Open-Source Workflow Orchestration Platform

Kestra is an open-source workflow orchestration platform used to define, schedule, and monitor automated workflows. It supports declarative YAML-based configurations and provides a web interface for managing workflow executions, logs, and metadata. Kestra is used for data pipelines, infrastructure automation, and event-driven processing.

This article explains how to deploy Kestra on a Linux server using Docker Compose with a PostgreSQL backend. It covers directory setup, environment variable configuration, service deployment, and demonstrates workflow execution through the web interface.

Prerequisites

Before you begin, you need to:

• Have access to a Linux-based server (with at least 2 CPU cores and 4 GB of RAM) as a non-root user with sudo privileges.
• Install Docker and Docker Compose.
• Configure a domain A record pointing to your server's IP address (for example, kestra.example.com).

Set Up the Directory Structure, Configuration, and Environment Variables

Kestra requires persistent storage for workflow data and PostgreSQL database files. The following steps create the necessary directories and define environment variables for database credentials and domain configuration.

1. Create the project directory with subdirectories for persistent data.

   CONSOLE

   Copy

   $ mkdir -p ~/kestra/data/{postgres,kestra,traefik}

   Explain Code

   Powered by Vultr AgentBeta
   • postgres: Persists PostgreSQL database files.
   • kestra: Contains workflow execution artifacts and uploaded files.
   • traefik: Holds TLS certificates obtained from Let's Encrypt.

2. Navigate to the project directory.

   CONSOLE

   Copy

   $ cd ~/kestra

   Explain Code

   Powered by Vultr AgentBeta

3. Create the .env file.

   CONSOLE

   Copy

   $ nano .env

   Explain Code

   Powered by Vultr AgentBeta

4. Add the following variables:

   INI

   Copy

   POSTGRES_DB=kestra
   POSTGRES_USER=kestra
   POSTGRES_PASSWORD=STRONG-PASSWORD
   DOMAIN=kestra.example.com
   LETSENCRYPT_EMAIL=admin@example.com

   Explain Code

   Powered by Vultr AgentBeta

   Replace:

   • STRONG-PASSWORD with a secure database password.
   • kestra.example.com with your registered domain name.
   • admin@example.com with your email address for Let's Encrypt certificate notifications.

   Save and close the file.

Deploy with Docker Compose

The deployment consists of three services: Traefik handles reverse proxy and TLS certificates, PostgreSQL provides persistent storage for workflow metadata, and Kestra runs the orchestration engine. Docker Compose coordinates all services and applies environment variables for database connectivity and domain routing.

1. Create a new Docker Compose manifest.

   CONSOLE

   Copy

   $ nano docker-compose.yml

   Explain Code

   Powered by Vultr AgentBeta

2. Add the following content:

   YAML

   Copy

   services:
     traefik:
       image: traefik:v3.6
       restart: unless-stopped
       command:
         - "--providers.docker=true"
         - "--providers.docker.exposedbydefault=false"
         - "--entrypoints.web.address=:80"
         - "--entrypoints.websecure.address=:443"
         - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
         - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
         - "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"
         - "--certificatesresolvers.letsencrypt.acme.email=${LETSENCRYPT_EMAIL}"
         - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
       ports:
         - "80:80"
         - "443:443"
       volumes:
         - /var/run/docker.sock:/var/run/docker.sock:ro
         - ./data/traefik:/letsencrypt
   
     postgres:
       image: postgres:15
       restart: unless-stopped
       environment:
         POSTGRES_DB: ${POSTGRES_DB}
         POSTGRES_USER: ${POSTGRES_USER}
         POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
       volumes:
         - ./data/postgres:/var/lib/postgresql/data
       healthcheck:
         test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER}"]
         interval: 10s
         timeout: 5s
         retries: 5
   
     kestra:
       image: kestra/kestra:v1.3.6
       restart: unless-stopped
       command: server standalone
       environment:
         KESTRA_CONFIGURATION: |
           datasources:
             postgres:
               url: jdbc:postgresql://postgres:5432/${POSTGRES_DB}
               driverClassName: org.postgresql.Driver
               username: ${POSTGRES_USER}
               password: ${POSTGRES_PASSWORD}
           kestra:
             repository:
               type: postgres
             storage:
               type: local
               local:
                 basePath: /app/storage
             queue:
               type: postgres
       volumes:
         - ./data/kestra:/app/storage
       depends_on:
         postgres:
           condition: service_healthy
       labels:
         - "traefik.enable=true"
         - "traefik.http.routers.kestra.rule=Host(`${DOMAIN}`)"
         - "traefik.http.routers.kestra.entrypoints=websecure"
         - "traefik.http.routers.kestra.tls.certresolver=letsencrypt"
         - "traefik.http.services.kestra.loadbalancer.server.port=8080"

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   This manifest defines:

   • services: Three containers work together to deliver the workflow orchestration platform with HTTPS enabled.
     • traefik: Acts as a reverse proxy, routes incoming traffic to Kestra, and automatically provisions HTTPS certificates using Let's Encrypt.
     • postgres: Stores workflow definitions, execution metadata, and state information.
     • kestra: Runs the Kestra server and provides the web-based user interface.
   • image: Uses official container images published by the upstream projects. Traefik uses a pinned version (v3.6) for stability.
   • environment (postgres): Defines database name, username, and password.
   • environment (kestra): Contains the KESTRA_CONFIGURATION block that configures the PostgreSQL datasource connection, repository type, local storage path, and queue backend.
   • volumes:
     • ./data/postgres: Persists PostgreSQL database files across container restarts.
     • ./data/kestra: Stores workflow execution artifacts and uploaded files.
     • ./data/traefik: Stores TLS certificates obtained from Let's Encrypt.
   • ports: Ports 80 and 443 expose HTTP and HTTPS traffic through Traefik.
   • labels (kestra): Configure Traefik routing rules to forward HTTPS traffic to Kestra's port 8080 and enable automatic TLS certificate provisioning for the specified domain.
   • depends_on: Ensures the kestra service waits for postgres to be healthy before starting.
   • command: Starts Kestra in standalone server mode with all core components (executor, worker, scheduler) running in a single process.
   • restart: unless-stopped: Ensures containers automatically restart after failures or system reboots unless manually stopped.

3. Create and start the services.

   CONSOLE

   Copy

   $ docker compose up -d

   Explain Code

   Powered by Vultr AgentBeta

4. Verify that the services are running.

   CONSOLE

   Copy

   $ docker compose ps

   Explain Code

   Powered by Vultr AgentBeta

   The output displays three running containers: Kestra, PostgreSQL, and Traefik with ports 80 and 443 exposed.

5. View the service logs to confirm all components started successfully.

   CONSOLE

   Copy

   $ docker compose logs

   Explain Code

   Powered by Vultr AgentBeta

   For more information on managing a Docker Compose stack, see the How To Use Docker Compose article.

Access the Kestra Web Interface

Kestra provides a web-based interface for workflow management and execution monitoring. The first access requires creating an administrator account that has full permissions across all workflows and system settings.

1. Open a web browser and navigate to https://kestra.example.com, replacing kestra.example.com with your configured domain.

   

2. On first access, Kestra displays the Create an admin user page. Enter the following:

   • Email: Your email address (used as the admin login).
   • Password: A secure password with at least 8 characters, 1 uppercase letter, and 1 number.

3. Click Create admin user.

4. (Optional) Complete the Help us improve Kestra OSS form or click Skip to proceed.

5. Click Start Kestra UI to access the main dashboard.

Create and Execute a Sample Workflow

A sample workflow tests the integration between Kestra's orchestration engine and the PostgreSQL backend. The following steps create a basic workflow that logs a message and verify execution through the web interface.

1. Click Flows in the left sidebar.

2. Click Create to open the workflow editor.

3. Paste the following workflow definition.

   YAML

   Copy

   id: hello_world
   namespace: demo
   
   tasks:
     - id: log_message
       type: io.kestra.core.tasks.log.Log
       message: "Hello from Kestra!"

   Explain Code

   Powered by Vultr AgentBeta

4. Click Save to store the workflow.

5. Click Execute in the top-right corner to run the workflow.

6. Click Executions in the left sidebar and select the completed workflow run.

   

   The execution detail page displays a SUCCESS status, confirming that the workflow engine and database integration are functioning correctly.

Conclusion

You have successfully deployed Kestra on a Linux server using Docker Compose with PostgreSQL for persistent workflow storage and Traefik for automatic HTTPS certificate provisioning. For advanced features including scheduling, triggers, and plugin integration, visit the official Kestra documentation.