How to Deploy Slurm on Vultr Kubernetes Engine

Slurm is an open-source job scheduler for High-Performance Computing (HPC) environments. It allocates resources, manages job queues, and orchestrates workloads across clusters, and it powers many of the world's largest supercomputers, offering scalability, flexibility, and support for complex batch workflows.

This guide explains how to deploy a Slurm cluster on Vultr Kubernetes Engine (VKE) using the Slinky Slurm Operator, a Kubernetes operator for Slurm that SchedMD maintains. It covers installing the operator, deploying a Slurm cluster with the controller persisting state to Vultr Block Storage, recording job accounting data in a MariaDB database, exposing controller metrics to Prometheus, and connecting to the cluster over SSH through its login node.

Prerequisites

Before you begin, ensure you:

• Have access to a VKE cluster.
• Have kubectl and helm installed on your machine.
• Have an SSH key pair on your machine to connect to the Slurm login node.

Install cert-manager and the Slurm Operator

The Slurm Operator manages Slurm cluster resources through a set of Custom Resource Definitions (CRDs) and uses cert-manager to issue the certificate for its admission webhook.

1. Install cert-manager and its CRDs.

   CONSOLE

   Copy

   $ helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
       --namespace cert-manager \
       --create-namespace \
       --set crds.enabled=true

   Explain Code

   Powered by Vultr AgentBeta

2. Install the Slurm Operator CRDs.

   CONSOLE

   Copy

   $ helm install slurm-operator-crds oci://ghcr.io/slinkyproject/charts/slurm-operator-crds \
       --version 1.2.1

   Explain Code

   Powered by Vultr AgentBeta

3. Install the Slurm Operator into the slinky namespace.

   CONSOLE

   Copy

   $ helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
       --version 1.2.1 \
       --namespace slinky \
       --create-namespace

   Explain Code

   Powered by Vultr AgentBeta

4. Verify the operator pods are running.

   CONSOLE

   Copy

   $ kubectl get pods --namespace slinky

   Explain Code

   Powered by Vultr AgentBeta

   Your output should be similar to the one below:

   NAME                                      READY   STATUS    RESTARTS   AGE
   slurm-operator-5d86d75979-6wflf           1/1     Running   0          1m
   slurm-operator-webhook-567c84547b-kr7zq   1/1     Running   0          1m

Deploy a Slurm Cluster on Vultr Block Storage

The Slurm chart deploys the controller, worker nodes, login node, and REST API as a single Slurm cluster. By default, the controller persists its state to a Persistent Volume Claim (PVC) that uses the cluster's default storage class, so specify Vultr Block Storage explicitly to control which class it uses.

1. List the storage classes available on the cluster.

   CONSOLE

   Copy

   $ kubectl get storageclasses.storage.k8s.io

   Explain Code

   Powered by Vultr AgentBeta

   The output includes vultr-block-storage, the default Vultr Block Storage class for persistent volumes.

2. Install the Slurm cluster with a worker node set, a partition, and controller persistence on Vultr Block Storage.

   CONSOLE

   Copy

   $ helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
       --version 1.2.1 \
       --set-json 'nodesets={"slinky":{}}' \
       --set 'partitions.all.enabled=true' \
       --set 'controller.persistence.storageClassName=vultr-block-storage' \
       --namespace slurm \
       --create-namespace

   Explain Code

   Powered by Vultr AgentBeta

   Note

   Kubernetes typically does not delete Persistent Volumes when you delete their PVC, so remove unused PVs manually. Disabling controller persistence with --set 'controller.persistence.enabled=false' is only appropriate for testing, because it loses the cluster's state on every controller restart.

3. Verify the Slurm cluster pods are running.

   CONSOLE

   Copy

   $ kubectl get pods --namespace slurm

   Explain Code

   Powered by Vultr AgentBeta

   Your output should be similar to the one below:

   NAME                                  READY   STATUS    RESTARTS   AGE
   slurm-controller-0                    3/3     Running   0          2m
   slurm-restapi-77b9f969f7-kh4r8        1/1     Running   0          2m
   slurm-worker-slinky-0                 2/2     Running   0          2m

Enable Job Accounting with MariaDB

Slurm accounting records job history and resource usage in a MariaDB or MySQL-compatible database. Deploy a database using the community-edition MariaDB Operator, then point Slurm's accounting service at it.

1. Add the MariaDB Operator Helm repository.

   CONSOLE

   Copy

   $ helm repo add mariadb-operator https://helm.mariadb.com/mariadb-operator

   Explain Code

   Powered by Vultr AgentBeta

2. Update the local Helm repository cache.

   CONSOLE

   Copy

   $ helm repo update

   Explain Code

   Powered by Vultr AgentBeta

3. Install the MariaDB Operator CRDs.

   CONSOLE

   Copy

   $ helm install mariadb-operator-crds mariadb-operator/mariadb-operator-crds

   Explain Code

   Powered by Vultr AgentBeta

4. Install the MariaDB Operator controller into the mariadb namespace.

   CONSOLE

   Copy

   $ helm install mariadb-operator mariadb-operator/mariadb-operator \
       --namespace mariadb \
       --create-namespace

   Explain Code

   Powered by Vultr AgentBeta

5. Create a manifest file for the accounting database.

   CONSOLE

   Copy

   $ nano mariadb.yaml

   Explain Code

   Powered by Vultr AgentBeta

6. Add the following configuration:

   YAML

   Copy

   apiVersion: k8s.mariadb.com/v1alpha1
   kind: MariaDB
   metadata:
     name: mariadb
     namespace: slurm
   spec:
     rootPasswordSecretKeyRef:
       name: mariadb-root
       key: password
       generate: true
     username: slurm
     database: slurm_acct_db
     passwordSecretKeyRef:
       name: mariadb-password
       key: password
       generate: true
     storage:
       size: 16Gi
       storageClassName: vultr-block-storage
     myCnf: |
       [mariadb]
       bind-address=*
       default_storage_engine=InnoDB
       binlog_format=row
       innodb_autoinc_lock_mode=2
       innodb_buffer_pool_size=4096M
       innodb_lock_wait_timeout=900
       innodb_log_file_size=1024M
       max_allowed_packet=256M

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   • username and database: The credentials and database name the Slurm accounting service connects with. Use these same values later to query the accounting database directly.
   • storage: Requests a 16Gi PVC on Vultr Block Storage for the database. This example aligns with the Slurm chart's default accounting.storageConfig; update it if your database configuration differs.
   • rootPasswordSecretKeyRef and passwordSecretKeyRef: Generates and stores the root and application passwords in Kubernetes Secrets automatically.

7. Apply the manifest to create the database in the slurm namespace.

   CONSOLE

   Copy

   $ kubectl apply -f mariadb.yaml

   Explain Code

   Powered by Vultr AgentBeta

8. Verify the database is ready.

   CONSOLE

   Copy

   $ kubectl get mariadbs.k8s.mariadb.com --namespace slurm

   Explain Code

   Powered by Vultr AgentBeta

   Verify the READY column shows True before continuing.

9. Enable accounting on the Slurm cluster.

   CONSOLE

   Copy

   $ helm upgrade slurm oci://ghcr.io/slinkyproject/charts/slurm \
       --version 1.2.1 \
       --reuse-values \
       --set 'accounting.enabled=true' \
       --namespace slurm

   Explain Code

   Powered by Vultr AgentBeta

   The --reuse-values flag keeps the node set, partition, and storage class values you set during the initial install.

10. Verify the accounting pod is running.

    CONSOLE

    Copy

    $ kubectl get pods --namespace slurm --selector 'app.kubernetes.io/component=accounting'

    Explain Code

    Powered by Vultr AgentBeta

    Verify the slurm-accounting-0 pod shows a Running status.

Enable Cluster Metrics with Prometheus

The Slurm controller can expose metrics for Prometheus to scrape through a ServiceMonitor resource, which the kube-prometheus-stack chart's Prometheus Operator watches for automatically.

1. Add the Prometheus Community Helm repository.

   CONSOLE

   Copy

   $ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

   Explain Code

   Powered by Vultr AgentBeta

2. Update the local Helm repository cache.

   CONSOLE

   Copy

   $ helm repo update

   Explain Code

   Powered by Vultr AgentBeta

3. Install the kube-prometheus-stack chart and its CRDs into the prometheus namespace.

   CONSOLE

   Copy

   $ helm install prometheus prometheus-community/kube-prometheus-stack \
       --set 'installCRDs=true' \
       --namespace prometheus \
       --create-namespace

   Explain Code

   Powered by Vultr AgentBeta

4. Enable Slurm controller metrics and the Prometheus service monitor.

   CONSOLE

   Copy

   $ helm upgrade slurm oci://ghcr.io/slinkyproject/charts/slurm \
       --version 1.2.1 \
       --reuse-values \
       --set 'controller.metrics.enabled=true' \
       --set 'controller.metrics.serviceMonitor.enabled=true' \
       --namespace slurm

   Explain Code

   Powered by Vultr AgentBeta

5. Verify the service monitor was created.

   CONSOLE

   Copy

   $ kubectl get servicemonitors.monitoring.coreos.com --namespace slurm

   Explain Code

   Powered by Vultr AgentBeta

   The output displays a service monitor for the Slurm controller.

Connect to the Slurm Cluster

The Slurm chart's login node accepts SSH connections and gives you a shell from which to run Slurm client commands against the cluster. Authorize your own SSH public key on the login node rather than configuring an external identity service.

1. Enable the login node and authorize your SSH public key. Replace SSH-PUBLIC-KEY-PATH with the path to your own public key, such as ~/.ssh/id_ed25519.pub or ~/.ssh/id_rsa.pub.

   CONSOLE

   Copy

   $ helm upgrade slurm oci://ghcr.io/slinkyproject/charts/slurm \
       --version 1.2.1 \
       --reuse-values \
       --set 'loginsets.slinky.enabled=true' \
       --set-file "loginsets.slinky.rootSshAuthorizedKeys=SSH-PUBLIC-KEY-PATH" \
       --namespace slurm

   Explain Code

   Powered by Vultr AgentBeta

2. Retrieve the load balancer IP address and port assigned to the login service.

   CONSOLE

   Copy

   $ SLURM_LOGIN_IP="$(kubectl get services --namespace slurm slurm-login-slinky -o jsonpath='{.status.loadBalancer.ingress[0].ip}')"
   $ SLURM_LOGIN_PORT="$(kubectl get services --namespace slurm slurm-login-slinky -o jsonpath='{.spec.ports[0].port}')"

   Explain Code

   Powered by Vultr AgentBeta

   Note

   VKE provisions a Vultr Load Balancer for the login service automatically. Allow a few minutes for VKE to assign an external IP address.

3. Verify both values were retrieved before continuing.

   CONSOLE

   Copy

   $ echo "${SLURM_LOGIN_IP}:${SLURM_LOGIN_PORT}"

   Explain Code

   Powered by Vultr AgentBeta

   The output shows an IP address and port, for example 66.135.9.113:22. If SLURM_LOGIN_IP is empty, the load balancer has not finished provisioning; wait a few minutes and rerun the previous step.

4. Connect to the login node over SSH.

   CONSOLE

   Copy

   $ ssh -p ${SLURM_LOGIN_PORT:-22} root@${SLURM_LOGIN_IP}

   Explain Code

   Powered by Vultr AgentBeta

5. From the login node, check the partition and node status.

   CONSOLE

   Copy

   $ sinfo

   Explain Code

   Powered by Vultr AgentBeta

   The output resembles the following:

   PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
   all*         up   infinite      1   idle slinky-0

   Verify the worker node shows idle, not down or drain.

6. Run a command directly on a worker node.

   CONSOLE

   Copy

   $ srun hostname

   Explain Code

   Powered by Vultr AgentBeta

   The output shows the hostname of the worker node the job ran on, confirming the controller can schedule work:

   slinky-0

7. Submit a batch job.

   CONSOLE

   Copy

   $ sbatch --wrap="sleep 60"

   Explain Code

   Powered by Vultr AgentBeta

   The output confirms the controller accepted the job:

   Submitted batch job 2

8. List jobs in the queue.

   CONSOLE

   Copy

   $ squeue

   Explain Code

   Powered by Vultr AgentBeta

   The output shows the job in a running (R) or pending (PD) state:

   JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
       2       all     wrap     root  R       0:04      1 slinky-0

9. Display accounting records for both jobs from the MariaDB database configured earlier.

   CONSOLE

   Copy

   $ sacct

   Explain Code

   Powered by Vultr AgentBeta

   The output includes the earlier srun job and the sbatch job submitted in the previous step:

   JobID           JobName  Partition    Account  AllocCPUS      State ExitCode
   ------------ ---------- ---------- ---------- ---------- ---------- --------
   1              hostname        all       root          2  COMPLETED      0:0
   1.0            hostname                  root          2  COMPLETED      0:0
   2                  wrap        all       root          2    RUNNING      0:0
   2.batch           batch                  root          2    RUNNING      0:0

   The sleep 60 job still shows RUNNING immediately after submission; rerun sacct after a minute to see it change to COMPLETED.

Run an Example Batch Script

The commands above submit a job inline with --wrap. A batch script defines the job's resource requests and commands in a single file, which is the standard way to submit repeatable work to Slurm.

1. From the login node, create the script.

   CONSOLE

   Copy

   $ cat > example.sh <<'EOF'
   #!/bin/bash
   #SBATCH --job-name=example
   #SBATCH --output=example.out
   #SBATCH --ntasks=1
   #SBATCH --time=00:01:00
   
   echo "Running on $(hostname)"
   date
   sleep 10
   echo "Job complete"
   EOF

   Explain Code

   Powered by Vultr AgentBeta
   • --job-name: The name Slurm displays for this job in squeue and sacct.
   • --output: The file Slurm writes the job's combined stdout and stderr to, relative to the worker node's own filesystem.
   • --ntasks: The number of tasks the job requests. This example requests one.
   • --time: The job's time limit, after which Slurm cancels it.

2. Submit the script to the queue.

   CONSOLE

   Copy

   $ sbatch example.sh

   Explain Code

   Powered by Vultr AgentBeta

   The output confirms the controller accepted the job:

   Submitted batch job 3

   Note

   The example.out file is written on the node the job is scheduled to, not the login node.

3. Wait for the job to complete, then verify it succeeded from the login node using sacct.

   CONSOLE

   Copy

   $ sacct --job 3 --format=JobID,JobName,State,ExitCode

   Explain Code

   Powered by Vultr AgentBeta

   The output confirms the job ran to completion with a zero exit code:

   JobID           JobName      State ExitCode
   ------------ ---------- ---------- --------
   3               example  COMPLETED      0:0
   3.batch           batch  COMPLETED      0:0

Conclusion

You have deployed a Slurm cluster on Vultr Kubernetes Engine using the Slinky Slurm Operator, with the controller persisting state to Vultr Block Storage, job accounting recorded in a MariaDB database, controller metrics exposed to Prometheus, and SSH access through the login node. From here, submit your own batch jobs with sbatch, add worker capacity by adjusting the nodesets value on the Slurm chart, or define additional partitions to separate workloads by hardware or priority. For more information, visit the Slinky documentation or the Slurm documentation.