How to Monitor Your VKE Cluster with tobs

tobs - The Observability Stack for Kubernetes is a Kubernetes monitoring stack that collects metrics with Prometheus, visualizes them with Grafana, and stores them long term in TimescaleDB through Promscale. It also bundles supporting components such as AlertManager, Node-Exporter, and Kube-State-Metrics, so a single Helm release provides collection, storage, alerting, and dashboards.

This guide explains how to install tobs in a Vultr Kubernetes Engine (VKE) cluster and back its metrics database with Vultr Object Storage. It covers installing the stack with Helm, customizing the chart values for Vultr Block Storage, visualizing metrics in Grafana, querying the metrics database directly with SQL, and troubleshooting a failed deployment.

Prerequisites

Before you begin, you need to:

• Deploy a Vultr Kubernetes Engine cluster with at least two 4 GB nodes. Nodes with 2 GB of memory are not sufficient for the tobs stack.
• Download the cluster configuration and configure kubectl on your local machine.
• Install Helm 3 or later.
• Provision an Object Storage subscription and create a bucket to use as the S3-compatible backup target for TimescaleDB.

Install tobs

The tobs Helm chart pins each bundled component to a version tested against a specific range of Kubernetes releases. Confirm that your cluster falls within the supported range before installing, because an unsupported version leaves pods in a crash loop that is difficult to diagnose later.

1. Check the Kubernetes version running on your nodes.

   CONSOLE

   Copy

   $ kubectl get nodes

   Explain Code

   Powered by Vultr AgentBeta

   The output displays each node with a Ready status and its Kubernetes version.

2. Check the compatibility matrix to confirm which tobs version supports your Kubernetes version.

3. Add the Timescale Helm repository.

   CONSOLE

   Copy

   $ helm repo add timescale https://charts.timescale.com/

   Explain Code

   Powered by Vultr AgentBeta

4. Update the repository cache.

   CONSOLE

   Copy

   $ helm repo update

   Explain Code

   Powered by Vultr AgentBeta

5. Generate a values file from the chart defaults.

   CONSOLE

   Copy

   $ helm show values timescale/tobs > my_values.yml

   Explain Code

   Powered by Vultr AgentBeta

6. Open the values file in a text editor.

   CONSOLE

   Copy

   $ nano my_values.yml

   Explain Code

   Powered by Vultr AgentBeta

7. Change every storage: 8Gi entry to storage: 10Gi. Vultr Block Storage enforces a 10 GB minimum, so a smaller request leaves the volume claim unbound.

8. Reduce the size: 150Gi value under timescaledb-single to a smaller volume if you want to lower the storage cost.

   Save and close the file.

9. Install the stack. Replace RELEASE-NAME with a name for the Helm release.

   CONSOLE

   Copy

   $ helm upgrade --wait --install RELEASE-NAME --values my_values.yml timescale/tobs

   Explain Code

   Powered by Vultr AgentBeta

   The installer prompts for your Object Storage bucket name, endpoint hostname, region, access key, and secret key. Enter the values for the bucket you created in the prerequisites, and leave the region blank to accept the default.

   Note

   • The --wait flag is required. The chart creates OpenTelemetry custom resources only after the OpenTelemetry operator is running, and omitting the flag causes those resources to fail. You can omit it when installing tobs without OpenTelemetry support.
   • Installing the full stack with OpenTelemetry support requires cert-manager in the cluster. Follow the cert-manager documentation to install it first. cert-manager is not required when OpenTelemetry support is disabled.

10. Watch the deployment progress.

    CONSOLE

    Copy

    $ kubectl get pods

    Explain Code

    Powered by Vultr AgentBeta

    The installation takes several minutes. Pods report CrashLoopBackOff while dependencies start, which is expected during this period.

11. Verify that the stack finished deploying.

    CONSOLE

    Copy

    $ kubectl get pods

    Explain Code

    Powered by Vultr AgentBeta

    Verify that the Prometheus, Grafana, Promscale, TimescaleDB, and exporter pods all report a Running status. Pods that remain in CrashLoopBackOff after the deployment settles indicate a problem, which the troubleshooting section addresses.

Configure the Stack

Every component in tobs is configured through the Helm values file rather than through individual component settings. Editing that single file and upgrading the release keeps the whole stack consistent.

1. Review the default values file in the project repository to see the available settings, which are documented inline.

2. Consult the Helm chart documentation for a description of each configuration option.

3. Apply any changes by upgrading the release with your edited values file. Replace RELEASE-NAME with your Helm release name.

   CONSOLE

   Copy

   $ helm upgrade --wait --install RELEASE-NAME --values my_values.yml timescale/tobs

   Explain Code

   Powered by Vultr AgentBeta

Troubleshoot the Deployment

Failed deployments surface as pods stuck in a crash loop, and the cause is usually visible in the cluster events or the pod logs. Work from the cluster-wide events down to the individual pod.

1. List cluster events in chronological order.

   CONSOLE

   Copy

   $ kubectl get events --sort-by='.metadata.creationTimestamp'

   Explain Code

   Powered by Vultr AgentBeta

2. Describe a failing pod to inspect its state and recent events. Replace POD-NAME with the name of the pod.

   CONSOLE

   Copy

   $ kubectl describe pod POD-NAME

   Explain Code

   Powered by Vultr AgentBeta

3. View the logs of a failing pod. Replace POD-NAME with the name of the pod.

   CONSOLE

   Copy

   $ kubectl logs POD-NAME

   Explain Code

   Powered by Vultr AgentBeta

Resolve a Promscale Authentication Failure

The Promscale pod stores its own copy of the database password. When that copy falls out of sync with the TimescaleDB superuser password, Promscale fails to connect and enters a crash loop.

1. View the Promscale pod log. Replace POD-NAME with your Promscale pod name.

   CONSOLE

   Copy

   $ kubectl logs POD-NAME

   Explain Code

   Powered by Vultr AgentBeta

   A password mismatch produces the following error.

   password authentication failed for user \"postgres\" (SQLSTATE 28P01))

2. Read the superuser password from the credentials secret.

   CONSOLE

   Copy

   $ kubectl get secrets tobs-credentials -o jsonpath="{.data.PATRONI_SUPERUSER_PASSWORD}"

   Explain Code

   Powered by Vultr AgentBeta

3. Edit the Promscale secret and set the PROMSCALE_DB_PASSWORD key to the value from the previous step.

   CONSOLE

   Copy

   $ kubectl edit secrets tobs-promscale

   Explain Code

   Powered by Vultr AgentBeta

4. Delete the Promscale pod so that the deployment recreates it with the corrected secret. Replace POD-NAME with your Promscale pod name.

   CONSOLE

   Copy

   $ kubectl delete pod POD-NAME

   Explain Code

   Powered by Vultr AgentBeta

5. Verify that the replacement pod reaches a Running status.

   CONSOLE

   Copy

   $ kubectl get pods

   Explain Code

   Powered by Vultr AgentBeta

Conclusion

You have installed the tobs observability stack in a Vultr Kubernetes Engine cluster, configured it to store long-term metrics in TimescaleDB with Vultr Object Storage as the backup target, and resolved the most common deployment failure. Because the project is archived, treat this deployment as a short-term or evaluation setup and plan a migration to a maintained stack. For more information, visit the tobs repository.