Install Prometheus and Grafana on Vultr Kubernetes Engine with Prometheus-Operator

Updated on January 12, 2023
Install Prometheus and Grafana on Vultr Kubernetes Engine with Prometheus-Operator header image

Monitoring a Kubernetes cluster is an essential task for any Kubernetes administrator. It will help you to ensure that all Kubernetes services are up and running. Many solutions are available to track and monitor cluster resources, including memory, CPU, and storage. Prometheus and Grafana are one of the most popular Kubernetes monitoring stacks.

Deploying Prometheus with Grafana helps you to verify if nodes are functioning properly and at the right capacity. Prometheus allows you to capture different metrics in Kubernetes, while Grafana is a data visualization platform that pulls up metrics from Prometheus and transforms metrics into meaningful visualizations.

This article explains the steps to set up Prometheus and Grafana monitoring stack on Vultr Kubernetes Engine (VKE) using Prometheus-Operator.


Before you begin, you should:

Add Prometheus Helm Chart

Prometheus Helm chart contains a group of four monitoring tools, including Prometheus, Grafana, Alertmanager, and Prometheus operator, to monitor the Kubernetes cluster.

  1. Create a monitoring namespace.

     # kubectl create namespace monitoring

    Sample output.

     namespace/monitoring created
  2. Add the Prometheus community Helm chart.

     # helm repo add prometheus-community
  3. Update the Helm repository.

     # helm repo update
  4. Verify the added Helm chart.

     # helm repo list

    Sample output.


Install Prometheus Stack

At this point, the Prometheus Helm chart is ready to deploy the full Prometheus Kubernetes monitoring stack.

  1. Now, run the helm install command to deploy the Prometheus stack on Kubernetes.

     # helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring

    Sample output.

     NAME: prometheus
     LAST DEPLOYED: Wed Dec 28 04:04:04 2022
     NAMESPACE: monitoring
     STATUS: deployed
     REVISION: 1
     kube-prometheus-stack has been installed. Check its status by running:
     kubectl --namespace monitoring get pods -l "release=prometheus"
  2. Verify the Kubernetes Prometheus deployment.

     # kubectl get pods -n monitoring

    Sample output.

     alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2 Running 1 (3m12s ago) 3m18s
     prometheus-grafana-7bdd794646-5jzht 3/3 Running 0 3m27s
     prometheus-kube-prometheus-operator-77dc45b447-scllf 1/1 Running 0 3m27s
     prometheus-kube-state-metrics-6cfd96f4c8-rqbs5 1/1 Running 0 3m27s
     prometheus-prometheus-kube-prometheus-prometheus-0 2/2 Running 0 3m18s
     prometheus-prometheus-node-exporter-6nmn8 1/1 Running 0 3m27s
     prometheus-prometheus-node-exporter-f8vmf 1/1 Running 0 3m27s
     prometheus-prometheus-node-exporter-kbpgw 1/1 Running 0 3m27s

Access Prometheus Dashboard

Prometheus is now deployed on Kubernetes and is ready to monitor Kubernetes metrics. However, you can not access Prometheus from the external network. You will need to run the kubectl port-forward command to forward a local port 9090 to your cluster.

  1. Run the kubectl get command to check Prometheus services.

     # kubectl get svc -n monitoring

    This will show you all services deployed in the Kubernetes cluster.

     alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 3m33s
     prometheus-grafana ClusterIP <none> 80/TCP 3m42s
     prometheus-kube-prometheus-alertmanager ClusterIP <none> 9093/TCP 3m42s
     prometheus-kube-prometheus-operator ClusterIP <none> 443/TCP 3m42s
     prometheus-kube-prometheus-prometheus ClusterIP <none> 9090/TCP 3m42s
     prometheus-kube-state-metrics ClusterIP <none> 8080/TCP 3m42s
     prometheus-operated ClusterIP None <none> 9090/TCP 3m33s
     prometheus-prometheus-node-exporter ClusterIP <none> 9100/TCP 3m42s
  2. Pick the Prometheus service prometheus-kube-prometheus-prometheus from the above output and run the kubectl port-forward command to forward a local port 9090.

     # kubectl port-forward svc/prometheus-kube-prometheus-prometheus -n monitoring 9090 &

    Sample output.

     Forwarding from -> 9090
     Forwarding from [::1]:9090 -> 9090
  3. Open your web browser and navigate to http://localhost:9090. If the Prometheus service works, you should see the following screen.

    Prometheus Dashboard

Access Kube-state-metrics (KSM)

Kube-state-metrics (KSM) tool allows you to see the Kubernetes cluster's internal state metrics. It will help you to monitor resource usage, health statistics, node metrics, deployment metrics, and pod metrics via a web browser.

  1. Run the following command to check the kube-state-metrics status.

     # kubectl get svc -n monitoring | grep kube-state-metrics

    Sample output.

     prometheus-kube-state-metrics ClusterIP <none> 8080/TCP 4m38s
  2. Use the prometheus-kube-state-metrics from the above output and run the kubectl port-forward command to forward a local port 8080.

     # kubectl port-forward svc/prometheus-kube-state-metrics -n monitoring 8080 &

    Sample output.

     Forwarding from -> 8080
     Forwarding from [::1]:8080 -> 8080
  3. Open your web browser and access the Kube-state-metrics (KSM) dashboard using the URL http://localhost:8080. You should see the Kubernetes internal state metrics on the following screen.

    Kube-state-metrics List Kube-state-metrics

Visualize Kubernetes Internal State Metric Via Prometheus

  1. Access the Prometheus endpoints using the URL http://localhost:9090/targets and verify whether these targets have been properly configured. You should see Kubernetes internal and monitoring components on the following screen.

    Prometheus Targets

  2. Click the Graph menu and type the PromQL query in the expression space.

     sum((container_memory_usage_bytes{container!= "POD",container!=""} - on (namespace,pod,container) avg by (namespace,pod,container)     (kube_pod_container_resource_requests{resource= "memory"})) * -1 >0 ) / (1024*1024*1024)
  3. Click the Execute. You should see the total amount of unused memory in your cluster in a graphical format.

    Prometheus Query Graph

Access Grafana Dashboard

Grafana is a very useful tool for visualizing Kubernetes matrics and workloads in a graphical format.

  1. Before accessing the Grafana dashboard, you must retrieve the Grafana login password stored as a secret in Kubernetes. Run the kubectl command followed by Grafana service prometheus-grafana to view data stored as a secret in the Kubernetes cluster.

     # kubectl get secret -n monitoring prometheus-grafana -o yaml

    You should see the admin-password in base64 format in the following output.

     apiVersion: v1
     admin-password: cHJvbS1vcGVyYXRvcg==
     admin-user: YWRtaW4=
     kind: Secret
     annotations: prometheus monitoring
     creationTimestamp: "2022-12-28T04:04:40Z"
     labels: prometheus Helm grafana 9.3.1 grafana-6.48.0
     name: prometheus-grafana
     namespace: monitoring
     resourceVersion: "2287"
     uid: f52d3bf0-c087-494f-ada0-794353930a36
     type: Opaque
  2. Copy the admin-password value cHJvbS1vcGVyYXRvcg== from the above output and decode it using the following command.

     # echo cHJvbS1vcGVyYXRvcg== | base64 --decode

    You should see the Grafana admin password in the following output.

  3. Run the kubectl command to port-forward Grafana port 80 to a local port at 3000.

     # kubectl port-forward svc/prometheus-grafana -n monitoring 3000:80 &

    Sample output.

     Forwarding from -> 3000
     Forwarding from [::1]:3000 -> 3000
  4. Open your web browser and access Grafana using the URL http://localhost:3000. You should see the Grafana login screen.

    Grafana Login Page

  5. Enter the username as admin and password as prom-operator, then click on the Log in button. You should see the Grafana dashboard on the following screen.

    Grafana Dashboard

  6. By default, Grafana has pre-configured dashboards for each target, so you don't need to set up a dashboard to visualize Prometheus metrics. Click on the Dashboard => Browse. You should see all pre-configured dashboards on the following screen.

    Grafana Dashboard List

  7. Click the Kubernetes / Compute Resources / Namespace (Pods) dashboard. You should see the dashboard for visualizing compute resource usage by Pods in the monitoring namespace.

    Grafana Monitoring Dashboard


In this post, you learned how to set up Prometheus and Grafana on Vultr Kubernetes using a Helm chart. You also learned how to visualize Kubernetes internal metrics and display them on the Grafana dashboard. You can now explore Prometheus and Grafana features and start monitoring your Kubernetes cluster.