How to Monitor Application Logs in Kubernetes using Loki

Introduction

Loki is a log aggregation system. The Loki stack comprises of three components that work with Grafana and Promtail to enable application log monitoring within a Kubernetes cluster.

Loki is the core component responsible for log ingestion, storage, and query processing while Promtail acts as the agent responsible for collecting log information from cluster sources. In addition, the Grafana dashboard serves as the main interface for querying, visualizing, and exploring the log data stored in Loki.

This article explains how to monitor application logs in Kubernetes using Loki. You are to deploy the Loki stack in a Vultr Kubernetes Engine (VKE) cluster, deploy a sample application, and monitor logs using the Grafana dashboard.

Prerequsites

Before you begin:

• Deploy a Vultr Kubernetes Engine (VKE) cluster with at least 3 nodes and 4096 MB per worker node

• Deploy a OneClick Docker instance using the Vultr Marketplace application to use as the management workstation.

• Create a new Vultr Container Registry to store containerized application images.

• Access your management server using SSH as a non-root sudo user.

• Install and Configure Kubectl to access your VKE cluster.

• Install the Helm package manager.

  CONSOLE

  Copy

  $ sudo snap install helm --classic

  Explain Code

  Powered by Vultr AgentBeta

Install the Loki stack

1. Add the grafana Helm repository to your sources.

   CONSOLE

   Copy

   $ helm repo add grafana https://grafana.github.io/helm-charts

   Explain Code

   Powered by Vultr AgentBeta

2. Install the Loki stack to your cluster using the Helm chart.

   CONSOLE

   Copy

   $ helm install loki grafana/loki-stack -n loki-stack --set grafana.enabled=true --set grafana.service.type=LoadBalancer --create-namespace

   Explain Code

   Powered by Vultr AgentBeta

   The above command installs the Loki stack with Grafana enabled, and sets up a LoadBalancer service which in turn creates a Vultr Load Balancer for external access. In addition, a new namespace loki-stack is created to centralize all Loki resources.

   When successful, your output should be similar to the one below:

   The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.
   See http://docs.grafana.org/features/datasources/loki/ for more detail.

3. Wait for at least 2 minutes for the deployment process to complete. Then, view all Pods in the loki-stack namespace.

   CONSOLE

   Copy

   $ kubectl get pods -n loki-stack

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME                            READY   STATUS    RESTARTS   AGE
   loki-0                          1/1     Running   0          2m
   loki-grafana-84d9c8dd87-sqtjr   2/2     Running   0          2m
   loki-promtail-mft9z             1/1     Running   0          2m

Create a Sample Cluster Application

To test the Loki stack functionalities, follow the steps below to create a sample Golang application to run and monitor within your Kubernetes cluster.

1. Create a new application directory.

   CONSOLE

   Copy

   $ mkdir vke-loki

   Explain Code

   Powered by Vultr AgentBeta

2. Switch to the directory.

   CONSOLE

   Copy

   $ cd vke-loki

   Explain Code

   Powered by Vultr AgentBeta

3. Create a new main application file main.go using a text editor such as Nano.

   CONSOLE

   Copy

   $ nano main.go

   Explain Code

   Powered by Vultr AgentBeta

4. Add the following contents to the file.

   GO

   Copy

   package main
   
   import (
       "log"
       "net/http"
   )
   
   func main() {
       http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
           log.Println("Received request for path:", r.URL.Path)
           w.Write([]byte("Hello, World!"))
       })
   
       http.HandleFunc("/error", func(w http.ResponseWriter, r *http.Request) {
           log.Println("Simulating an error...")
           http.Error(w, "simulated error", http.StatusInternalServerError)
       })
      
       log.Println("Starting server on :8080")
       if err := http.ListenAndServe(":8080", nil); err != nil {
           log.Fatal(err)
       }
   }

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   The above application code defines an HTTP server that runs on the host port 8080 using the net/http standard library package. The main function sets up two HTTP handlers, the root (/) and /error paths. When triggered, the root handler responds with a "Hello, World!" message and logs information about the received request, while the /error handler simulates an error by returning a simulated internal server error (HTTP status code 500). The server is configured to listen on port 8080.

5. Create a new dependency file go.mod.

   CONSOLE

   Copy

   $ nano go.mod

   Explain Code

   Powered by Vultr AgentBeta

6. Add the following contents to the file.

   GO

   Copy

   module go-web-app
   
   go 1.21.4

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

Containerize the Application

1. Create a Dockerfile to define the application container environment.

   CONSOLE

   Copy

   $ nano Dockerfile

   Explain Code

   Powered by Vultr AgentBeta

2. Add the following contents to the file.

   DOCKERFILE

   Copy

   FROM golang AS build
   
   WORKDIR /app
   COPY go.mod ./
   
   COPY main.go ./
   RUN CGO_ENABLED=0 go build -o /go-app
   
   FROM gcr.io/distroless/base-debian10
   WORKDIR /
   COPY --from=build /go-app /go-app
   EXPOSE 8080
   USER nonroot:nonroot
   ENTRYPOINT ["/go-app"]

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   The above configuration defines a multi-stage build for the Go application. Within the first stage, it uses the official Golang base image to build the application. Then, it uses /app as the working directory, copies the go.mod and main.go files, and builds the Go application with CGO disabled, to create the go-app binary.

   The second stage uses the gcr.io/distroless/base-debian10 image, / as the working directory. Then. it copies the compiled binary from the build stage to /go-app that works as the application entry point that listens for connections on port 8080.

3. Build the Docker image.

   CONSOLE

   Copy

   $ docker build -t go-web-app .

   Explain Code

   Powered by Vultr AgentBeta

4. Export your Vultr Container Registry access credentials as environmental variables to store Docker images.

   CONSOLE

   Copy

   $ export VULTR_CONTAINER_REGISTRY_USERNAME=<enter the Vultr Container Registry username>
   $ export VULTR_CONTAINER_REGISTRY_API_KEY=<enter the Vultr Container Registry API key>
   $ export VULTR_CONTAINER_REGISTRY_NAME=<enter the Vultr Container Registry name>

   Explain Code

   Powered by Vultr AgentBeta

5. Log in to your Vultr Container Registry.

   CONSOLE

   Copy

   $ docker login https://sjc.vultrcr.com/$VULTR_CONTAINER_REGISTRY_NAME -u $VULTR_CONTAINER_REGISTRY_USERNAME -p $VULTR_CONTAINER_REGISTRY_API_KEY

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   Login Succeeded

6. Tag the local Docker image with the Vultr Container Registry repository name.

   CONSOLE

   Copy

   $ docker tag go-web-app:latest sjc.vultrcr.com/$VULTR_CONTAINER_REGISTRY_NAME/go-web-app:latest

   Explain Code

   Powered by Vultr AgentBeta

7. Push the image to your registry.

   CONSOLE

   Copy

   $ docker push sjc.vultrcr.com/$VULTR_CONTAINER_REGISTRY_NAME/go-web-app:latest

   Explain Code

   Powered by Vultr AgentBeta

Apply Vultr Container Registry Credentials to the Kubernetes Cluster

1. Access your Vultr Container Registry control panel in the Vultr Console.

2. Navigate to the Docker/Kubernetes tab.

   

3. Click Generate Kubernetes YAML within the Docker Credentials For Kubernetes section to generate a new YAML file with your access details.

4. Open the downloaded file and copy all contents to your clipboard.

5. Create a new Secret resource file secret.yaml.

   CONSOLE

   Copy

   $ nano secret.yaml

   Explain Code

   Powered by Vultr AgentBeta

6. Paste your generated Kubernetes YAML contents to the file. For example:

   YAML

   Copy

   apiVersion: v1
   kind: Secret
   metadata:
     name: vultr-cr-credentials
   data:
     .dockerconfigjson: eyJhifX19
   type: kubernetes.io/dockerconfigjson
   

   Explain Code

   Powered by Vultr AgentBeta

7. Apply the Secret to your VKE cluster.

   CONSOLE

   Copy

   $ kubectl apply -f secret.yaml

   Explain Code

   Powered by Vultr AgentBeta

   When successful, all cluster resources can access and pull data from your Vultr Container Registry repositories.

Deploy the Sample Application to your VKE cluster

1. Create a new Deployment resource file app.yaml.

   CONSOLE

   Copy

   $ nano app.yaml

   Explain Code

   Powered by Vultr AgentBeta

2. Add the following contents to the file. Replace testreg with your actual Vultr Container Registry name.

   YAML

   Copy

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: go-web-app
     labels:
       app: go-web-app
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: go-web-app
     template:
       metadata:
         labels:
           app: go-web-app
       spec:
         containers:
         - name: go-web-app
           image: sjc.vultrcr.com/testreg/go-web-app:latest
           imagePullPolicy: Always
         imagePullSecrets:
         - name: vultr-cr-credentials
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: go-web-app-service
     labels:
       app: go-web-app-service
   spec:
     type: LoadBalancer
     ports:
     - port: 8080
     selector:
       app: go-web-app

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   The above configuration creates a new Deployment and Service resource for the Go web application with the following specifications:

   • The Deployment specifies a container within a Pod that uses the Go web application image from your Vultr Container Registry.
   • The Service uses the LoadBalancer resource to expose the application port 8080 and direct traffic to Pods with the go-web-app label.
   • The imagePullSecrets section uses the vultr-cr-credentials Secret variable to authenticate and pull the container image from your Vultr Container Registry repository.

3. Deploy the application to your cluster.

   CONSOLE

   Copy

   $ kubectl apply -f app.yaml

   Explain Code

   Powered by Vultr AgentBeta

4. Wait for at least 1 minute for the deployment process to complete, then, view all Pods with the go-web-app label.

   CONSOLE

   Copy

   $ kubectl get pods -l=app=go-web-app

   Explain Code

   Powered by Vultr AgentBeta

   Your output should be similar to the one below:

   NAME                          READY   STATUS             RESTARTS   AGE
   go-web-app-76c9ffdf67-dgh86   1/1     Running            0          20s

Access the Grafana Dashboard

1. Retrieve the default Grafana login password from the loki-grafana Secret.

   CONSOLE

   Copy

   $ kubectl get secret -n loki-stack loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

   Explain Code

   Powered by Vultr AgentBeta

   Copy the generated administrator password similar to the one below:

   AABBHHS662SHH

2. View the loki-grafana Service and retrieve the assigned Vultr Load Balancer IP Address for external access.

   CONSOLE

   Copy

   $ kubectl get svc loki-grafana -n loki-stack -o=jsonpath='{.status.loadBalancer.ingress[0].ip}'

   Explain Code

   Powered by Vultr AgentBeta

   Your output should be similar to the one below:

   192.168.0.240

3. Visit your cluster Load Balancer IP to access the Grafana web interface using a web browser such as Chrome.

   http://192.168.0.240/login

   When prompted, enter the following details in the respective fields to log in.

   • Username: admin
   • Password: Your Generated Password

   When successful, navigate to Explore within the Grafana dashboard.

   

Invoke the application

1. Open your SSH session and view the go-web-app Load Balancer Service IP Address. Access the application endpoint using Vultr Load Balancer. To get the Load Balancer IP, run the below command:

   CONSOLE

   Copy

   $ kubectl get svc go-web-app-service -o=jsonpath='{.status.loadBalancer.ingress[0].ip}'

   Explain Code

   Powered by Vultr AgentBeta

   Your output should be similar to the one below:

   102.168.0.200

2. Test access to the application on port 8080 using the Curl utility.

   CONSOLE

   Copy

   $ curl http://192.168.0.200:8080

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   Hello, World!

3. To generate more HTTP traffic, invoke the application endpoint using the for loop.

   CONSOLE

   Copy

   $ for i in {1..100}; do curl http://192.168.0.200:8080/ & done; wait

   Explain Code

   Powered by Vultr AgentBeta

4. To simulate an error scenario, invoke a different endpoint such as error.

   CONSOLE

   Copy

   $ curl http://192.168.0.200:8080/error

   Explain Code

   Powered by Vultr AgentBeta

5. Generate more HTTP traffic for the error scenario using the for loop.

   CONSOLE

   Copy

   $ for i in {1..50}; do curl http://192.168.0.200:8080/error & done; wait

   Explain Code

   Powered by Vultr AgentBeta

Query the Application Logs with Grafana

1. Switch to your web browser session, and access the Grafana dashboard.

2. Within the Explore section, enter the following query in the Log browser field.

   {app="go-web-app", namespace="default"}

   

3. Click Run query to execute the statement and view all application access metrics.

   

Conclusion

You have deployed the Loki stack to a VKE cluster and monitored application logs using Grafana. In the process, you containerized a sample application image to the Vultr Container Registry to experiment with different Loki stack functionalities. For more information and implementation samples, visit the Loki documentation.

More Information

For more information, visit the following resources.

• Grafana documentation
• Promtail documentation
• Install Prometheus and Grafana on Vultr Kubernetes Engine with Prometheus-Operator