How to Improve VKE Cluster Observability and Security Using Kubespy

Introduction

DevOps practices are effective in delivering quality applications and maintaining them because continuous application monitoring and application observability is enforced. Developers and IT administrators can identify bugs before they are exploited by hackers in a production environment.

In Kubernetes, it's important to implement tools that help you observe, and monitor your clusters and applications. Luckily, a variety of observability tools can monitor a cluster's security and performance. These include Kubespy which is a free Kubernetes CLI tool that enables you to track resources such as deployments and services.

Through event tracking, you can observe how a deployment is performing internally and view all events made by the deployment. This is important because you can notice any failed events that require your immediate attention to keep the deployment running.

Many dynamic Kubernetes components can be monitored and regulated for optimal cluster performance, and these include the following.

1. Cost
2. Security
3. Resource consumption
4. Logs
5. Cluster Events

Critical Kubernetes metrics that should be continuously monitored include:

1. CPU and memory requests: CPU cycles and memory are the core resources that a container needs to run. It's important to be aware of the rate at which your containers consume and request these resources. Running short of CPU cycles and memory can cause major service disruptions.
2. Percentage of available pods: It is important to know how many pods are available in your cluster because some nodes may fail to get assigned or scheduled to a node.
3. Persistent volume utilization: To make sure that your application state is maintained you should monitor and make sure that there is enough disk storage to prevent stateful storage disruptions.
4. Network traffic and bandwidth: The Network performance of your cluster has to be monitored for easy identification of any security threats. Also, the bandwidth quota should be scaled in cases of high network requests to keep applications running.

In this article, you will learn how to install Kubespy and use it to track cluster services and deployments. Also, you will learn factors that have to be considered when monitoring various aspects of a Kubernetes cluster.

Why Monitor Kubernetes Resources

Understanding the status and health of your VKE (Vultr Kubernetes Engine) cluster is critical when making essential decisions. The Kubernetes environment is big and forever dynamic, but you have to take into account every single component in your cluster to avoid any ripple effects that can be caused by a security breach.

When observing cluster activities you can view and identify the following in real time:

1. Application or Service Vulnerabilities that can be exploited by attackers.
2. Containers with escalated privileges.
3. Containers consuming more cluster resources than allocated.
4. Weak network protocols and routes.
5. Failed or unauthorized logins.

Being able to track the above events helps you to make informed decisions on when to scale your cluster depending on the amount of user traffic, identify any security backdoors before an attacker exploits them, and reduce any escalated container privileges. Hence, by acquiring valuable cluster data such as deployment and resource metrics, you can keep your cluster healthy.

Kubernetes Resource Execution Statuses

Deployments, pods, and services have a life cycle when executing tasks. These tasks have statuses that indicate whether the task execution is successful or unsuccessful. The lifecycle of a deployment is the most complex, and it involves the following tasks:

1. Application configuration.
2. Container configuration.
3. Pod replica specifications.
4. Deployment strategies.

You have to be able to know when:

1. An ephemeral pod is successfully replaced by the ReplicaSet.

2. The deployment or Replicaset fails in replacing the pods.

   Sometimes deployment tasks may fail due to misconfigurations or crashed synchronous events. You have to know when the application changes roll back or roll on successfully.

3. Hooks are executed.

   Kubernetes hooks are executed when a specific container activity starts. For example, the PostStart executes when a container is created. PreStop executes when a container is terminated.

A container has the following statuses:

1. Running.
2. Waiting.
3. Terminated.

On the other hand, Pods have the following statuses:

1. Pending.
2. Running.
3. Succeeded.
4. Failed.

Kubernetes tracks all of these statuses to determine the required action in case of a failure and move on to the next task.

Install Kubespy

Kubespy is available as a Homebrew package on Linux and macOS machines. To install Kubespy, run the following command:

 $ brew install kubespy

To install Kubespy on Windows, download the latest release file from the official GitHub repository, and run the Kubespy executable file using PowerShell or Command Prompt.

Verify the installed Kubespy version.

 $ kubespy version

Output:

 0.6.1

Your output may differ depending on your installed version.

Example

You can use Kubespy to monitor multiple Kubernetes resources. In this section, you will use Kubespy to track deployments and services in a Vultr Kubernetes Engine (VKE) cluster.

Before you start, make sure you:

• Deploy a Vultr Kubernetes Engine (VKE) Cluster.
• Install Kubectl to access the cluster.

Use Kubespy to Track Deployments

1. Create a new YAML file, and add the following contents to create the example deployment boemo.

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: boemo
      labels:
        app: boemo-app
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: boemo-app
      template:
        metadata:
          labels:
            app: boemo-app
        spec:
          containers:
          - name: server
            image: nginx:1.17
            volumeMounts:
              - name: boemo-app
                mountPath: /usr/share/nginx/html
            ports:
            - containerPort: 80
              protocol: TCP
            resources:
              requests:
                cpu: 100m
                memory: "128M"
              limits:
                cpu: 100m
                memory: "256M"
            env:
            - name: LOG_LEVEL
              value: "DEBUG"
          volumes:
          - name: boemo-app
            configMap:
              name: boemo-app
              items:
              - key: body
                path: index.html

   Save the file as new-deployment.yaml.

2. Apply the deployment to your cluster.

    $ kubectl apply -f new-deployment.yaml

3. Using Kubespy, track the new deployment.

    $ kubespy trace deploy boemo

   Your output should look like the one below:

    ←[32mADDED←[0m ←[36;1mapps/v1/Deployment←[0m]  default/boemo
    ←[1m    Rolling out Deployment revision 1
    ←[0m    ❌ Deployment is failing; 0 out of 0 Pods are available: [MinimumReplicasUnavailable] Deployment does not have minimum availability.
        ⌛ Rollout proceeding: [ReplicaSetUpdated] ReplicaSet "boemo-59d4567b6b" is progressing.
   
    ←[36;1mROLLOUT STATUS:nt controller to create ReplicaSet
    ←[0m- [←[33;1mCurrent rollout←[0m | Revision 1] [←[32mADDED←[0m]  default/boemo-59d4567b6b
        ⌛ Waiting for ReplicaSet to attain minimum available Pods (0 available of a 1 minimum)
           - [←[31;1mContainersNotReady←[0m] ←[36mboemo-59d4567b6b-6stcn←[0m containers with unready status: [server]

As displayed in the above output, the deployment fails because there are no available pods in the cluster before proceeding to the ReplicaSet which also fails.

Track Services using Kubespy

1. Create a new YAML file, and add the following contents to create the example service my-service.

    apiVersion: v1
    kind: Service
    metadata:
      name: my-service
      labels:
        app: nginx
    spec:
      externalTrafficPolicy: Local
      ports:
      - name: http
        port: 80
        protocol: TCP
        targetPort: 80
      selector:    
        app: nginx
      type: LoadBalancer

   Save the file as service.yaml.

2. Apply the service to your cluster.

    $ kubectl apply -f service.yaml

3. Using Kubespy, track the new service.

    $ kubespy trace service my-service

   Your Output should look like the one below:

    [←[32mADDED←[0m ←[36;1mv1/Service←[0m]  default/my-service
        ✅ Successfully created Endpoints object 'my-service' to direct traffic to Pods
        ❌ Waiting for public IP/host to be allocated
   
    [←[32mADDED←[0m ←[36;1mv1/Endpoints←[0m]  default/my-service
        ❌ Directs traffic to the following live Pods:
           - [←[31;1mNot live←[0m] ←[36mnginx-deployment-f7ccf9478-4zlg8←[0m @ ←[33m172.17.0.7←[0m
           - [←[31;1mNot live←[0m] ←[36mnginx-deployment-f7ccf9478-cc4q2←[0m @ ←[33m172.17.0.8←[0m
           - [←[31;1mNot live←[0m] ←[36mnginx-deployment-f7ccf9478-snlkn←[0m @ ←[33m172.17.0.6←[0m

   As displayed in the output, endpoints are successfully created, but the service is failing to direct traffic to the pods.

More Information

Kubernetes observability is a vast and complex concept, you need more than one tool to achieve full cluster observability. Below are Kubespy alternative tools you can add to your Kubernetes environment to improve cluster observability.

• Install Prometheus and Grafana on Vultr Kubernetes Engine with Prometheus-Operator.
• How to Set Kubernetes Resource Requests and Limits.
• Kubernetes Dashboard.
• ELK Stack.
• Eliminate Kubernetes Cluster Insecurities with Kubescape.

For more information about Kubespy, visit the official project documentation.