Best Practices for Kubernetes Security

Published on January 12, 2024•Updated on January 24, 2024

Introduction

Kubernetes is an open-source container orchestration system designed to automate deployments, scale, and manage applications. Similar to all cloud-powered systems, Kubernetes clusters are open to active security threats that evolve over time. Implementing strong security measures safeguards your cluster services and deployments from common threats that may lead to data loss or corruption.

Cluster deployments are commonly exposed to multiple threats that occur at different levels. For example, vulnerable container images can create an entry point for attackers to other cluster resources. On the other hand, unrestricted network traffic allows attackers to access private network ports and open backdoors to your cluster applications.

This article explains the best practices for Kubernetes security you can implement to protect Pods, Services, Deployments and other resources in your cluster.

Prerequisites

Before you begin:

Limit Pod Privileges

Security contexts in Kubernetes allow you to define privileges for individual pods or containers. For example, follow the steps below to implement pod-level and container-level contexts.

A pod-level security context applies to all containers with a pod. To test the privilege, create a sample pod that specifies users and a timeout value for running containers.
- Create a new file named security-context-1.yaml.
  console
  $ nano security-context-1.yaml
- Add the following contents to the file.
  yaml
  apiVersion: v1 kind: Pod metadata: name: example-pod spec: securityContext: runAsUser: 2000 runAsGroup: 1000 containers: - name: my-container image: busybox command: [ "sh", "-c", "sleep 3600" ]
  Save and close the file.
  
  The above configuration assigns the Pod processes to the User ID 2000, and the User Group 1000. It uses the busybox container image that runs for 3600 seconds (1 hour). This allows you to control the Pod users, and active container activity.
- Apply the resource to your cluster.
  console
  $ kubectl apply -f security-context-1.yaml
Container-level security contexts apply directly to individual containers within a pod. To test the privilege, create a sample pod that limits process privileges.
- Create a new file named security-context-2.yaml.
  console
  $ nano security-context-2.yaml
- Add the following contents to the file.
  yaml
  apiVersion: v1 kind: Pod metadata: name: example-pod-2 spec: containers: - name: container-1 image: busybox command: [ "sh", "-c", "sleep 3600" ] securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true - name: container-2 image: busybox command: [ "sh", "-c", "sleep 3600" ]
  Save and close the file.
  
  In the above configuration:
  - allowPrivilegeEscalation: Defines how container processes can elevate privileges. When set to false, container processes cannot escalate privileges. When set to true, processes can escalate privileges.
  - readOnlyRootFileSystem: Mounts the container root file system as read-only. In addition, the security contexts only apply to the container container-1 without affecting container-2.
- Apply the pod policy to your cluster
  console
  $ kubectl apply -f security-context-2.yaml

Apply RBAC (Role-Based Access Control) Policies

Role-Based Access Control (RBAC) is a Kubernetes security mechanism for managing user and group permissions in a cluster. Follow the steps in this section to deploy sample roles, a pod, and a new user example_user with restricted access to specific permissions defined in the role configurations.

Create a new namespace resource file my-namespace.yaml.
console
```
$ nano my-namespace.yaml
```
Add the following contents to the file. Replace my-namespace with your desired naming scheme.
yaml
```
apiVersion: v1
kind: Namespace
metadata:
    name: my-namespace
```
Save and close the file.
Apply the namespace resource to your cluster.
console
```
$ kubectl apply -f my-namespace.yaml
```
Create a new role definition file role.yaml.
console
```
$ nano role.yaml
```
A role includes a set of rules that specify what operations a user or group can perform on specific resources within a namespace. Roles declare the authorized actions a user can perform. These include, create, read, update, and delete operations on Kubernetes objects.
Add the following contents to the file to define a new role.
yaml
```
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-role
  namespace: my-namespace
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
```
Within the rules section, below are the applied values:
- resources: Defines the type of resources the role policy affects. The value pods applies the role rules to all Pod resources.
- verbs: Sets the type of operations Role users can perform on the defined resources. These include get, list, watch, logs, among others.
Save and exit the file.
Apply the role to your cluster.
console
```
$ kubectl apply -f role.yaml
```
To apply Role binding, create a new resource file role-binding.yaml.
console
```
$ nano role-binding.yaml
```
Role binding assigns specific roles to users or groups. Each binding must match an active role in your cluster.

Add the following contents to the file to define a new Role binding resource.

                            yaml
                            
                        
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-role-binding
  namespace: my-namespace
subjects:
- kind: User
  name: example_user
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: my-role
  apiGroup: rbac.authorization.k8s.io

Save and close the file.

The above configuration creates a new role-binding for the user example_user with the cluster permissions you defined in the role resource earlier.

Apply role binding to your cluster.
console
```
$ kubectl apply -f role-binding.yaml
```
To test your RBAC policy, create a new Pod resource file my-pod.yaml.
console
```
$ nano my-pod.yaml
```

Add the following contents to the file to define an example Nginx Pod.

                            yaml
                            
                        
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
  namespace: my-namespace
spec:
  containers:
  - name: nginx
    image: nginx:latest

Save and close the file.

Apply the pod to your cluster.
console
```
$ kubectl apply -f my-pod.yaml
```
To test your RBAC Role binding, create a new user example_user using the OpenSSL utility to generate a new private key.
console
```
$ openssl genrsa -out example_user.key 2048
```

Create a Certificate Signing Request (CSR) for the new Kubernetes user account.

                            console
                            
$ openssl req -new -key example_user.key -out example_user.csr -subj "/CN=example_user"

Encode the certificate signing request to base64 and copy the generated base64 string to your clipboard.
console
```
$ cat example_user.csr | base64 | tr -d "\n"
```
Output:
```
LS0tLS1CRUdJTiB...
```
Create a new Certificate resource file csr-resource.yaml.
console
```
$ nano csr-resource.yaml
```

Add the following contents to the file. Add the base64 string you generated earlier as the request field value to create the Kubernetes CertificateSigningRequest (CSR) resource.

                            yaml
                            
                        
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: example_user_csr
spec:
  request: YOUR-BASE64-STRING
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400
  usages:
  - client auth

Save and close the file.

Apply the CSR resource to your cluster.
console
```
$ kubectl apply -f csr-resource.yaml
```

View the list of cluster CSR resources to verify that the new user request is available.

                            console
                            
$ kubectl get csr

Output:

NAME               AGE   SIGNERNAME                            REQUESTOR   REQUESTEDDURATION   CONDITION
example_user_csr   2s    kubernetes.io/kube-apiserver-client   admin       24h                 Pending

Approve the user CSR resource.

                            console
                            
$ kubectl certificate approve example_user_csr

Export the issued certificate from the cluster CSR to a host file such as example_user_signed_cert.crt.

                            console
                            
$ kubectl get csr example_user_csr -o jsonpath='{.status.certificate}'| base64 -d > example_user_signed_cert.crt

In Kubernetes, a context is a specific configuration part of the kubectl utility for interacting with a cluster. To modify users, view your available context to reveal the administrative Kubernetes account in use, and copy the generated output to your clipboard.
console
```
$ kubectl config current-context
```

Add the new example_user credentials to your cluster using your signed certificate and private key files.

                            console
                            
$ kubectl config set-credentials example_user --client-key=example_user.key --client-certificate=example_user_signed_cert.crt --embed-certs=true

Add the new example_user context to your kubectl config. Replace example_cluster with the actual available context name you retrieved earlier.
console
```
$ kubectl config set-context example_user_context --cluster=example_cluster --user=example_user
```
Activate the new context to start using your cluster as the new user example_user.
console
```
$ kubectl config use-context example_user_context
```
To test the new user privileges on the cluster. View all cluster pods in the namespace you created earlier.
console
```
$ kubectl get pods/my-pod -n my-namespace
```
Viewing cluster pods is successful because the user has get permissions for all cluster pod granted by the namespace Role you created earlier.
Try to delete the Nginx pod, and verify that the operation fails due to low privileges.
console
```
$ kubectl delete pods/my-pod -n my-namespace
```
To allow the user to delete pods, you must edit your Role resource and include the delete operation in your rule verbs.
To switch to your privileged cluster account, use the default context you generated earlier. Replace example-context with your default administrative account.
console
```
$ kubectl config use-context example-context
```

You have implemented RBAC using Roles and Role Bindings in a single namespace. To apply cluster wide privileges, use Cluster Roles and Cluster Role Bindings by changing the YAML metadata values.

Implement Network Policies

Network policies allow you to control internal and external network traffic within a cluster. By default, the namespace does not contain any network policies, this enables all ingress and egress traffic between pods in the same namespace.

To tighten cluster network security, create a new network policy resource file default-deny-policy.yaml that disables the default policy.
console
```
$ nano default-deny-policy.yaml
```
Add the following contents to the file. Replace example-namespace with your target namespace.
yaml
```
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: example-namespace
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress
```
Save and exit the file.

The above network policy configuration enforces a default deny rule for ingress and egress traffic on all pods in the example-namespace. This prevents any external or internal traffic from reaching the pods unless additional network policies are created to explicitly allow specific traffic types.
Apply the network policy to your cluster to disable the default traffic policy.
console
```
$ kubectl apply -f default-deny-policy.yaml
```
To allow network traffic between pods, define new network policies depending on your cluster structure. For example, to allow traffic only from a web deployment named nginx to a database deployment named mysql-db within the namespace example-namespace, create a new allow policy resource file allow-policies.yaml.
console
```
$ nano allow-policies.yaml
```

Add the following contents to the file to define two network policies.

                            yaml
                            
                        
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: np-allow-ingress
  namespace: example-namespace
spec:
  podSelector:
    matchLabels:
      app: mysql-db
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: nginx
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: np-allow-egress
  namespace: example-namespace
spec:
  podSelector:
    matchLabels:
      app: nginx
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: mysql-db - Egress

Save and exit the file.

In the above configuration, the first network policy allows ingress traffic. Then, the second policy allows egress traffic from the nginx pod to the mysql-db pod within the namespace example-namespace.

Apply the new network policy to your cluster.
console
```
$ kubectl apply -f allow-policies.yaml
```

You have disabled the default Kubernetes network policy that allows network traffic across all pods, then, you created a separate policy that only allows communication between the nginx and mysql-db pods within the example-namespace resource.

You can apply multiple network policies and also filter traffic using IP blocks, ports, and protocols. For more information, visit the Kubernetes Network Policies documentation.

Manage Kubernetes Secrets

Kubernetes secrets allow you to securely store small amounts of sensitive data like passwords, Vultr Container Registry credentials, service account tokens, and more within a cluster. You can create secrets to define special files and access them within specific pods or deployments. Follow the steps below to create sample secrets and access them in a specific example pod.

Create a new Secret resource file example-secret.yaml.
console
```
$ nano example-secret.yaml
```
Add the following contents to the file to define new dbPassword and apiToken example variables with base64 data.
yaml
```
apiVersion: v1
kind: Secret
metadata:
  name: example-secret
data:
  dbPassword: ZXhhbXBsZVB3ZA==
  apiToken: dG9rZW4=
          app: mysql-db - Egress
```
Save and exit the file.

The above configuration creates a new example-secret with accessible variables you can apply in specific cluster resources.
Apply the secret to your cluster
console
```
$ kubectl apply -f example-secret.yaml
```

Access Secrets in Pods

To access cluster Secrets, call the secret and variable as a key in your resource definition file. For example, create a new Pod resource example-pod.yaml to access your new Secret resource.
console
```
$ nano example-pod.yaml
```

Add the following contents to the file.

                            yaml
                            
                        
apiVersion: v1
kind: Pod
metadata:
  name: example-pod-with-secret
spec:
  containers:
    - name: my-container
      image: busybox
      command: [ "sh", "-c", "sleep 3600" ]
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: example-secret
              key: dbPassword Egress

Save and close the file.

Within the above configuration, the example Pod example-pod-with-secret imports the dbPassword variable data from the example-secret to the Pod variable DB_PASSWORD. This allows the Pod to use the Secret values to authenticate with the local variable using values from the Secret resource.

Apply the Pod to your cluster.
console
```
$ kubectl apply -f example-pod.yaml
```

You have created a Kubernetes Secret resource and accessed the variable keys in a Pod resource. You can create multiple secrets to store important data such as authentication values to pass directly to pods, deployments and services within the cluster depending on the required variable names.

Pod Security Standards (PSSs)

Kubernetes Pod Security Standards (PSS) are key to maintaining Kubernetes security. PSS contains three profiles:

Privileged: An unrestricted policy. Should be used only for trusted users performing critical infrastructure workloads, along with additional security measures such as multifactor authentication.
Baseline: Offers minimal restrictive guardrails while preventing known privilege escalations. Suitable for non-critical applications and typical containerized workloads. It may require different configurations based on specific use cases.
Restricted: Enforces pod hardening best practices, making it the most secure profile. It's ideal for security-critical applications and vulnerable workloads.

You can configure PSS in your namespace with the following different modes:

Enforce: Rejects Pods with policy violations.
Audit: Allows pods with policy violations but includes audit annotations in the event log.
Warn: Allows pods with policy violations but warns users about the pontetial security risks.

You can use a specific profile and mode in a namespace by setting the corresponding labels. For example, the following command sets the example-namespace namespace Baseline warning, and the warn-version label applies the latest policy versions.

                            console
                            
                        
$ kubectl label --overwrite ns example-namespace \
   pod-security.kubernetes.io/warn=baseline \
   pod-security.kubernetes.io/warn-version=latest

The following command sets a Baseline enforcement with an audit for the restricted level on the namespace example-namespace. The audit-version and enforce-version labels apply the latest policy versions.

                            console
                            
                        
$ kubectl label --overwrite ns example-namespace \
   pod-security.kubernetes.io/enforce=baseline \
   pod-security.kubernetes.io/audit=restricted \
   pod-security.kubernetes.io/enforce-version=latest \
   pod-security.kubernetes.io/audit-version=latest

Follow the steps below to set the labels in a namespace resource definition file instead of directly using Kubectl commands.

Create a new namespace resource file example-namespace.yaml.
console
```
$ nano example-namespace.yaml
```

Add the following contents to the file.

                            yaml
                            
                        
apiVersion: v1
kind: Namespace
metadata:
  name: example-namespace
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/enforce-version: latest

    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest

    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest

Save and exit the file.

The above configuration sets a Baseline enforcement. Then, applies the latest policy versions to the Restricted level audits and warnings.

Apply the namespace resource to your cluster.
console
```
$ kubectl apply -f example-namespace.yaml
```

Monitoring and Logging

Monitoring is essential for a smooth Kubernetes cluster. It offers visibility into health, performance, and resource usage. This allows you to identify bottlenecks and ensure efficiency. In this section, verify pod and container logs to view specific resource values.

View the latest Pod logs.
console
```
$ kubectl logs my-pod
```
View the latest logs from a specific container within a pod.
console
```
$ kubectl logs my-pod -c my-container
```
View the latest logs from a specific container and limit the output lines to 50.
console
```
$ kubectl logs my-pod -c my-container --tail=50
```
View Pod logs from a specific container with the respective timestamps.
console
```
$ kubectl logs my-pod -c my-container --timestamps=true
```
View the latest Pod logs with a specific time range. For example, from the last 2 hours.
console
```
$ kubectl logs my-pod -c my-container --since=2h
```

To implement advanced monitoring on your cluster, visit the following resources:

Security Audits and Penetration Testing

Regular security audits and penetration testing are important when maintaining cluster security. Follow the steps below to audit resource files using the kubeaudit tool that allows you to perform security audits.

Download the latest kubeaudit binary to your machine.
console
```
$ wget https://github.com/Shopify/kubeaudit/releases/download/v0.22.0/kubeaudit_0.22.0_linux_amd64.tar.gz
```
This article uses version 0.22.0, to verify the latest version, visit the application releases page.

Extract files from the archive.

                            console
                            
$ tar -xzvf kubeaudit_0.22.0_linux_amd64.tar.gz

Verify the Kubeaudit tool version
console
```
$ ./kubeaudit version
```
To access the tool as a system-wide package, move it to the /usr/local/bin/ directory and define a new environment variable to run the tool on your server.
Create a new pod resource file busybox-pod.yaml to define an example busybox Pod with a time-to-live of 1 hour.
console
```
$ nano busybox-pod.yaml
```

Add the following contents to the file.

                            yaml
                            
                        
apiVersion: v1
kind: Pod
metadata:
  name: busybox-pod
spec:
  containers:
  - name: my-container
    image: busybox
    command: ["sh", "-c", "sleep 3600"]

Save and exit the file.

Using Kubeaudit, scan the busybox pod manifest file for security threats.
console
```
$ ./kubeaudit all -f "busybox-pod.yaml"
```
To fix any detected security issues, use the autofix option to generate a new busybox-pod-fixed.yaml file.
console
```
$ ./kubeaudit autofix -f "busybox-pod.yaml" -o "busybox-pod-fixed.yaml"
```

To implement advanced scanning and secure your cluster resources, visit the following resources:

Conclusion

You have implemented Kubernetes security practices using example resources to secure a VKE cluster. Visit the Kubernetes security documentation to explore more tools and policies you can apply in your cluster.

Best Practices for Kubernetes Security

Introduction

Prerequisites

Limit Pod Privileges

Apply RBAC (Role-Based Access Control) Policies

Implement Network Policies

Manage Kubernetes Secrets

Access Secrets in Pods

Pod Security Standards (PSSs)

Monitoring and Logging

Security Audits and Penetration Testing

Conclusion

Products

Features

Marketplace

Resources

Company