Best Practices for Kubernetes Security
![Best Practices for Kubernetes Security header image](https://docs.vultr.com/public/doc-assets/1728/d6aaa01be207058a.png)
Introduction
Kubernetes is an open-source container orchestration system designed to automate deployments, scale, and manage applications. Similar to all cloud-powered systems, Kubernetes clusters are open to active security threats that evolve over time. Implementing strong security measures safeguards your cluster services and deployments from common threats that may lead to data loss or corruption.
Cluster deployments are commonly exposed to multiple threats that occur at different levels. For example, vulnerable container images can create an entry point for attackers to other cluster resources. On the other hand, unrestricted network traffic allows attackers to access private network ports and open backdoors to your cluster applications.
This article explains the best practices for Kubernetes security you can implement to protect Pods, Services, Deployments and other resources in your cluster.
Prerequisites
Before you begin:
- Deploy a Vultr Kubernetes Engine cluster.
- Deploy a Vultr Ubuntu server to use as the management machine.
- Access the server using SSH as a non-root user with sudo privileges.
- Install and Configure Kubectl to access the cluster.
Limit Pod Privileges
Security contexts in Kubernetes allow you to define privileges for individual pods or containers. For example, follow the steps below to implement pod-level and container-level contexts.
A pod-level security context applies to all containers with a pod. To test the privilege, create a sample pod that specifies users and a timeout value for running containers.
Create a new file named
security-context-1.yaml
.console$ nano security-context-1.yaml
Add the following contents to the file.
yamlapiVersion: v1 kind: Pod metadata: name: example-pod spec: securityContext: runAsUser: 2000 runAsGroup: 1000 containers: - name: my-container image: busybox command: [ "sh", "-c", "sleep 3600" ]
Save and close the file.
The above configuration assigns the Pod processes to the User ID
2000
, and the User Group1000
. It uses thebusybox
container image that runs for3600
seconds (1 hour). This allows you to control the Pod users, and active container activity.Apply the resource to your cluster.
console$ kubectl apply -f security-context-1.yaml
Container-level security contexts apply directly to individual containers within a pod. To test the privilege, create a sample pod that limits process privileges.
Create a new file named
security-context-2.yaml
.console$ nano security-context-2.yaml
Add the following contents to the file.
yamlapiVersion: v1 kind: Pod metadata: name: example-pod-2 spec: containers: - name: container-1 image: busybox command: [ "sh", "-c", "sleep 3600" ] securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true - name: container-2 image: busybox command: [ "sh", "-c", "sleep 3600" ]
Save and close the file.
In the above configuration:
allowPrivilegeEscalation
: Defines how container processes can elevate privileges. When set tofalse
, container processes cannot escalate privileges. When set totrue
, processes can escalate privileges.readOnlyRootFileSystem
: Mounts the container root file system as read-only. In addition, the security contexts only apply to the containercontainer-1
without affectingcontainer-2
.
Apply the pod policy to your cluster
console$ kubectl apply -f security-context-2.yaml
Apply RBAC (Role-Based Access Control) Policies
Role-Based Access Control (RBAC) is a Kubernetes security mechanism for managing user and group permissions in a cluster. Follow the steps in this section to deploy sample roles, a pod, and a new user example_user
with restricted access to specific permissions defined in the role configurations.
Create a new namespace resource file
my-namespace.yaml
.console$ nano my-namespace.yaml
Add the following contents to the file. Replace
my-namespace
with your desired naming scheme.yamlapiVersion: v1 kind: Namespace metadata: name: my-namespace
Save and close the file.
Apply the namespace resource to your cluster.
console$ kubectl apply -f my-namespace.yaml
Create a new role definition file
role.yaml
.console$ nano role.yaml
A role includes a set of rules that specify what operations a user or group can perform on specific resources within a namespace. Roles declare the authorized actions a user can perform. These include, create, read, update, and delete operations on Kubernetes objects.
Add the following contents to the file to define a new role.
yamlapiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: my-role namespace: my-namespace rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"]
Within the rules section, below are the applied values:
resources
: Defines the type of resources the role policy affects. The valuepods
applies the role rules to all Pod resources.verbs
: Sets the type of operations Role users can perform on the defined resources. These includeget
,list
,watch
,logs
, among others.
Save and exit the file.
Apply the role to your cluster.
console$ kubectl apply -f role.yaml
To apply Role binding, create a new resource file
role-binding.yaml
.console$ nano role-binding.yaml
Role binding assigns specific roles to users or groups. Each binding must match an active role in your cluster.
Add the following contents to the file to define a new Role binding resource.
yamlapiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: my-role-binding namespace: my-namespace subjects: - kind: User name: example_user apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: my-role apiGroup: rbac.authorization.k8s.io
Save and close the file.
The above configuration creates a new role-binding for the user
example_user
with the cluster permissions you defined in the role resource earlier.Apply role binding to your cluster.
console$ kubectl apply -f role-binding.yaml
To test your RBAC policy, create a new Pod resource file
my-pod.yaml
.console$ nano my-pod.yaml
Add the following contents to the file to define an example Nginx Pod.
yamlapiVersion: v1 kind: Pod metadata: name: my-pod namespace: my-namespace spec: containers: - name: nginx image: nginx:latest
Save and close the file.
Apply the pod to your cluster.
console$ kubectl apply -f my-pod.yaml
To test your RBAC Role binding, create a new user
example_user
using the OpenSSL utility to generate a new private key.console$ openssl genrsa -out example_user.key 2048
Create a Certificate Signing Request (CSR) for the new Kubernetes user account.
console$ openssl req -new -key example_user.key -out example_user.csr -subj "/CN=example_user"
Encode the certificate signing request to base64 and copy the generated base64 string to your clipboard.
console$ cat example_user.csr | base64 | tr -d "\n"
Output:
LS0tLS1CRUdJTiB...
Create a new Certificate resource file
csr-resource.yaml
.console$ nano csr-resource.yaml
Add the following contents to the file. Add the base64 string you generated earlier as the
request
field value to create the Kubernetes CertificateSigningRequest (CSR) resource.yamlapiVersion: certificates.k8s.io/v1 kind: CertificateSigningRequest metadata: name: example_user_csr spec: request: YOUR-BASE64-STRING signerName: kubernetes.io/kube-apiserver-client expirationSeconds: 86400 usages: - client auth
Save and close the file.
Apply the CSR resource to your cluster.
console$ kubectl apply -f csr-resource.yaml
View the list of cluster CSR resources to verify that the new user request is available.
console$ kubectl get csr
Output:
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION example_user_csr 2s kubernetes.io/kube-apiserver-client admin 24h Pending
Approve the user CSR resource.
console$ kubectl certificate approve example_user_csr
Export the issued certificate from the cluster CSR to a host file such as
example_user_signed_cert.crt
.console$ kubectl get csr example_user_csr -o jsonpath='{.status.certificate}'| base64 -d > example_user_signed_cert.crt
In Kubernetes, a context is a specific configuration part of the kubectl utility for interacting with a cluster. To modify users, view your available context to reveal the administrative Kubernetes account in use, and copy the generated output to your clipboard.
console$ kubectl config current-context
Add the new
example_user
credentials to your cluster using your signed certificate and private key files.console$ kubectl config set-credentials example_user --client-key=example_user.key --client-certificate=example_user_signed_cert.crt --embed-certs=true
Add the new
example_user
context to your kubectl config. Replaceexample_cluster
with the actual available context name you retrieved earlier.console$ kubectl config set-context example_user_context --cluster=example_cluster --user=example_user
Activate the new context to start using your cluster as the new user
example_user
.console$ kubectl config use-context example_user_context
To test the new user privileges on the cluster. View all cluster pods in the namespace you created earlier.
console$ kubectl get pods/my-pod -n my-namespace
Viewing cluster pods is successful because the user has
get
permissions for all cluster pod granted by the namespace Role you created earlier.Try to delete the Nginx pod, and verify that the operation fails due to low privileges.
console$ kubectl delete pods/my-pod -n my-namespace
To allow the user to delete pods, you must edit your Role resource and include the
delete
operation in your rule verbs.To switch to your privileged cluster account, use the default context you generated earlier. Replace
example-context
with your default administrative account.console$ kubectl config use-context example-context
You have implemented RBAC using Roles and Role Bindings in a single namespace. To apply cluster wide privileges, use Cluster Roles and Cluster Role Bindings by changing the YAML metadata values.
Implement Network Policies
Network policies allow you to control internal and external network traffic within a cluster. By default, the namespace does not contain any network policies, this enables all ingress and egress traffic between pods in the same namespace.
To tighten cluster network security, create a new network policy resource file
default-deny-policy.yaml
that disables the default policy.console$ nano default-deny-policy.yaml
Add the following contents to the file. Replace
example-namespace
with your target namespace.yamlapiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-ingress namespace: example-namespace spec: podSelector: {} policyTypes: - Ingress - Egress
Save and exit the file.
The above network policy configuration enforces a default deny rule for ingress and egress traffic on all pods in the
example-namespace
. This prevents any external or internal traffic from reaching the pods unless additional network policies are created to explicitly allow specific traffic types.Apply the network policy to your cluster to disable the default traffic policy.
console$ kubectl apply -f default-deny-policy.yaml
To allow network traffic between pods, define new network policies depending on your cluster structure. For example, to allow traffic only from a web deployment named
nginx
to a database deployment namedmysql-db
within the namespaceexample-namespace
, create a new allow policy resource fileallow-policies.yaml
.console$ nano allow-policies.yaml
Add the following contents to the file to define two network policies.
yamlapiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: np-allow-ingress namespace: example-namespace spec: podSelector: matchLabels: app: mysql-db policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: app: nginx --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: np-allow-egress namespace: example-namespace spec: podSelector: matchLabels: app: nginx policyTypes: - Egress egress: - to: - podSelector: matchLabels: app: mysql-db - Egress
Save and exit the file.
In the above configuration, the first network policy allows ingress traffic. Then, the second policy allows egress traffic from the
nginx
pod to themysql-db
pod within the namespaceexample-namespace
.Apply the new network policy to your cluster.
console$ kubectl apply -f allow-policies.yaml
You have disabled the default Kubernetes network policy that allows network traffic across all pods, then, you created a separate policy that only allows communication between the nginx
and mysql-db
pods within the example-namespace
resource.
You can apply multiple network policies and also filter traffic using IP blocks, ports, and protocols. For more information, visit the Kubernetes Network Policies documentation.
Manage Kubernetes Secrets
Kubernetes secrets allow you to securely store small amounts of sensitive data like passwords, Vultr Container Registry credentials, service account tokens, and more within a cluster. You can create secrets to define special files and access them within specific pods or deployments. Follow the steps below to create sample secrets and access them in a specific example pod.
Create a new Secret resource file
example-secret.yaml
.console$ nano example-secret.yaml
Add the following contents to the file to define new
dbPassword
andapiToken
example variables with base64 data.yamlapiVersion: v1 kind: Secret metadata: name: example-secret data: dbPassword: ZXhhbXBsZVB3ZA== apiToken: dG9rZW4= app: mysql-db - Egress
Save and exit the file.
The above configuration creates a new
example-secret
with accessible variables you can apply in specific cluster resources.Apply the secret to your cluster
console$ kubectl apply -f example-secret.yaml
Access Secrets in Pods
To access cluster Secrets, call the secret and variable as a
key
in your resource definition file. For example, create a new Pod resourceexample-pod.yaml
to access your new Secret resource.console$ nano example-pod.yaml
Add the following contents to the file.
yamlapiVersion: v1 kind: Pod metadata: name: example-pod-with-secret spec: containers: - name: my-container image: busybox command: [ "sh", "-c", "sleep 3600" ] env: - name: DB_PASSWORD valueFrom: secretKeyRef: name: example-secret key: dbPassword Egress
Save and close the file.
Within the above configuration, the example Pod
example-pod-with-secret
imports thedbPassword
variable data from theexample-secret
to the Pod variableDB_PASSWORD
. This allows the Pod to use the Secret values to authenticate with the local variable using values from the Secret resource.Apply the Pod to your cluster.
console$ kubectl apply -f example-pod.yaml
You have created a Kubernetes Secret resource and accessed the variable keys in a Pod resource. You can create multiple secrets to store important data such as authentication values to pass directly to pods, deployments and services within the cluster depending on the required variable names.
Pod Security Standards (PSSs)
Kubernetes Pod Security Standards (PSS) are key to maintaining Kubernetes security. PSS contains three profiles:
- Privileged: An unrestricted policy. Should be used only for trusted users performing critical infrastructure workloads, along with additional security measures such as multifactor authentication.
- Baseline: Offers minimal restrictive guardrails while preventing known privilege escalations. Suitable for non-critical applications and typical containerized workloads. It may require different configurations based on specific use cases.
- Restricted: Enforces pod hardening best practices, making it the most secure profile. It's ideal for security-critical applications and vulnerable workloads.
You can configure PSS in your namespace with the following different modes:
- Enforce: Rejects Pods with policy violations.
- Audit: Allows pods with policy violations but includes audit annotations in the event log.
- Warn: Allows pods with policy violations but warns users about the pontetial security risks.
You can use a specific profile and mode in a namespace by setting the corresponding labels. For example, the following command sets the example-namespace
namespace Baseline warning, and the warn-version
label applies the latest policy versions.
$ kubectl label --overwrite ns example-namespace \
pod-security.kubernetes.io/warn=baseline \
pod-security.kubernetes.io/warn-version=latest
The following command sets a Baseline enforcement with an audit for the restricted level on the namespace example-namespace
. The audit-version
and enforce-version
labels apply the latest policy versions.
$ kubectl label --overwrite ns example-namespace \
pod-security.kubernetes.io/enforce=baseline \
pod-security.kubernetes.io/audit=restricted \
pod-security.kubernetes.io/enforce-version=latest \
pod-security.kubernetes.io/audit-version=latest
Follow the steps below to set the labels in a namespace resource definition file instead of directly using Kubectl commands.
Create a new namespace resource file
example-namespace.yaml
.console$ nano example-namespace.yaml
Add the following contents to the file.
yamlapiVersion: v1 kind: Namespace metadata: name: example-namespace labels: pod-security.kubernetes.io/enforce: baseline pod-security.kubernetes.io/enforce-version: latest pod-security.kubernetes.io/audit: restricted pod-security.kubernetes.io/audit-version: latest pod-security.kubernetes.io/warn: restricted pod-security.kubernetes.io/warn-version: latest
Save and exit the file.
The above configuration sets a Baseline enforcement. Then, applies the latest policy versions to the Restricted level audits and warnings.
Apply the namespace resource to your cluster.
console$ kubectl apply -f example-namespace.yaml
Monitoring and Logging
Monitoring is essential for a smooth Kubernetes cluster. It offers visibility into health, performance, and resource usage. This allows you to identify bottlenecks and ensure efficiency. In this section, verify pod and container logs to view specific resource values.
View the latest Pod logs.
console$ kubectl logs my-pod
View the latest logs from a specific container within a pod.
console$ kubectl logs my-pod -c my-container
View the latest logs from a specific container and limit the output lines to
50
.console$ kubectl logs my-pod -c my-container --tail=50
View Pod logs from a specific container with the respective timestamps.
console$ kubectl logs my-pod -c my-container --timestamps=true
View the latest Pod logs with a specific time range. For example, from the last 2 hours.
console$ kubectl logs my-pod -c my-container --since=2h
To implement advanced monitoring on your cluster, visit the following resources:
- Install Prometheus and Grafana on Vultr Kubernetes Engine with Prometheus-Operator
- How to Monitor Your VKE Cluster with tobs
Security Audits and Penetration Testing
Regular security audits and penetration testing are important when maintaining cluster security. Follow the steps below to audit resource files using the kubeaudit
tool that allows you to perform security audits.
Download the latest
kubeaudit
binary to your machine.console$ wget https://github.com/Shopify/kubeaudit/releases/download/v0.22.0/kubeaudit_0.22.0_linux_amd64.tar.gz
This article uses version
0.22.0
, to verify the latest version, visit the application releases page.Extract files from the archive.
console$ tar -xzvf kubeaudit_0.22.0_linux_amd64.tar.gz
Verify the Kubeaudit tool version
console$ ./kubeaudit version
To access the tool as a system-wide package, move it to the
/usr/local/bin/
directory and define a new environment variable to run the tool on your server.Create a new pod resource file
busybox-pod.yaml
to define an examplebusybox
Pod with a time-to-live of 1 hour.console$ nano busybox-pod.yaml
Add the following contents to the file.
yamlapiVersion: v1 kind: Pod metadata: name: busybox-pod spec: containers: - name: my-container image: busybox command: ["sh", "-c", "sleep 3600"]
Save and exit the file.
Using Kubeaudit, scan the busybox pod manifest file for security threats.
console$ ./kubeaudit all -f "busybox-pod.yaml"
To fix any detected security issues, use the
autofix
option to generate a newbusybox-pod-fixed.yaml
file.console$ ./kubeaudit autofix -f "busybox-pod.yaml" -o "busybox-pod-fixed.yaml"
To implement advanced scanning and secure your cluster resources, visit the following resources:
- How To Secure Kubernetes deployments using Trivy-Operator
- How to Execute Deployments Safely Using Kube-bench
- Eliminate Kubernetes Cluster Insecurities with Kubescape
Conclusion
You have implemented Kubernetes security practices using example resources to secure a VKE cluster. Visit the Kubernetes security documentation to explore more tools and policies you can apply in your cluster.