How To Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Kubernetes

The Elastic Stack (commonly known as the ELK Stack) is a collection of open-source tools for searching, analyzing, and visualizing log data in real time. It consists of Elasticsearch, Logstash, Kibana, and Beats (including Filebeat), which work together to collect, process, store, and visualize logs from applications and infrastructure. Organizations use the Elastic Stack to centralize logging, monitor system performance, detect security threats, and gain actionable insights from their data.

This article outlines the deployment of the ELK Stack on Kubernetes using the Elastic Cloud on Kubernetes (ECK) operator. It covers deploying Elasticsearch, Logstash, Kibana, and Filebeat, securing access with Traefik and Let's Encrypt SSL certificates, and visualizing cluster logs through the Kibana dashboard.

Prerequisites

Before you begin, you need to:

Have a Kubernetes cluster with at least 3 nodes, each with a minimum of 4 GB RAM. Actual requirements vary based on the number of log sources and ingestion volume.
Install and configure kubectl on your workstation to access your Kubernetes cluster.
Install Helm on your workstation.
Have a registered domain name for accessing Kibana (for example, kibana.example.com).

Install Elasticsearch

Elasticsearch is a distributed, RESTful search and analytics engine that stores and indexes log data for fast retrieval. It serves as the central data store for the Elastic Stack, enabling full-text search, structured queries, and aggregations across large volumes of log data.

The Elastic Cloud on Kubernetes (ECK) operator simplifies deploying and managing Elasticsearch on Kubernetes by handling cluster configuration, scaling, and upgrades automatically.

Install the Elastic operator custom resource definitions (CRDs). Replace 3.4.0 with the latest stable version from the ECK releases page.
console
```
$ kubectl create -f https://download.elastic.co/downloads/eck/3.4.0/crds.yaml
```
Install the Elastic operator.
console
```
$ kubectl apply -f https://download.elastic.co/downloads/eck/3.4.0/operator.yaml
```
The output confirms the operator resources are created including the namespace, service account, and statefulset.
List all pods in the elastic-system namespace and verify that the Elastic Operator is running.
console
```
$ kubectl get -n elastic-system pods
```
The output shows the elastic-operator-0 pod with Running status.
Create a new elasticsearch.yaml file.
console
```
$ nano elasticsearch.yaml
```
Add the following YAML contents to the file.
yaml
```
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 9.4.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 40Gi
        storageClassName: vultr-block-storage-hdd
```
Save and close the file.

The above configuration creates an Elasticsearch cluster with the following settings:
- version: Specifies the Elasticsearch version to deploy.
- nodeSets: Defines the cluster topology with one node in the default node set.
- config.node.store.allow_mmap: Disables memory-mapped files to avoid permission issues in containerized environments.
- volumeClaimTemplates: Configures persistent storage for Elasticsearch data. The storageClassName must match your cluster's available storage class. For Vultr Kubernetes Engine (VKE), use vultr-block-storage-hdd. Run kubectl get storageclass to list available storage classes for your provider.
Apply the elasticsearch.yaml configuration to your cluster.
console
```
$ kubectl apply -f elasticsearch.yaml
```
Wait at least 2 minutes, then list the Elasticsearch pods and verify that they are running.
console
```
$ kubectl get pods -l elasticsearch.k8s.elastic.co/cluster-name=quickstart
```
The output shows the quickstart-es-default-0 pod with Running status.
Get the Elasticsearch service status and verify the assigned cluster IP.
console
```
$ kubectl get service quickstart-es-http
```
The output displays the ClusterIP service listening on port 9200.
List all Persistent Volume Claims (PVCs) and verify that the Elasticsearch volume is bound.
console
```
$ kubectl get pvc
```
The output shows the elasticsearch-data-quickstart-es-default-0 PVC with Bound status.
Monitor the Elasticsearch health and verify that the cluster is ready.
console
```
$ kubectl get elasticsearch
```
The output shows the cluster with Ready phase. With a single-node setup, the health may show yellow because replica shards cannot be placed. This is expected. If the health shows red, verify that the pods, service, and PVC resources are running and bound correctly.
Retrieve and store the generated password for the elastic user.
console
```
$ PASSWORD=$(kubectl get secret quickstart-es-elastic-user -o go-template='{{.data.elastic | base64decode}}')
$ echo $PASSWORD
```
Copy the generated password from the output. You need this password to authenticate with Elasticsearch and Kibana.

Install Logstash

Logstash is a server-side data processing pipeline that ingests, transforms, and forwards logs to Elasticsearch. It receives logs from various sources (including Filebeat), applies filters to parse and enrich the data, and sends the processed logs to Elasticsearch for storage and indexing.

Create a new logstash.yaml file.
console
```
$ nano logstash.yaml
```

Add the following YAML contents to the file.

                            yaml
                            
                        
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: quickstart
spec:
  version: 9.4.0
  count: 1
  elasticsearchRefs:
  - name: quickstart
    clusterName: quickstart
  pipelines:
  - pipeline.id: main
    config.string: |
      input {
        beats {
          port => 5044
        }
      }
      filter {
        if [kubernetes][namespace] {
          mutate {
            add_field => { "environment" => "%{[kubernetes][namespace]}" }
          }
        }
      }
      output {
        elasticsearch {
          hosts => [ "${QUICKSTART_ES_HOSTS}" ]
          user => "${QUICKSTART_ES_USER}"
          password => "${QUICKSTART_ES_PASSWORD}"
          ssl_certificate_authorities => "${QUICKSTART_ES_SSL_CERTIFICATE_AUTHORITY}"
          index => "logstash-%{+YYYY.MM.dd}"
        }
      }
  services:
  - name: beats
    service:
      spec:
        type: ClusterIP
        ports:
        - port: 5044
          name: beats
          protocol: TCP
          targetPort: 5044
  volumeClaimTemplates:
  - metadata:
      name: logstash-data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 40Gi
      storageClassName: vultr-block-storage-hdd

Save and close the file.

The above configuration creates a Logstash instance with the following settings:

elasticsearchRefs: Links Logstash to the Elasticsearch cluster named quickstart. The ECK operator automatically injects connection credentials as environment variables.
pipelines: Defines the data processing pipeline:
- input.beats: Configures Logstash to receive logs from Filebeat on port 5044.
- filter.mutate: Adds an environment field to each log entry based on the Kubernetes namespace, enabling filtering by namespace in Kibana.
- output.elasticsearch: Sends processed logs to Elasticsearch using credentials injected by ECK. Logs are indexed with a daily pattern (logstash-YYYY.MM.dd).
services: Exposes a ClusterIP service on port 5044 for Filebeat to send logs.
volumeClaimTemplates: Configures persistent storage for Logstash data and queue persistence. For Vultr Kubernetes Engine (VKE), use vultr-block-storage-hdd. Adjust the storageClassName based on your cluster's storage provider.

Apply the logstash.yaml configuration to install Logstash in your cluster.
console
```
$ kubectl apply -f logstash.yaml
```
List the Logstash pods and verify that they are running.
console
```
$ kubectl get pods -l logstash.k8s.elastic.co/name=quickstart
```
The output shows the quickstart-ls-0 pod with Running status.
Monitor the Logstash health and verify that it is ready.
console
```
$ kubectl get logstash
```
The output shows green health and 1/1 available replicas.

Install Filebeat

Filebeat is a lightweight log shipper that runs as a DaemonSet on every node in the Kubernetes cluster. It collects container logs from /var/log/containers/ and ships them to Logstash or Elasticsearch for processing. Filebeat automatically enriches logs with Kubernetes metadata such as pod name, namespace, and labels, making it easier to filter and analyze logs in Kibana.

Create a new filebeat.yaml file.
console
```
$ nano filebeat.yaml
```

Add the following YAML contents to the file.

                            yaml
                            
                        
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: quickstart
spec:
  type: filebeat
  version: 9.4.0
  config:
    filebeat.inputs:
    - type: filestream
      id: kubernetes-container-logs
      paths:
      - /var/log/containers/*.log
      parsers:
      - container: {}
      prospector.scanner.symlinks: true
      processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
          - logs_path:
              logs_path: /var/log/containers/
      - drop_event.when:
          or:
          - equals:
              kubernetes.namespace: "kube-system"
          - equals:
              kubernetes.namespace: "kube-public"
          - equals:
              kubernetes.namespace: "elastic-system"
          - equals:
              kubernetes.namespace: "kube-node-lease"
    output.logstash:
      hosts: ["quickstart-ls-beats:5044"]
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        containers:
        - name: filebeat
          env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          securityContext:
            runAsUser: 0
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

Save and close the file.

The above configuration deploys Filebeat as a DaemonSet with the following settings:

spec.type: Specifies Filebeat as the Beat type.
config.filebeat.inputs: Configures log collection:
- type: filestream: Uses the filestream input to read container log files.
- paths: Collects all container logs from /var/log/containers/.
- parsers.container: Parses container log format automatically.
- prospector.scanner.symlinks: Follows symbolic links to locate actual log files.
processors:
- add_kubernetes_metadata: Enriches each log entry with Kubernetes metadata (pod name, namespace, labels) based on the log file path.
- drop_event.when: Filters out logs from system namespaces (kube-system, kube-public, elastic-system, kube-node-lease) to reduce noise.
output.logstash: Sends collected logs to Logstash on port 5044 for processing before indexing in Elasticsearch. Internal cluster communication over ClusterIP does not require additional SSL configuration.
daemonSet.podTemplate: Configures the Filebeat pods:
- hostNetwork: true: Allows Filebeat to access node-level network for metadata enrichment.
- volumes: Mounts host paths to access container log files.
- securityContext.runAsUser: 0: Runs as root to read log files with restricted permissions.

Create a new filebeat-rbac.yaml file to define the Role-Based Access Control (RBAC) configuration for Filebeat.
console
```
$ nano filebeat-rbac.yaml
```

Add the following YAML contents to the file.

                            yaml
                            
                        
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
rules:
- apiGroups: [""]
  resources: [namespaces, pods, nodes]
  verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: default
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

Save and close the file.

The above configuration creates RBAC resources that allow Filebeat to:

ServiceAccount: Provides an identity for Filebeat pods.
ClusterRole: Grants read-only access to namespaces, pods, and nodes across the cluster for metadata enrichment.
ClusterRoleBinding: Binds the ClusterRole to the Filebeat ServiceAccount.

Apply the configurations to install Filebeat in your cluster.
console
```
$ kubectl apply -f filebeat-rbac.yaml
$ kubectl apply -f filebeat.yaml
```
List the Filebeat pods and verify that they are running on all nodes.
console
```
$ kubectl get pods -A -l common.k8s.elastic.co/type=beat
```
The output shows one Filebeat pod per node with Running status.
Monitor the Filebeat health and verify that it is ready.
console
```
$ kubectl get beat
```
The output shows green health with the expected number of available replicas matching your node count.

Install and Configure Kibana Dashboards

Kibana is the visualization and exploration interface for the Elastic Stack. It provides a web-based dashboard for searching, viewing, and analyzing logs stored in Elasticsearch. Kibana supports interactive charts, graphs, and maps, enabling real-time monitoring and troubleshooting of applications and infrastructure.

Create a new kibana.yaml file.
console
```
$ nano kibana.yaml
```
Add the following YAML contents to the file.
yaml
```
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: quickstart
  namespace: default
spec:
  version: 9.4.0
  count: 1
  elasticsearchRef:
    name: quickstart
  http:
    tls:
      selfSignedCertificate:
        disabled: true
```
Save and close the file.

The above configuration deploys Kibana with the following settings:
- elasticsearchRef: Connects Kibana to the Elasticsearch cluster named quickstart. ECK automatically configures authentication credentials.
- http.tls.selfSignedCertificate.disabled: Disables the self-signed certificate since Traefik handles TLS termination with Let's Encrypt certificates.
Apply the kibana.yaml file to install Kibana in your cluster.
console
```
$ kubectl apply -f kibana.yaml
```
List all Kibana pods and verify that they are running.
console
```
$ kubectl get pods --selector='kibana.k8s.elastic.co/name=quickstart'
```
The output shows the quickstart-kb-* pod with Running status.
Get the Kibana service status and verify the assigned cluster IP.
console
```
$ kubectl get service quickstart-kb-http
```
The output displays the ClusterIP service listening on port 5601.
Monitor the Kibana health and verify that it is ready.
console
```
$ kubectl get kibana
```
The output shows green health and Ready status.

Secure the ELK Stack

The Kibana dashboard requires secure TLS connections for production use. Traefik functions as an Ingress controller to route external traffic to Kibana, while cert-manager automatically provisions and renews Let's Encrypt SSL certificates.

Add the Traefik Helm chart repository.

                            console
                            
$ helm repo add traefik https://traefik.github.io/charts

Update the local Helm repository.
console
```
$ helm repo update
```

Install Traefik as an Ingress Controller in your cluster.

                            console
                            
                        
$ helm install traefik traefik/traefik \
  --namespace traefik \
  --create-namespace \
  --set "ports.web.exposedPort=80" \
  --set "ports.websecure.exposedPort=443" \
  --set "ports.web.http.redirections.entryPoint.to=websecure" \
  --set "ports.web.http.redirections.entryPoint.scheme=https" \
  --set persistence.enabled=false

Get all resources deployed to the traefik namespace and verify that the deployment is ready.
console
```
$ kubectl get all -n traefik
```
The output shows the Traefik pod, service, deployment, and replicaset. Verify that the pod shows Running status.
Wait at least 3 minutes for the LoadBalancer to provision, then retrieve the external IP address.
console
```
$ kubectl get service traefik --namespace traefik
```
The output displays the EXTERNAL-IP column with your public IP address. If it shows <pending>, wait a few more minutes for the cloud provider to assign an IP.
Log in to your DNS provider (such as Vultr DNS) and create a DNS A record pointing your domain (for example, kibana.example.com) to the LoadBalancer's external IP address.
Install cert-manager in your cluster. Replace v1.20.2 with the latest stable version from the cert-manager releases page.
console
```
$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.20.2/cert-manager.yaml
```
List the cert-manager pods and verify that they are running.
console
```
$ kubectl get pods -n cert-manager
```
The output shows all cert-manager pods with Running status.
Create a new issuer.yaml file.
console
```
$ nano issuer.yaml
```
Add the following YAML contents to the file. Replace admin@example.com with your active email address.
yaml
```
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    email: admin@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: traefik
```
Save and close the file.

The above configuration creates a ClusterIssuer that:
- Uses the Let's Encrypt production Automated Certificate Management Environment (ACME) server for trusted certificates.
- Stores the account private key in a Kubernetes secret.
- Uses HTTP-01 challenge verification with Traefik as the Ingress controller.
Apply the ClusterIssuer configuration.
console
```
$ kubectl apply -f issuer.yaml
```
Create a new kibana-ingress.yaml file.
console
```
$ nano kibana-ingress.yaml
```

Add the following Ingress contents to the file. Replace kibana.example.com with your actual domain.

                            yaml
                            
                        
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kibana
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  ingressClassName: traefik
  tls:
    - hosts:
        - kibana.example.com
      secretName: kibana-tls
  rules:
    - host: kibana.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: quickstart-kb-http
                port:
                  number: 5601

Save and close the file.

The above configuration:

Routes all HTTPS traffic from kibana.example.com to the Kibana service on port 5601.
Uses the letsencrypt ClusterIssuer to automatically provision and renew TLS certificates.
Stores the certificate in a Kubernetes secret named kibana-tls.

Apply the kibana-ingress.yaml configuration to your cluster.
console
```
$ kubectl apply -f kibana-ingress.yaml
```
List all Ingress resources and verify that Kibana is available.
console
```
$ kubectl get ingress -n default
```
The output shows the kibana Ingress with your domain in the HOSTS column.
Verify the certificate is issued successfully.
console
```
$ kubectl get certificate -n default
```
The output shows kibana-tls with True in the READY column. If it shows False, wait a few minutes for cert-manager to complete the ACME challenge.
Test HTTPS access to your domain.
console
```
$ curl -I https://kibana.example.com
```
The output shows HTTP/2 200 or a redirect to the Kibana login page, confirming TLS is working.

Access Kibana Dashboards

The Kibana web interface provides tools for exploring and visualizing log data from your Kubernetes cluster. Access the dashboard using your configured domain with the Elasticsearch credentials.

Retrieve the elastic user password.

                            console
                            
$ kubectl get secret quickstart-es-elastic-user -o go-template='{{.data.elastic | base64decode}}' && echo

Copy the password from the output.

Open a web browser and navigate to https://kibana.example.com. Replace kibana.example.com with your configured domain.
Log in to Kibana with the following credentials:
- Username: elastic
- Password: The password retrieved in the previous step
Click Explore on my own to access the main dashboard.

Visualize the Kubernetes Cluster

Kibana's Discover feature allows you to search and filter logs collected by Filebeat from your Kubernetes cluster.

Expand the main menu in the top left corner.
Click Discover within the Analytics group.
Select the logstash-* data view from the dropdown menu. If no data view exists, create one:
- Navigate to Stack Management > Data Views.
- Click Create data view.
- Enter logstash-* as the name and index pattern.
- Select @timestamp as the time field.
- Click Save data view to Kibana.
Use the search bar to filter logs by namespace, pod name, or message content. For example:
- kubernetes.namespace: default: Shows logs from the default namespace.
- kubernetes.pod.name: demo-app*: Shows logs from pods with names starting with demo-app.
- message: error: Shows logs containing the word "error".
Adjust the time range using the date picker in the top right corner to view logs from specific time periods.
Click on any log entry to expand and view the full document, including Kubernetes metadata such as pod labels, container name, and node information.

Troubleshooting

Use the following checks to diagnose and resolve common issues encountered during deployment.

Common Deployment Errors and Solutions

Elasticsearch pod stuck in Pending state

Run kubectl describe pod quickstart-es-default-0 to check the events. Common causes include:
- Insufficient resources: The node lacks sufficient CPU or memory. Scale up your cluster or reduce resource requests in the Elasticsearch configuration.
- PVC not binding: The storage class does not exist or has no available capacity. Run kubectl get storageclass to verify available storage classes and update the storageClassName in elasticsearch.yaml.
Elasticsearch health shows yellow or red

Run kubectl get elasticsearch to check the cluster status. A yellow status is normal for a single-node deployment because replica shards cannot be placed. For production workloads, increase nodeSets.count to at least 3 to achieve green health. If shards remain unassigned after scaling, run kubectl logs quickstart-es-default-0 to check for disk space or memory issues.

A red status indicates primary shards are unavailable. Check pod logs for out-of-memory errors or storage failures.
Filebeat pods not collecting logs

Verify the RBAC configuration is applied correctly:
console
```
$ kubectl get serviceaccount filebeat
$ kubectl get clusterrole filebeat
$ kubectl get clusterrolebinding filebeat
```
If any resource is missing, reapply filebeat-rbac.yaml. Also verify the host path volumes are accessible by checking Filebeat logs:
console
```
$ kubectl logs -l common.k8s.elastic.co/type=beat
```
Certificate not issued by cert-manager

Check the certificate status and events:
console
```
$ kubectl describe certificate kibana-tls -n default
$ kubectl describe certificaterequest -n default
$ kubectl logs -n cert-manager -l app=cert-manager
```
Common causes include:
- DNS not propagated: Verify your domain resolves to the LoadBalancer IP using dig kibana.example.com.
- HTTP-01 challenge failing: Verify Traefik is running and port 80 is accessible from the internet.
Kibana shows "Kibana server is not ready yet"

Kibana requires Elasticsearch to be healthy before starting. Verify Elasticsearch status:
console
```
$ kubectl get elasticsearch
```
If Elasticsearch shows green health but Kibana still fails, check Kibana logs:
console
```
$ kubectl logs -l kibana.k8s.elastic.co/name=quickstart
```
LoadBalancer IP shows pending

Cloud providers may take several minutes to provision a LoadBalancer. If the status remains <pending> after 5 minutes:
- Verify your Kubernetes cluster supports LoadBalancer services (requires cloud provider integration).
- Check the Traefik service events: kubectl describe service traefik -n traefik.

Conclusion

You have deployed the ELK Stack on Kubernetes using the ECK operator. Filebeat collects logs from every node in the cluster, Logstash processes and enriches them, Elasticsearch stores and indexes the data, and Kibana visualizes it through a dashboard secured with a Let's Encrypt TLS certificate. For more information on advanced configurations including multi-node clusters, custom pipelines, and alerting, refer to the official Elastic Cloud on Kubernetes documentation.

Tags:

Kubernetes

How To Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Kubernetes

How To Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Kubernetes

Prerequisites

Install Elasticsearch

Install Logstash

Install Filebeat

Install and Configure Kibana Dashboards

Secure the ELK Stack

Access Kibana Dashboards

Visualize the Kubernetes Cluster

Troubleshooting

Common Deployment Errors and Solutions

Conclusion

Comments

Products

Features

Solutions

Marketplace

Resources

Company

Tech Talks

Vultr Blogs