How to Deploy Kafka – Distributed Event Streaming

Apache Kafka is a high-throughput, distributed event-streaming platform used for building real-time data pipelines and streaming applications. It provides persistent, fault-tolerant message storage and supports large-scale publish-subscribe workloads with low latency. Kafka is widely used for log aggregation, metrics collection, event processing, microservices communication, and real-time analytics.

This article explains how to deploy Apache Kafka using Docker Compose, configure persistent storage, define environment variables, and verify that the Kafka broker is reachable.

Prerequisites

Before you begin, you need to:

• Have access to an Ubuntu 24.04-based server as a non-root user with sudo privileges.
• Install Docker and Docker Compose.

Set Up the Directory Structure and Environment Variables

In this section, you create a project directory for Kafka and define environment variables in a .env file that Docker Compose loads automatically.

1. Create the project directory.

   CONSOLE

   Copy

   $ mkdir -p ~/kafka-stack/kafka-data

   Explain Code

   Powered by Vultr AgentBeta

2. Change the data directory permission.

   CONSOLE

   Copy

   $ sudo chown 1000:1000 ~/kafka-stack/kafka-data

   Explain Code

   Powered by Vultr AgentBeta

   This permission change is required because the Apache Kafka container runs using an internal application user appuser with UID 1000, and it must have write access to the kafka-data directory for logs and persisted messages.

3. Navigate into the project directory.

   CONSOLE

   Copy

   $ cd ~/kafka-stack

   Explain Code

   Powered by Vultr AgentBeta

4. Generate a UUID for the cluster identifier.

   CONSOLE

   Copy

   $ uuidgen | tr -d '-'

   Explain Code

   Powered by Vultr AgentBeta

   Copy the generated value for use in the .env file.

5. Create a .env file to store the environment variables.

   CONSOLE

   Copy

   $ nano .env

   Explain Code

   Powered by Vultr AgentBeta

   Add the following variables:

   INI

   Copy

   KAFKA_NODE_ID=1
   KAFKA_CLUSTER_ID=YOUR_KAFKA_CLUSTER_ID
   KAFKA_PORT=9092
   KAFKA_ADVERTISED_HOST=localhost

   Explain Code

   Powered by Vultr AgentBeta

   Replace:

   • KAFKA_NODE_ID with a unique broker ID (keep 1 for single-node).
   • KAFKA_CLUSTER_ID with the unique cluster identifier you generated in the previous step.
   • KAFKA_PORT with the port Kafka should listen on.
   • KAFKA_ADVERTISED_HOST with the hostname or IP clients should use to connect.

   Note

   If your applications run on the same server as Kafka, keep KAFKA_ADVERTISED_HOST=localhost. If your applications run on a different server, replace localhost with your public server IP address so external clients can connect.

   Save and close the file.

Deploy with Docker Compose

In this section, you create the Docker Compose manifest that deploys Apache Kafka in KRaft mode, enables persistent storage, and exposes the broker for client connections.

1. Create the Docker Compose manifest.

   CONSOLE

   Copy

   $ nano docker-compose.yaml

   Explain Code

   Powered by Vultr AgentBeta

   Add the following content:

   YAML

   Copy

   services:
     kafka:
       image: apache/kafka:4.1.1
       container_name: kafka
       ports:
         - "${KAFKA_PORT}:9092"
       environment:
         KAFKA_NODE_ID: ${KAFKA_NODE_ID}
         KAFKA_PROCESS_ROLES: broker,controller
         KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
         KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://${KAFKA_ADVERTISED_HOST}:9092
         KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
         KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
         KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
         KAFKA_LOG_DIRS: /kafka/data
         KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
         KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
         KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
         KAFKA_CLUSTER_ID: ${KAFKA_CLUSTER_ID}
       volumes:
         - "./kafka-data:/kafka/data"
       restart: unless-stopped
   
       healthcheck:
         test: ["CMD", "kafka-broker-api-versions", "--bootstrap-server=localhost:9092"]
         interval: 10s
         timeout: 5s
         retries: 5

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   In the manifest:

   • services: Defines the Kafka service managed by Docker Compose.
     • kafka: Runs Apache Kafka in single-node KRaft mode.
   • image: Uses the official Apache Kafka image.
   • ports: Publishes Kafka’s default port (9092) on the host.
   • environment:
     • KAFKA_NODE_ID: Unique broker ID.
     • KAFKA_PROCESS_ROLES: Runs both broker and controller in one container.
     • KAFKA_LISTENERS: Defines internal listener endpoints.
     • KAFKA_ADVERTISED_LISTENERS: Defines the external address clients use to connect.
     • KAFKA_CONTROLLER_QUORUM_VOTERS: Configures the internal controller quorum.
     • KAFKA_LOG_DIRS: Path where Kafka stores log and topic data.
     • KAFKA_CLUSTER_ID: Initializes the KRaft cluster.
     • Replication and ISR values are set to 1 for single-node operation.
   • volumes:
     • ./kafka-data persists Kafka topic and log data across restarts.
   • healthcheck: Verifies Kafka availability using the broker API.
   • restart: unless-stopped: Restarts Kafka automatically unless manually stopped.

2. Start the Kafka service.

   CONSOLE

   Copy

   $ docker compose up -d

   Explain Code

   Powered by Vultr AgentBeta

3. Verify that Kafka service is running.

   CONSOLE

   Copy

   $ docker compose ps

   Explain Code

   Powered by Vultr AgentBeta

   Ensure that the output shows the Kafka container is healthy and running.

4. View logs if needed.

   CONSOLE

   Copy

   $ docker compose logs

   Explain Code

   Powered by Vultr AgentBeta

   For more information on managing Docker Compose services, see the How To Use Docker Compose article.

Verify Apache Kafka Connectivity

In this section, you verify that your Apache Kafka broker is reachable and accepting client connections by using the Kafka CLI tools.

Install Kafka CLI tools

1. Update the package index and install the default Java runtime.

   CONSOLE

   Copy

   $ sudo apt update && sudo apt install -y default-jdk

   Explain Code

   Powered by Vultr AgentBeta

2. Download the Kafka CLI tools archive. Refer to the official Apache Kafka downloads page for newer versions.

   CONSOLE

   Copy

   $ wget https://downloads.apache.org/kafka/4.1.1/kafka_2.13-4.1.1.tgz

   Explain Code

   Powered by Vultr AgentBeta

3. Extract the archive to /opt.

   CONSOLE

   Copy

   $ sudo tar -xzf kafka_2.13-4.1.1.tgz -C /opt

   Explain Code

   Powered by Vultr AgentBeta

4. Add the Kafka binaries from /opt to your PATH.

   CONSOLE

   Copy

   $ echo 'export PATH=$PATH:/opt/kafka_2.13-4.1.1/bin' | sudo tee /etc/profile.d/kafka.sh

   Explain Code

   Powered by Vultr AgentBeta

5. Apply the updated PATH.

   CONSOLE

   Copy

   $ source /etc/profile.d/kafka.sh

   Explain Code

   Powered by Vultr AgentBeta

Test Message Production and Consumption

1. Create a test topic.

   CONSOLE

   Copy

   $ kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test-topic --partitions 3 --replication-factor 1

   Explain Code

   Powered by Vultr AgentBeta

2. Start the producer.

   CONSOLE

   Copy

   $ kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test-topic

   Explain Code

   Powered by Vultr AgentBeta

   This command opens an interactive console where you type messages and press ENTER after each one to publish the messages to the topic.

3. Open a new terminal and start a consumer to read messages from the beginning.

   CONSOLE

   Copy

   $ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning

   Explain Code

   Powered by Vultr AgentBeta

   This command starts the consumer and displays all messages sent to the specified topic.

4. Verify the topic configuration.

   CONSOLE

   Copy

   $ kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic test-topic

   Explain Code

   Powered by Vultr AgentBeta

   This command displays detailed information about the topic configuration.

Conclusion

You successfully deployed Apache Kafka using Docker Compose in single-node KRaft mode with persistent storage enabled. You configured all required environment variables, prepared the data directory with correct permissions, and exposed the broker for client access. After installing the Kafka CLI tools, you verified end-to-end message flow by producing and consuming test messages.