Install Apache Kafka on Ubuntu 20.04

Introduction

Apache Kafka is an open-source, distributed message broker based on Scala and Java. It's used as a publish-subscribe messaging system; when a producer publishes a message, it's broadcast, and subscribers can get notified and process the information. It can handle large volumes of real-time data and event streaming pipelines with higher throughput than popular message brokers like ActiveMQ and RabbitMQ. In addition, it's highly scalable and fault-tolerant.

Apache Kafka uses Apache Zookeeper to keep track of its cluster node's status, topics, partitions, and so on. Apache ZooKeeper is an open-source project built for centralized services. It provides configuration information, naming, synchronization and group services for large clusters in a distributed systems.

This article explains how to install Apache Kafka on Ubuntu 20.04 server.

Prerequisites

• Deploy a fully updated Vultr Ubuntu 20.04 Server.
• Create a non-root user with sudo access.

1. Install Java

Update system package list.

$ sudo apt update

Install Java.

$ sudo apt install openjdk-11-jre-headless -y

Verify the installation.

$ java --version

2. Install Apache Kafka

Download Apache Kafka source files. To find the latest version of this software, visit the official download website.

$ wget https://dlcdn.apache.org/kafka/3.0.0/kafka_2.12-3.0.0.tgz

Create an installation directory /usr/local/kafka-server.

$ sudo mkdir /usr/local/kafka-server

Extract the downloaded files.

$ sudo tar -xzf kafka_2.12-3.0.0.tgz

Move the extracted files to the installation directory.

$ sudo mv kafka_2.12-3.0.0/* /usr/local/kafka-server

For Apache Kafka and Apache Zookeeper to run as daemons, create systemd files for both of them.

Creating systemd file for Apache Zookeeper.

$ sudo nano /etc/systemd/system/zookeeper.service

Add the following lines of code to the file, save and close the file.

[Unit]

Description=Apache Zookeeper Server

Requires=network.target remote-fs.target

After=network.target remote-fs.target

[Service]

Type=simple

ExecStart=/usr/local/kafka-server/bin/zookeeper-server-start.sh /usr/local/kafka-server/config/zookeeper.properties

ExecStop=/usr/local/kafka-server/bin/zookeeper-server-stop.sh

Restart=on-abnormal

[Install]

WantedBy=multi-user.target

Creating systemd file for Apache Kafka.

$ sudo nano /etc/systemd/system/kafka.service

Add the following lines of code to the file, save and close the file.

[Unit]

Description=Apache Kafka Server

Documentation=http://kafka.apache.org/documentation.html

Requires=zookeeper.service

After=zookeeper.service

[Service]

Type=simple

Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"

ExecStart=/usr/local/kafka-server/bin/kafka-server-start.sh /usr/local/kafka-server/config/server.properties

ExecStop=/usr/local/kafka-server/bin/kafka-server-stop.sh

Restart=on-abnormal

[Install]

WantedBy=multi-user.target

Reload the systemd daemons to apply the changes.

$ sudo systemctl daemon-reload

Start and enable the Apache Zookeeper service for startup on system boot.

$ sudo systemctl enable --now zookeeper.service

Start and enable the Apache Kafka service for startup on system boot.

$ sudo systemctl enable --now kafka.service

Verify the status of both Apache Zookeeper and Apache Kafka services.

$ sudo systemctl status kafka zookeeper

3. Install Cluster Manager for Apache Kafka (CMAK).

CMAK is a graphical management tool for Apache Kafka clusters. It provides the user with various tools to avoid using the CLI interface.

Clone the CMAK repository from GitHub.

$ git clone https://github.com/yahoo/CMAK.git

Edit the CMAK configuration file.

$ sudo nano CMAK/conf/application.conf

Change the value of cmak.zkhosts as follows.

cmak.zkhosts="localhost:2181"

Change to the CMAK directory.

$ cd CMAK

Create a zip file used for deploying the application.

$ ./sbt clean dist

Change into CMAK/target/universal directory.

$ cd target/universal

Install unzip package.

$ sudo apt install unzip -y

Extract the created zip file.

$ sudo unzip cmak-3.0.0.5.zip

Change to the extracted directory.

$ cd cmak-3.0.0.5

Execute the CMAK binary to start the service.

$ sudo bin/cmak

4. Access CMAK Web Interface

Open your web browser and access the CMAK web interface using the URL http://ServerIP:9000. For example:

http://192.0.2.10:9000

Conclusion

You have installed Apache Kafka and Apache Zookeeper on your server. You can now access the CMAK dashboard and add a Kafka cluster.

More Information

For more information on Apache Kafka, please see the official documentation.