Install Apache Kafka on CentOS 8

Introduction

Apache Kafka is an open-source event stream processing application. You can use it as a distributed publish-subscribe messaging system to decouple your business logic processes from data producers. Other use-cases include high-volume website activity tracking, server log aggregation, event sourcing, and commit log.

Apache Kafka is highly scalable because it is a distributed system with better throughput and built-in replication. Apache Kafka can scale without downtime in a production environment, making it suitable for deployment in fault-tolerant system architecture.

In this guide, you'll install Apache Kafka on your CentOS 8 server.

Prerequisites

Before you begin, ensure you have the following:

• A CentOS 8 server.
• A non-root sudo user.

1. Install the Java Virtual Machine

SSH to your server and update your system.

$ sudo yum update -y

Apache Kafka is written in Java and Scala. Hence, it requires Java Runtime Environment to run. Install the OpenJDK 8.

$ sudo yum install -y java-1.8.0-openjdk

Verify the installation.

$ java -version

Make sure you get the output below.

openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)

Edit /etc/profile and configure the Java environment variables.

$ sudo vi /etc/profile

Add the two lines below to the bottom of the file.

export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
export JRE_HOME=/usr/lib/jvm/jre

Then, press Colon, W, Q and Enter to save the file. Reload the new profile changes.

$ source /etc/profile

2. Install Apache Kafka

Install the wget utility.

$ sudo yum install -y wget

Navigate to your home directory.

$ cd ~

In a web browser, navigate to the Kafka download page to find the download URL of the latest version of Apache Kafka. Download the latest stable version of the Apache Kafka package from the official repository. For example, Kafka version 2.8.0 for Scala 2.12:

$ wget https://downloads.apache.org/kafka/2.8.0/kafka_2.12-2.8.0.tgz

Use the tar command to extract the compressed file.

$ tar -xvf kafka_2.12-2.8.0.tgz

Move the extracted files to the /opt directory. This is an ideal location for storing add-on applications.

$ sudo mv kafka_2.12-2.8.0 /opt

Navigate to the new directory.

$ cd /opt/kafka_2.12-2.8.0

Open bin/kafka-server-start.sh to edit the Apache Kafka configuration.

$ sudo vi bin/kafka-server-start.sh

Find the line below.

export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"

Change the value of export KAFKA_HEAP_OPTS from 1G to 128M

export KAFKA_HEAP_OPTS="-Xmx1G -Xms128M"

The Apache Kafka and Zookeeper packages are now in place; you'll run them in the next step.

3. Run Zookeeper and Apache Kafka Services

Make sure you're in the Apache Kafka working directory.

$ cd /opt/kafka_2.12-2.8.0

Start the Zookeeper server.

$  bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

Start the Kafka server.

$ bin/kafka-server-start.sh config/server.properties

You now have the right Apache Kafka environment, and you can start creating test topics.

4. Create Test Topics

Leave the Zookeeper and the Apache Kafka services running. Then, establish another SSH connection in a new terminal window and cd to the Apache Kafka directory.

$ cd /opt/kafka_2.12-2.8.0

In Apache Kafka, a topic is a unique channel for streaming data. Use the kafka-topics.sh script to create some test topics by running the commands below one by one.

$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test1
$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test2
$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test3

After running the commands, you will get the outputs shown below.

Created topic test1.
...

Created topic test2.
...

Created topic test3.

Display the list of topics you've just created.

$ bin/kafka-topics.sh --list --zookeeper localhost:2181

You should now get the list below.

test1
test2
test3

After creating topics, you'll produce messages to them in the next step.

5. Produce a Message to a Topic

To produce a message to a topic such as test1, use the kafka-console-producer.sh script by typing the commands below.

$ cd /opt/kafka_2.12-2.8.0
$ bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test1

After you get a user input mode prompt >, type the single-line messages below and press Enter after each message.

> Message Line 1 :key_enter:
> Message Line 2 :key_enter:
> Message Line 3 :key_enter:

6. Consume a Message from a Topic

To retrieve data from a topic such as test1, create another SSH connection in a new terminal window. Then, cd to the Apache Kafka directory.

$ cd /opt/kafka_2.12-2.8.0

Type the command below to listen to any messages you've created in the test1 topic.

$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test1 --from-beginning

If you get the messages that you created earlier as shown below, your Apache Kafka installation is working as expected.

Message Line 1
Message Line 2
Message Line 3

Conclusion

In this guide, you've installed Apache Kafka on your CentOS 8 server. With this setup, you can begin using the high-performance software in your mission-critical applications for stream processing. To learn more, please see the Apache Kafka documentation.