Apache Kafka installation on Linux

In the following post, I will show how to install Apache Kafka on a Linux VM.

Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java.

First of all we need a Linux VM (e.g Centos 7) with at least 1 GB of RAM

Connect as root and add a new user for Kafka

set the password

now add the newly created user to sudoers group

or

find the follwing code

add the following code below

update and reboot the system

login with the user created above and install java

 

check the java version

add java variables to .bash_profile

resource profile and check settings

Now we are ready to install and confgiure Kafka.We will use the confluent platform for this environment. Confluent open source platform is freely available for download.

 

So first of all we have to import Confluents public key.

After that we need to add the Confluent repository to yum.For that create a new file called confluent.repo in /etc/yum.repos.d

and paste the following:

Now clean up your yum caches with

The repo is now ready for use and we could install Confluent Open Source

As the binaries are installed we’re able to launch Kafka and Zookeper 🙂

Check the Zookeeper configuration:

 

For a lab environment the above setting should be fine no need to change anything.

So now we could start zookeeper

As Zookeeper is running we could now start Kafka.
Therefore leave the current session running and open a new ssh or terminal session.

(output shortened)

Now Kafka is running and we are able to create a new topic.

To check Kafka you could run

The output should be similar to (at least no error should occur 😉 )

To create a new topic leave the current terminal or ssh session running an start another terminal or ssh session.

creates the topic “mytopic”

Topic creation is also visible in Kafka brokers standard out and log:

list the newly created topic

Now we could produce some message in the topic mytopic.

Leave the producer session running and open a new terminaL or ssh session.
We could run a consumer session which shows all the message of the topic from the beginning

Also every new message you type in the producer session is visible immediate to your consumer session.

To stop the the producer, consumer and the cluster itself just press CTRL+C.

Hope you enjoyed this post about how to create a small Kafka cluster.

Next post will follow soon 😉

Further reading

http://confluent.io/

http://kafka.apache.org

 

3 Comments

Add a Comment

Your email address will not be published. Required fields are marked *