How to install kafka - Standalone Installation

As part of the this tutorial, we will be setting up Apache Kafka in a single node. Apache Kafka is an open-source message broker project developed by the Apache Software Foundation written in Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.

Download Kafka

Download Latest Kafka From the official Apache Kafka Website

[root@monitor tmp]# wget http://apachemirror.wuchna.com/kafka/2.4.0/kafka_2.11-2.4.0.tgz
--2020-02-28 14:38:08-- http://apachemirror.wuchna.com/kafka/2.4.0/kafka_2.11-2.4.0.tgz
Resolving apachemirror.wuchna.com (apachemirror.wuchna.com)... 159.65.154.237
Connecting to apachemirror.wuchna.com (apachemirror.wuchna.com)|159.65.154.237|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 70057083 (67M) [application/x-gzip]
Saving to: ‘kafka_2.11-2.4.0.tgz’

100%[==========================================================================================>] 70,057,083 19.6MB/s in 4.5s

2020-02-28 14:38:12 (15.0 MB/s) - ‘kafka_2.11-2.4.0.tgz’ saved [70057083/70057083]

[root@monitor tmp]#

Install Java

[root@monitor tmp]# yum install java-1.8.0-openjdk
[root@monitor tmp]# yum install java-1.8.0-openjdk-devel

Untar the tarball

[root@monitor tmp]# tar xzvf kafka_2.11-2.4.0.tgz
kafka_2.11-2.4.0/
kafka_2.11-2.4.0/LICENSE
kafka_2.11-2.4.0/NOTICE
kafka_2.11-2.4.0/bin/
kafka_2.11-2.4.0/bin/kafka-delete-records.sh
kafka_2.11-2.4.0/bin/trogdor.sh
kafka_2.11-2.4.0/bin/kafka-preferred-replica-election.sh
kafka_2.11-2.4.0/bin/connect-mirror-maker.sh
kafka_2.11-2.4.0/bin/kafka-console-consumer.sh

Start Zookeeper

Kafka relies on Zookeeper to for configuration information.

[root@monitor tmp]# bin/zookeeper-server-start.sh config/zookeeper.properties

Start the Kafka Servics

[root@monitor tmp]# kafka_2.11-2.4.0/bin/kafka-server-start.sh kafka_2.11-2.4.0/config/server.properties
[2020-02-28 14:41:17,097] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2020-02-28 14:41:17,386] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2020-02-28 14:41:17,386] INFO starting (kafka.server.KafkaServer)
[2020-02-28 14:41:17,387] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2020-02-28 14:41:17,431] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)

Create Topic

Kafka partitions incoming messages for a topic, and assigns those partitions to the available Kafka brokers. The number of partitions is configurable and can be set per-topic and per-broker. The following command will create a topic test with one partition

[root@monitor kafka_2.11-2.4.0]# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic test.
[root@monitor kafka_2.11-2.4.0]#

List Topics

To make sure wheather the topic is created successfully, we can list out the topics hosted by the broker with the following command.

[root@monitor kafka_2.11-2.4.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
test
[root@monitor kafka_2.11-2.4.0]#

Start Producer

Producers publish messages to Kafka topics and are able to choose which topic, and which partition within the topic, to publish the message to. we will start a producer no

[root@monitor kafka_2.11-2.4.0]# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>test
>this is manas
>who are you
>how are you
>^CYou have new mail in /var/spool/mail/root
[root@monitor kafka_2.11-2.4.0]#

Start Consumer

Consumers subscribe to these topics and consume the messages. Lets consume the messages with the following command

[root@monitor kafka_2.11-2.4.0]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
test
this is manas
who are you
how are you
^CProcessed a total of 4 messages
You have new mail in /var/spool/mail/root
[root@monitor kafka_2.11-2.4.0]#

How to install kafka - Standalone Installation