How to install Kafka Manager for Managing kafka cluster

In this article, we are going to set up the Kafka management software to manage and overview our cluster. Kafka manager developed by at Yahoo. It is now called CMAK (Cluster Manager for Apache Kafka, previously known as Kafka Manager).

This application support following tasks.

Manage multiple clusters
Easy inspection of cluster state (topics, consumers, offsets, brokers, replica distribution, partition distribution)
Run preferred replica election
Generate partition assignments with option to select brokers to use
Run reassignment of partition (based on generated assignments)
Create a topic with optional topic configs (0.8.1.1 has different configs than 0.8.2+)
Delete topic (only supported on 0.8.2+ and remember set delete.topic.enable=true in broker config)
Topic list now indicates topics marked for deletion (only supported on 0.8.2+)
Batch generate partition assignments for multiple topics with option to select brokers to use
Batch run reassignment of partition for multiple topics
Add partitions to existing topic
Update config for existing topic
Optionally enable JMX polling for broker level and topic level metrics.
Optionally filter out consumers that do not have ids/ owners/ & offsets/ directories in zookeeper.

Prerequisites :

Apache Zookeeper
Apache Kafka
Java 8
SBT 13 +

Install Apache Zookeeper

Download Apache Zookeeper from Apache Zookeeper site

root@monitoringserver /]# wget http://www.us.apache.org/dist/zookeeper/current/apache-zookeeper-3.5.5-bin.tar.gz
--2019-07-10 20:52:48-- http://www.us.apache.org/dist/zookeeper/current/apache-zookeeper-3.5.5-bin.tar.gz
Resolving www.us.apache.org (www.us.apache.org)... 40.79.78.1
Connecting to www.us.apache.org (www.us.apache.org)|40.79.78.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10622522 (10M) [application/x-gzip]
Saving to: ‘apache-zookeeper-3.5.5-bin.tar.gz’

100%[======================================================================================>] 10,622,522 11.6MB/s in 0.9s

2019-07-10 20:52:50 (11.6 MB/s) - ‘apache-zookeeper-3.5.5-bin.tar.gz’ saved [10622522/10622522]

Untar zookeeper

[root@monitoringserver /]# tar -xzf apache-zookeeper-3.5.5-bin.tar.gz

Create Zookeeper Log Folder and Set Permission

[root@monitoringserver /]# cd apache-zookeeper-3.5.5-bin
[root@monitoringserver apache-zookeeper-3.5.5-bin]# ls
bin conf docs lib LICENSE.txt NOTICE.txt README.md README_packaging.txt
[root@monitoringserver apache-zookeeper-3.5.5-bin]#

[root@monitoringserver conf]# mkdir -p /zookeeper/data/logs
[root@monitoringserver conf]# chmod -R 777 /zookeeper/
[root@monitoringserver conf]#

Change the directory to conf and Create a new file zoo.cfg.

[root@monitoringserver conf]# vi zoo.cfg
[root@monitoringserver conf]#

Add following Content to zoo.cfg

[root@monitoringserver conf]# cat zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/zookeeper/data/
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
dataLogDir=/zookeeper/data/logs/
[root@monitoringserver conf]#

Start Zookeeper -

[root@monitoringserver apache-zookeeper-3.5.5-bin]# nohup bin/zkServer.sh start &
[1] 59786
[root@monitoringserver apache-zookeeper-3.5.5-bin]#

Install Kafka

To install a standalone kafka please follow this link.

Install of Yahoo Kafka Manager:

[root@monitoringserver kafkaManager]# wget https://github.com/yahoo/kafka-manager/archive/master.zip
--2019-07-10 20:07:19-- https://github.com/yahoo/kafka-manager/archive/master.zip
Resolving github.com (github.com)... 140.82.118.4
Connecting to github.com (github.com)|140.82.118.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/yahoo/kafka-manager/zip/master [following]
--2019-07-10 20:07:19-- https://codeload.github.com/yahoo/kafka-manager/zip/master
Resolving codeload.github.com (codeload.github.com)... 140.82.114.9
Connecting to codeload.github.com (codeload.github.com)|140.82.114.9|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘master.zip’

[ <=> ] 1,106,343 2.18MB/s in 0.5s

2019-07-10 20:07:20 (2.18 MB/s) - ‘master.zip’ saved [1106343]

[root@monitoringserver kafkaManager]#

Install java

[root@monitoringserver kafkaManager]# yum install java-1.8.0-openjdk
[root@monitoringserver kafkaManager]# yum install java-1.8.0-openjdk-devel

Unzip kafka-manager:

[root@monitoringserver kafkaManager]# unzip master.zip

Build kafka-manager

[root@monitoringserver kafka-manager-master]# ./sbt clean dist
Downloading sbt launcher for 1.2.8:
From http://repo.scala-sbt.org/scalasbt/maven-releases/org/scala-sbt/sbt-launch/1.2.8/sbt-launch.jar
To /root/.sbt/launchers/1.2.8/sbt-launch.jar
Getting org.scala-sbt sbt 1.2.8 (this may take some time)...
downloading https://repo1.maven.org/maven2/org/scala-sbt/sbt/1.2.8/sbt-1.2.8.jar ...
[SUCCESSFUL ] org.scala-sbt#sbt;1.2.8!sbt.jar (17ms)
downloading https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.12.7/scala-library-2.12.7.jar ...
[SUCCESSFUL ] org.scala-lang#scala-library;2.12.7!scala-library.jar (238ms)
downloading https://repo1.maven.org/maven2/org/scala-sbt/main_2.12/1.2.8/main_2.12-1.2.8.jar ...
[SUCCESSFUL ] org.scala-sbt#main_2.12;1.2.8!main_2.12.jar (86ms)
[info] Packaging /kafkaManager/kafka-manager-master/target/scala-2.12/kafka-manager_2.12-2.0.0.2-sans-externalized.jar ...
[info] Done packaging.
[success] All package validations passed
[info] Your package is ready in /kafkaManager/kafka-manager-master/target/universal/kafka-manager-2.0.0.2.zip
[success] Total time: 41 s, completed Jul 10, 2019 8:24:54 PM
[root@monitoringserver kafka-manager-master]#

Copy package and Unzip:

cd target/universal/
unzip kafka-manager-2.0.0.2.zip
cd kafka-manager-2.0.0.2

Start the Service using below command

bin/kafka-manager -Dkafka-manager.zkhosts=”localhost:2181″

or You can edit the "conf/application.conf" with zookeeper values "kafka-manager.zkhosts=". Then start the Kafka Manager Service.

bin/kafka-manager