Kafka is an open source distributed messaging system that is been used by many organizations for many use cases. Its use cases include stream processing, log aggregation, metrics collection and so on.<\/p>\n
Note: This tutorial is based on Redhat 7 derivative. However, it will work on most Linux systems.<\/p>\n
This tutorial will guide you to set up a latest Kafka cluster from scratch.<\/p>\n
1. You need a Zookeeper cluster before setting up a Kafka cluster. Refer this zookeeper cluster setup if you don’t have one.<\/p>\n
2. Launch three instances. Make sure you allow the traffic between Zookeeper and Kafka instances in the security groups.<\/p>\n
3. Set hostnames for three instances for identification using the following command.<\/p>\n
hostnamectl set-hostname (node1,2,3)<\/code><\/pre>\nKafka Installation<\/h3>\n
Perform the following tasks on all the servers. 1. Update the server. <\/p>\n
sudo yum update -y<\/code><\/pre>\n2. Install java 8. <\/p>\n
sudo yum -y install java-1.8.0-openjdk<\/code><\/pre>\n3. Get the latest version of Kafka from here. <\/p>\n
cd \/opt\nsudo wget http:\/\/mirror.fibergrid.in\/apache\/kafka\/0.10.0.0\/kafka_2.11-0.10.0.0.tgz<\/code><\/pre>\n4. Untar the Kafka binary. <\/p>\n
sudo tar -xvf kafka_2*<\/code><\/pre>\n5. Rename the extracted Kafka folder with versions to Kafka. <\/p>\n
sudo mv kafka_2.11-0.10.0.0 kafka<\/code><\/pre>\nCreating a Kafka Service<\/h3>\n
6. Open the server.properties<\/code> file, find zookeeper.connect<\/code><\/p>\nat the bottom and enter the zookeeper IPs as shown below. Replace zk1, zk2, and zk3 with the IPs or DNS names of your zookeeper instances.<\/p>\n
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181<\/code><\/pre>\nCreate a Kafka Service<\/h3>\n
1. Create a systemd file. <\/p>\n
sudo vi \/lib\/systemd\/system\/kafka.service<\/code><\/pre>\nCopy the following contents on to the kafka.service unit file.<\/p>\n
[Unit]\nDescription=Kafka\nBefore=\nAfter=network.target\n\n[Service]\nUser=ec2-user\nCHDIR= {{ data_dir }}\nExecStart=\/opt\/kafka\/bin\/kafka-server-start.sh \/opt\/kafka\/config\/server.properties\nRestart=on-abort\n\n[Install]\nWantedBy=multi-user.target<\/code><\/pre>\n2. Reload the daemon.<\/p>\n
sudo systemctl daemon-reload<\/code><\/pre>\nManaging Kafka Service<\/h3>\n
Once the Kafka service is created, you can manage the Kafka service using the Linux service module.<\/p>\n
\n- To start the Kafka service,<\/li>\n<\/ol>\n
sudo service kafka start<\/code><\/pre>\n2. To stop and restart,<\/p>\n
sudo service kafka stop\nsudo service kafka restart<\/code><\/pre>\nTesting The Kafka Cluster<\/h3>\n
To test the kafka cluter setup, we will create a topic and few messages. Andn we will try to consume it from different node to conform that the cluster is working as intended.<\/p>\n
To test, cd in to the kafka bin directory to get access to the kafka scripts.<\/p>\n
cd \/opt\/kafka\/bin<\/code><\/pre>\nStep 1: Create a Topic from kafka node 1 with replication-factor 3. Replace zk<\/code> with zookeper IP or DNS name.<\/p>\n.\/kafka-topics.sh –create –zookeeper zk:2181 –replication-factor 3 –partitions 1 –topic test<\/p>\n
Step 2: Describe Topic from kafka node 2.<\/p>\n
.\/kafka-topics.sh –describe –zookeeper zk:2181 –topic test<\/p>\n
Step 3: Create a Message using the following command. Enter the message in the terminal with weach message per line. Use Ctrl + C<\/code> to exit.<\/p>\n.\/kafka-console-producer.sh –broker-list kafka-node:9092 –topic test<\/p>\n
Step 4: From a different node, try to consume the message using the following command.<\/p>\n
.\/kafka-console-consumer.sh –zookeeper zk:2181 –from-beginning –topic test<\/p>\n
If you were able to do all the test mentioned above, you have a working kafka cluster.<\/p>\n
If you face any issue during the setup, feel free to drop us a comment below.<\/p>\n
\n