Kafka集群部署

没有提示，所有机器都执行

在kafka集群中引入zookeeper，主要是为了管理kafka集群的broker。负责管理集群的元数据信息，确保 Kafka 集群的高可用性、高性能和高可靠性。

一、实例配置

IP	用途
192.168.0.217	kafka、zk
192.168.0.218	kafka、zk
192.168.0.219	kafka、zk

二 、zookeeper集群安装

1. 解压zk安装包

$ tar xf apache-zookeeper-3.6.2-bin.tar.gz
$ mv apache-zookeeper-3.6.2-bin /data/zookeeper

2. 修改zookeeper配置文件

$ vim /data/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
clientPort=2181
maxClientCnxns=512
server.217=192.168.0.217:2888:3888
server.218=192.168.0.218:2888:3888
server.219=192.168.0.219:2888:3888

$ mkdir /data/zookeeper/data
#根据实例自行修改
$ echo 217 > /data/zookeeper/data/myid

3. 托管到systemd

$ vim /usr/lib/systemd/system/zookeeper.service
[Unit]
Description=Apache Zookeeper service
Requires=network.target
After=network.target

[Service]
Type=forking
Environment=JAVA_HOME=/usr/local/java
ExecStart=/data/zookeeper/bin/zkServer.sh  start
ExecStop=/data/zookeeper/bin/zkServer.sh  stop
Restart=on-failure
RestartSec=10s

[Install]
WantedBy=multi-user.target

$ systemctl daemon-reload

4. 启动zk

$ systemctl start zookeeper

三、kafka集群安装

1. 解压kafka安装包

$ tar xf kafka_2.12-2.6.0.tgz 
$ mv kafka_2.12-2.6.0 /data/kafka

2. 修改kafka配置

$ vim /data/kafka/config/server.properties
# 每个实例不同
broker.id=217 
listeners=PLAINTEXT://192.168.0.217:9092 #
advertised.listeners=PLAINTEXT://192.168.0.217:9092 #
# kafka存储日志文件的目录
log.dirs=/data/kafka/kafka-logs 
# 存储消费者偏移量的主题的副本因子
offsets.topic.replication.factor=3
# 设置事务日志主题的副本因子
transaction.state.log.replication.factor=3
log.retention.hours=48 # 数据保留48小时
zookeeper.connect=192.168.0.217:2181,192.168.0.218:2181,192.168.0.219:2181 #

$ mkdir /data/kafka/kafka-logs

3. 托管到systemd

$ vim /usr/lib/systemd/system/kafka.service
[Unit]
Description=Apache Kafka server
Documentation=http://kafka.apache.org/documentation.html
Requires=network.target remote-fs.target
After=network.target remote-fs.target zookeeper.service

[Service]
Type=forking
Environment=JAVA_HOME=/usr/local/java
ExecStart=/data/kafka/bin/kafka-server-start.sh -daemon /data/kafka/config/server.properties
ExecStop=/data/kafka/bin/kafka-server-stop.sh /data/kafka/config/server.properties
Restart=on-failure
RestartSec=10s

[Install]
WantedBy=multi-user.target

$ systemctl daemon-reload

4. 启动kafka

$ systemctl start kafka

四、验证

1. 登录zookeeper查看kafka信息

/data/zookeeper/bin/zkCli.sh
> ls /brokers/ids
[217, 218, 219]
> get /brokers/ids/217

可以看到kafka已经注册到zookeeper