Kafka安裝部署

?1下載并安裝JDK

????參考jdk的安裝，需要jdk1.6+。

2????Kafka的安裝

????2.1 下載kafka

??????下載地址：http://mirrors.hust.edu.cn/apache/kafka/0.10.0.1/kafka_2.11-0.10.0.1.tgz

????2.2 解壓安裝包

????????$ tar zvxf kafka_2.11-0.10.0.1.tgz

2.3?????復制解壓后的文件夾到安裝目錄

????????$ cp -r?kafka_2.11-0.10.0.1 /software/

2.4?????創建kafka安裝目錄的軟鏈接

????????$ ln -s /software/kafka_2.11-0.10.0.1/software/kafka

2.5?????配置環節變量

????????修改/etc/profile文件內容：

????????export KAFKA_HOME=/software/kafka

????????export PATH=$KAFKA_HOME/bin:$FLUME_HOME/bin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:

? ? ? ? ? ? ? ? ? ? $JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

????????export CLASSPATH=.:$KAFKA_HOME/libs/*:$FLUME_HOME/lib/*:$HBASE_HOME/lib/*:$JAVA_

?????????????????????????HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

2.6?????啟動kafka

????????1）、啟動kafka前要先啟動zookeeper

????????$ zookeeper-server-start.sh /software/kafka/config/zookeeper.properties &

????????2）、啟動kafka

????????$ kafka-server-start.sh /software/kafka/config/server.properties &

3????體驗單節點Kafka

????3.1 創建topic

????????$ kafka-topics.sh --create --zookeeperlocalhost:2181 --replication-factor 1 --partitions 1 --topic test

?

????????查詢創建的topic

????????$ kafka-topics.sh --list --zookeeperlocalhost:2181

????3.2 發送消息

????????$ kafka-console-producer.sh --broker-listlocalhost:9092 --topic test

????????This is a message

????????This is another message

????3.3 消費消息

????????$ kafka-console-consumer.sh --bootstrap-serverlocalhost:9092 --zookeeper localhost:2181 --topic test --from-beginning

????????This is a message

????????This is another message

4????搭建多broker的Kafka集群

????4.1 復制配置文件

????????$ cd /software/kafka/config/

????????$ cp server.properties s1.properties

????????$ cp server.properties s2.properties

????????$ cp server.properties s3.properties

????4.2 修改配置文件

????config/s1.properties:

???????broker.id=1

???????listeners=PLAINTEXT://:9093

???????log.dir=/tmp/kafka-logs-1

?

????config/s2.properties:

???????broker.id=2

???????listeners=PLAINTEXT://:9094

????log.dir=/tmp/kafka-logs-2

?

????config/s3.properties:

???????broker.id=3

???????listeners=PLAINTEXT://:9095

????log.dir=/tmp/kafka-logs-3

4.3?????啟動kafka

????$ kafka-server-start.sh /software/kafka/config/s1.properties &

????$ kafka-server-start.sh /software/kafka/config/s2.properties &

????$ kafka-server-start.sh /software/kafka/config/s3.properties&

4.4?????創建topic

????$ kafka-topics.sh --create--zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic

?

????查看topic描述

????$ kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

4.5?????發送消息

????$ kafka-console-producer.sh --broker-listlocalhost:9093 --topic my-replicated-topic

my test msg 1

my test msg 2

?????my test msg 3

4.6?????消費消息

????$ kafka-console-consumer.sh--zookeeper localhost:2181 --from-beginning --topic my-replicated-topic

5????容錯測試

????5.1 找到s1進程

????????$ ps -ef| grep s1.properties

????5.2 殺死進程s1

kill pid

????5.3 查看主題描述

????????$>kafka-topics.sh --describe --zookeeperlocalhost:2181 --topic my-replicated-topic

????????Topic:my-replicated-topic????? PartitionCount:1?????? ????????????????ReplicationFactor:3??? Configs:

??????? Topic: my-replicated-topic???? Partition: 0?? Leader: 2????? Replicas:1,2,0 Isr: 2,0

????5.4 啟動消費者消費主題消息

????????$>kafka-console-consumer.sh--zookeeper localhost:2181 --from-beginning --topic my-replicated-topic

????????my?test?message?1

????????my?test?message?2

6????完全分布式搭建

????6.1 將kafka安裝文件夾分發到其余集群服務器

????????$ xsync kafka_2.11-0.10.0.1/

????????$ xsync kafka

????6.2 分發環境變量

????????#xsync /etc/profile

?6.3?????修改server.properties屬性文件并分發

????????broker.id=203 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?#修改broker的id

????????log.dirs=/home/hadoop/kafka/logs ? ? ? ? ? ? ? ? ? #修改日志的存放路徑

????????zookeeper.connect=s201:2181,s202:2181,s203:2181??? #修改zookeeper的連接路徑

6.4?????啟動Kafka集群

????到部署kafka每臺服務器上面啟動kafka集群

????$ kafka-server-start.sh /software/kafka/config/server.properties &

????$ kafka-server-start.sh /software/kafka/config/server.properties &

????$ kafka-server-start.sh /software/kafka/config/server.properties &

?

?