kafka学习指南（总结版）

kafka学习指南（总结版）
版本介绍
从使⽤上来看，以0.9为分界线，0.9开始不再区分⾼级（相当于mysql binlog的GTID，只需要跟topic打交道，服务器⾃动管理偏移量和负载均衡）/低级消费者API（相当于mysql binlog的⽂件+position，直接和分区以及偏移量打交道）。

同时，0.9版本增加了，相当于提供了⼀个封装好直接采⽤其它源同步数据（主要是批量模式）的框架，有利有弊吧，门槛低、让⼀般开发更依赖kafka。

从兼容性上来看，以0.8.x为分界线，0.8.x不兼容以前的版本。

总体拓扑架构
从上可知：
0、⼀个kafka集群由N个broker组成，⾄少⼀个（单机），其中⼀个broker是controller（其职责见下⽂）；集群中包含很多topic，topic可以分区，每个分区可以配置多个副本，每个副本肯定在不同的broker，其中⼀个副本为leader（写），其它为follower，可⽤于读（跟couchbase架构是⼀样的）。

1、⽣产者不需要访问zookeeper（0.8.x版本的kafka consumer直连zk得到偏移量信息，之后的版本直接从cluster获取，所以这两个版本的API并不兼容，上图是0.8x的结
构，0.9.x以及之后略有失真）。

2、消费者fetch消息、⽣产者发布消息总是向leader节点发请求，不会发送给follower（broker之间，⽽不是broker和客户端之间协调复制）。

4、kafka 0.8.x使⽤zk存储每个consumer-group在每个topic的每个partition中的点位（每个消息都有⼀个offset，且在分区内单调递增），0.9版本开始存储在专门的topic中，该topic名为"__consumer_offset"，这样consumer-group+topic就确定了点位，便于随时可恢复运⾏，采⽤⽇志压缩存储，也就是仅存储每个key的最新值，⽽⾮所有。

5、每个topic本地有⼀个local log，broker会持续顺序写⼊。

6、每条消息可以有key，也可以没有。

有的话，⽤于确定消息发往哪个parition，否则就是轮询机制，java中是对key应⽤hash（实际为了重复消费的问题，⼀般会设置key），每个分区内的记录是保证有序的，所以选择合适的key能够将串⾏转为并⾏，这个需要⾮常理解业务逻辑要求，很多时候，严格递增并⾮必须（OLTP更是如此，可以根据产品、客户、商家、甚⾄某⼀次活动），只是实现简单⽽已。

需要记住的是：是⽣产者⽽⾮broker决定去哪个分区。

7、在replicas模式下，⼀致性遵循的是全⼀致性模式，⽽⾮过半模式，如下：
ISRs见下⽂所述。

⼀个topic中的不同parition可以为不同broker中的leader，这种模式可以提⾼性能，因为读写都是leader负责。

committed记录所在的截⽌位置也成为⾼⽔位"High Watermark"。

虽然使⽤⾓度不直接care，但是partition是HA和扩展性的真正落地之处。

kafka⾃⾝整体架构
这⾥要提及的是controller，其中⼀个broker会被作为controller，controller主要负责处理kafka集群范围内的事件，包括leader选举、topic变化、paritions副本数跟踪、broker 的变化等，主要和zk通信，这根zk的主节点还不同，更像是Hadoop的nameserver，只负责元数据的全局管理。

kafka controller架构如下：
综合起来，kafka的核⼼要素包括：brokers(所有的进程都是broker), topics, logs, partitions, controller, message, cluster。

broker内部机制
消息发送过程
那客户端是怎么知道哪个broker是leader呢？因为每个broker都缓存了元数据，所以在连接初始建⽴的时候，客户端可以从任何⼀个broker获取每个topic的元数据信息，如下：消息消费过程
核⼼⾼级API（不允许⽤户控制消费者和broker的交互过程）
ConsumerConnector
KafkaStream
ConsumerConfig
低级API则允许控制各个交互过程，⽐如从哪⾥开始读以及在客户端维护点位，rocketmq实现其实采⽤的就是⾼层和底层结合的API，也就是kafka 0.9之后合并的api版本。

底层API的主要接⼝是SimpleConsumer。

消费者group和partition的关系
每个消费者group会记录⾃⼰在每个分区中的消费进度（该信息记录在专门的topic log中，见上⽂）。

⼀个分区只能由被每个消费者group中的任意⼀个消费者成员消费，因为⼀般情况下微服务都是集群部署，所以这会导致N-1个微服务节点中的topic listener空跑，这是需要注意的，但是如果当前消费者所在的服务挂了，kafka会⾃动选择其中⼀个剩下的consumer，但是如果已经消费但是ack未被kafka收到，其它consumer接管时就会重复消费，要注意幂等。

想要⼀个topic被消费者group中的成员并⾏消费的话，就需要配置不低于集群成员数的partition。

简单的说，就是管理粒度是消费者组（在其他MQ中称订阅者）和topic，底层消息接收粒度分区和消费者。

不仅集群微服务可以从多partition受益，单JVM也可以收益，只要启动多个独⽴的线程，每个线程都作为topic的consumer就可以并发处理，这主要⽤于SMP服务器的时候，所以当消息处理需要⼀定时间或消息TPS⼤的时候，都应该使⽤多parition。

⼏个关键的消息点位
消息堆积是消费滞后(Lag)的⼀种表现形式，消息中间件服务端中所留存的消息与消费掉的消息之间的差值即为消息堆积量，也称之为消费滞后(Lag)量。

对于Kafka⽽⾔，消息被发送⾄Topic中，⽽Topic⼜分成了多个分区(Partition)，每⼀个Partition都有⼀个预写式的⽇志⽂件，虽然Partition可以继续细分为若⼲个段⽂件(Segment)，但是对于上层应⽤来说可以将Partition看成最⼩的存储单元(⼀个由多个Segment⽂件拼接的“巨型⽂件”)。

每个Partition都由⼀系列有序的、不可变的消息组成，这些消息被连续的追加到Partition 中。

上图中有四个概念：
1.LogStartOffset：表⽰⼀个Partition的起始位移，初始为0，虽然消息的增加以及⽇志清除策略的影响，这个值会阶段性的增⼤。

2.ConsumerOffset：消费位移，表⽰Partition的某个消费者消费到的位移位置。

3.HighWatermark：简称HW，代表消费端所能“观察”到的Partition的最⾼⽇志位移，HW⼤于等于ConsumerOffset的值。

4.LogEndOffset：简称LEO, 代表Partition的最⾼⽇志位移，其值对消费者不可见。

⽐如在ISR（In-Sync-Replicas）副本数等于3的情况下（如下图所⽰），消息发送到Leader A 之后会更新LEO的值，Follower B和Follower C也会实时拉取Leader A中的消息来更新⾃⼰，HW就表⽰A、B、C三者同时达到的⽇志位移，也就是A、B、C三者中LEO最⼩的那个值。

由于B、C拉取A消息之间延时问题（这就涉及到发送可靠性问题，见下⽂），所以HW必然不会⼀直与Leader的LEO相等，即LEO>=HW。

所以，实际可能是这样的：
要计算Kafka中某个消费者的滞后量很简单，⾸先看看其消费了⼏个Topic，然后针对每个Topic来计算其中每个Partition的Lag，每个Partition的Lag计算就显得⾮常的简单了，参考下图：
由图可知消费Lag=HW - ConsumerOffset。

ISRs
ISR是指当前同步副本集中的成员，如果leader失败，其中⼀个ISR会被选为的新的leader。

Topic Log Compaction
kafka的topic log会持续增长，所以为了保持稳定，应该定期回收。

这涉及到两⽅⾯：消息的key是否会相同，它们的策略是不同的。

Log Compaction主要⽤于key会相同的情况，也就是⾮UUID作为消息的键，否则就没有意义了。

其机制是根据消息保留的时间或⽂件⼤⼩来删除key相同的历史value，如下所⽰：
可知，历史版本被清了。

启⽤compact后，topic log分为了head和tail部分，只有tail的才会被压缩，但是删除还要根据其它配置决定，如下。

kafka参数g.ms控制消息⾄少过多久才会被压缩，delete.retention.ms控制多久会被删除，log.cleanup.policy=compact控制启⽤压缩，所以消费者只要在此之内进⾏消费，就可以保证⾄少看到最新记录（因为producer可能⼜写⼊了，所以⾄少会看到最新，也许更多）。

消息保留时长
每个topic可以基于时间或topic log的⼤⼩声明消息的保留时间，由下列参数决定：
属性名含义默认值
log.cleanup.polict⽇志清理保存的策略只有delete和compact两种delete
log.retention.hours⽇志保存的时间，可以选择hours,minutes和ms168(7day)
log.retention.bytes删除前⽇志⽂件允许保存的最⼤值（任意⼀个达到都会执⾏删除）-1
log.segment.delete.delay.ms⽇志⽂件被真正删除前的保留时间60000
log.cleanup.interval.mins每隔⼀段时间多久调⽤⼀次清理的步骤10
300000
log.retention.check.interval.ms周期性检查是否有⽇志符合删除的条件（新版本使⽤）
ACK⼀致性级别
⽣产者（现在⾯试，我们都问如何保证发出的消息不丢失）可以通过ack设置数据⼀致性要求（和mysql机制类似）。

ack=0（不需要ACK，⾄多⼀次）, ack=all（leader和所有follows都写⼊成功，默认）, ack=1（leader成功即可）。

可以通过在producer properties中设置，如下：
props.put("acks", "0");
props.put("acks", "1");
props.put("acks", "all");
在消费者层⾯，kafka⽀持⾄多⼀次和⾄少⼀次两种模式。

To implement “at-most-once” consumer reads a message, then saves its offset in the partition by sending it to the broker, and finally process the message. The issue with “at-most-once” is a consumer could die after saving its position but before processing the message. Then the consumer that takes over or gets restarted would leave off at the last position and message in question is never processed.
To implement “at-least-once” the consumer reads a message, process messages, and finally saves offset to the broker. The issue with “at-least-once” is a consumer could crash after processing a message but before saving last offset position. Then if the consumer is restarted or another consumer takes over, the consumer could receive the message that was already processed. The “at-least-once” is the most common set up for messaging, and it is your responsibility to make the messages idempotent, which means getting the same message twice will not cause a problem (two debits).
To implement “exactly once” on the consumer side, the consumer would need a two-phase commit between storage for the consumer position, and storage of the consumer’s message process output. Or, the consumer could store the message process output in the same location as the last offset.
kafka仅⽀持前两种消费者ACK，第三种需要⽤户⾃⼰实现，⼀般⼤家都是⽤第⼆种+幂等来实现，也就是消费者⾃⾝的⼀致性，通过幂等+ACK保证，就不重复阐述了。

通过如下可以保证⼿⼯管理ack提交：
props.put("mit", "false");
try {
while (running) {
ConsumerRecords<String, String> records = consumer.poll(1000);
for (ConsumerRecord<String, String> record : records)
System.out.println(record.offset() + ": " + record.value());
try {
mitSync();
} catch (CommitFailedException e) {
// application specific failure handling
}
}
} finally {
consumer.close();
}
Kafka的⽣态
MirrorMaker是kafka集群之间同步的组件，本质上是⼀个⽣产者+消费者，如下：
如上所⽰，它已经到V2版本（要求kafka 2.4），V2相⽐V1⽽⾔，最重要的是解决了两个集群之间消费者点位的⾃动换算问题，⽽不需要依赖于kafka的时间戳来计算（这⼀步⾮常关键，否则灾备切换成本过⾼、且容易出错）。

内部架构如下：
上图在实际部署中其实是不太正确的，consumer（应⽤）肯定是两个机房都部署了，且双活存在。

这时候就有个问题了，__consumer_offset和业务topic本⾝到达的先后问题。

如果业务先，__consumer_offset后，那么灾备中⼼的应⽤读取到的就是⽼的offset，这样就会重复消费。

如果__consumer_offset先，也有个问题，超过了实际的长度，则会重置为实际的offset，后到的业务topic也会被重复消费。

所以作为应⽤消费者这⾥，幂等极为重要，其次作为topic设计者，确定message的key唯⼀⾮常重要。

如果延时太⼤，导致的问题可能是⼤量的重复消息要丢弃，进⽽影响RTO。

除此之外，Kafka包括所有MQ在灾备时还存在⼀个很严重的问题，依赖于MQ实现最终⼀致性的破坏，通常对于最终⼀致性，依赖MQ可以极⼤的提升可靠性和性能、并解耦微服务，如果消息丢了，那么意味着最终⼀致性的破坏，需要依赖于微服务间的对账机制。

Kafka REST Proxy and Confluent Schema Registry
kafka在zk中的存储
为了显⽰⽅便，LZ设置了chroot为localKakfa，如下：
各个zk节点的含义如下⽰意图所⽰（之前版本的kafka，consumer-offset也维护在zk中），其中kafka01就是chroot，在kafka的server.properties中设置，加载zookeeper.connect后即可。

如zookeeper.connect=localhost:2181/localKafka。

kafka监控管理
主流的⼏种kafka监控程序分别为：
1、Kafka Web Conslole，从https:///claudemamo/kafka-web-console下载源码编译，https:///s/1cJ2OefPqlxfWzTHElG6AjA 下载编译好的，提取
码:f8xz，如果是测试和开发、为了排查⽅便，推荐使⽤它，其缺陷参见https:///qq_33314107/article/details/81099091。

[root@node1 bin]# sh kafka-web-console
Play server process ID is 4154
[info] play - database [default] connected at jdbc:h2:file:play
[warn] play - Your production database [default] needs evolutions!
INSERT INTO settings (key_, value) VALUES ('PURGE_SCHEDULE', '0 0 0 ? * SUN *');
INSERT INTO settings (key_, value) VALUES ('OFFSET_FETCH_INTERVAL', '30');
[warn] play - Run with -DapplyEvolutions.default=true if you want to run them automatically (be careful)
Oops, cannot start the server.
@74e0p173o: Database 'default' needs evolution!
at play.api.db.evolutions.EvolutionsPlugin$$anonfun$onStart$1$$anonfun$apply$1.apply$mcV$sp(Evolutions.scala:484)
at play.api.db.evolutions.EvolutionsPlugin.withLock(Evolutions.scala:507)
at play.api.db.evolutions.EvolutionsPlugin$$anonfun$onStart$1.apply(Evolutions.scala:461)
at play.api.db.evolutions.EvolutionsPlugin$$anonfun$onStart$1.apply(Evolutions.scala:459)
at scala.collection.immutable.List.foreach(List.scala:318)
at play.api.db.evolutions.EvolutionsPlugin.onStart(Evolutions.scala:459)
at play.api.Play$$anonfun$start$1$$anonfun$apply$mcV$sp$1.apply(Play.scala:88)
at play.api.Play$$anonfun$start$1$$anonfun$apply$mcV$sp$1.apply(Play.scala:88)
at scala.collection.immutable.List.foreach(List.scala:318)
at play.api.Play$$anonfun$start$1.apply$mcV$sp(Play.scala:88)
at play.api.Play$$anonfun$start$1.apply(Play.scala:88)
at play.api.Play$$anonfun$start$1.apply(Play.scala:88)
at play.utils.Threads$.withContextClassLoader(Threads.scala:18)
at play.api.Play$.start(Play.scala:87)
at play.core.StaticApplication.<init>(ApplicationProvider.scala:52)
at tyServer$.createServer(NettyServer.scala:243)
at tyServer$$anonfun$main$3.apply(NettyServer.scala:279)
at tyServer$$anonfun$main$3.apply(NettyServer.scala:274)
at scala.Option.map(Option.scala:145)
at tyServer$.main(NettyServer.scala:274)
at tyServer.main(NettyServer.scala)
第⼀次启动时要加个参数：
./kafka-web-console -DapplyEvolutions.default=true
2、Kafka Manager，也不错，yahoo开发的，可⽤来监控流量。

3、KafkaOffsetMonitor
4、，新开发的，参见https:///HomeAdvisor/Kafdrop。

可以通过java -cp KafkaOffsetMonitor-assembly-0.2.0.jar com.quantifind.kafka.offsetapp.OffsetGetterWeb --zk 10.20.30.10:2181 --port 8088 --refresh 10.seconds --retain 2.days启动，各配置含义可以参考github。

（重要）
常见问题
如何通过java api获取所有topic？
消费和如何⼀次性订阅多个topic？
它适合所有topic具有相同的语义范围时，可通过this.consumer.subscribe(Arrays.asList(topic));订阅多个主题。

⽐如缓存通常要求每个应⽤实例都消费、所以
ip+port+app.group+app.version作为consumer-group-id⽐较合适，否则app.group+app.version作为consumer-group-id更合适。

如何查看所有的topic？
[root@hs-test-10-20-30-16 bin]# ./kafka-topics.sh --list --zookeeper 10.20.30.17:2191
__consumer_offsets
hs_ta_channel_Metadata
hsta_channelprocess20100513_aop-YOGA - marked for deletion
hsta_channelprocess20100514_aop-TA4 - marked for deletion
hsta_channelprocess20100530_aop-TA4
hsta_channelprocess20200118_aop-YOGA
hsta_channelprocessnull - marked for deletion
和连接--bootstrap-server出来的结果不⼀样，如下：
[root@ta5host bin]# ./kafka-topics.sh --list --bootstrap-server 10.20.30.17:9092
__consumer_offsets
hs_ta_channel_Metadata
uft_individual_1
uft_individual_1_oracle
uft_inst_1
uft_inst_1_oracle
uft_spliter_1_oracle
uft_spliter_2_oracle
uft_splitter_1
uft_trade_1
uft_trade_1_bar
uft_trade_1_oracle
uft_trade_1_oracle_bar
uft_trade_2_oracle
uft_trade_2_oracle_bar
是不是哪⾥不对
前者和kafka-manager的监控结果是⼀样的，毕竟连接的是同⼀个zk。

要想知道哪些kafka集群连接本zk，可以通过netstat -ano | grep 2191找到客户端。

查看特定topic的配置？
[root@hs-test-10-20-30-11 kafka]# bin/kafka-topics.sh --zookeeper 10.20.30.10:2181 --topic global --describe
Topic:global PartitionCount:1 ReplicationFactor:1 Configs:
Topic: global Partition: 0 Leader: 0 Replicas: 0 Isr: 0
[root@hs-test-10-20-30-10 bin]# ./kafka-topics.sh --bootstrap-server localhost:9092 --topic hs_ta_channel_Metadata --describe
Topic:hs_ta_channel_Metadata PartitionCount:1 ReplicationFactor:1 Configs:min.insync.replicas=1,segment.bytes=1073741824,max.message.bytes=10485760
Topic: hs_ta_channel_Metadata Partition: 0 Leader: 0 Replicas: 0 Isr: 0
如何删除topic?
删除kafka topic及其数据，严格来说并不是很难的操作。

但是，往往给kafka 使⽤者带来诸多问题。

项⽬组之前接触过多个开发者，发现都会偶然出现⽆法彻底删除kafka的情况。

本⽂总结多个删除kafka topic的应⽤场景，总结⼀套删除kafka topic的标准操作⽅法。

step1：
如果需要被删除topic 此时正在被程序 produce和consume，则这些⽣产和消费程序需要停⽌。

因为如果有程序正在⽣产或者消费该topic，则该topic的offset信息⼀致会在broker更新。

调⽤kafka delete命令则⽆法删除该topic。

同时，需要设置 auto.create.topics.enable = false，默认设置为true。

如果设置为true，则produce或者fetch 不存在的topic也会⾃动创建这个topic。

这样会给删除topic带来很多意向不到的问题。

所以，这⼀步很重要，必须设置auto.create.topics.enable = false，并认真把⽣产和消费程序彻底全部停⽌。

step2：
server.properties 设置 delete.topic.enable=true
如果没有设置 delete.topic.enable=true，则调⽤kafka 的delete命令⽆法真正将topic删除，⽽是显⽰（marked for deletion）
step3：
调⽤命令删除topic：
./bin/kafka-topics --delete --zookeeper 【zookeeper server:port】 --topic 【topic name】
step4：
删除kafka存储⽬录（server.properties⽂件log.dirs配置，默认为"/data/kafka-logs"）相关topic的数据⽬录。

注意：如果kafka 有多个 broker，且每个broker 配置了多个数据盘（⽐如 /data/kafka-logs,/data1/kafka-logs ...），且topic也有多个分区和replica，则需要对所有broker的所有数据盘进⾏扫描，删除该topic的所有分区数据。

⼀般⽽⾔，经过上⾯4步就可以正常删除掉topic和topic的数据。

但是，如果经过上⾯四步，还是⽆法正常删除topic，则需要对kafka在zookeeer的存储信息进⾏删除。

具体操作如下：
（注意：以下步骤⾥⾯，kafka在zk⾥⾯的节点信息是采⽤默认值，如果你的系统修改过kafka在zk⾥⾯的节点信息，则需要根据系统的实际情况找到准确位置进⾏操作）
step5：
找⼀台部署了zk的服务器，使⽤命令：
bin/zkCli.sh -server 【zookeeper server:port】
登录到zk shell，然后找到topic所在的⽬录：ls /brokers/topics，找到要删除的topic，然后执⾏命令：
rmr /brokers/topics/【topic name】
即可，此时topic被彻底删除。

如果topic 是被标记为 marked for deletion，则通过命令 ls /admin/delete_topics，找到要删除的topic，然后执⾏命令：
rmr /admin/delete_topics/【topic name】
备注：
⽹络上很多其它⽂章还说明，需要删除topic在zk上⾯的消费节点记录、配置节点记录，⽐如：
rmr /consumers/【consumer-group】
rmr /config/topics/【topic name】
其实正常情况是不需要进⾏这两个操作的，如果需要，那都是由于操作不当导致的。

⽐如step1停⽌⽣产和消费程序没有做，step2没有正确配置。

也就是说，正常情况下严格按照step1 -- step5 的步骤，是⼀定能够正常删除topic的。

step6：
完成之后，调⽤命令：
./bin/kafka-topics.sh --list --zookeeper 【zookeeper server:port】
查看现在kafka的topic信息。

正常情况下删除的topic就不会再显⽰。

但是，如果还能够查询到删除的topic，则重启zk和kafka即可。

如何查看所有的消费者组？
新的⽅式，也就是不是使⽤基于zk的客户端（kafka.consumer.Consumer.createJavaConsumerConnector、内部是bootstrap）。

[root@hs-test-10-20-30-11 kafka]# bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server 10.20.30.11:9092 --list
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
⽼的⽅式：基于zk的客户端（kafka.javaapi.consumer.ZookeeperConsumerConnector，已经deprecated）。

[root@hs-test-10-20-30-11 kafka]# bin/kafka-consumer-groups.sh --zookeeper 10.20.30.10:2181 --list
Note: This will only show information about consumers that use ZooKeeper (not those using the Java consumer API).
AAA
TA50-Aggr-Logger-ConsumerGroup
console-consumer-23104
console-consumer-37858
查看⼀个消费者组的消费点位
[root@hs-test-10-20-30-10 bin]# ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-topics --group tabase-service-10.20.30.10:8383
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
hs_ta_channel_Metadata 0440 consumer-1-b8c4c0c5-dbf3-4a55-8a9d-fda9d7b00c17 /10.20.30.10 consumer-1
[root@hs-test-10-20-30-10 bin]# ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-topics --group tabase-10.20.30.10:8080
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
hs_ta_channel_Metadata 0440 consumer-1-1681831e-8c97-48bc-9c8d-21f2f7667aa9 /10.20.30.10 consumer-1
[root@hs-test-10-20-30-10 bin]# ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-topics --group tabase-10.20.30.10:8080 --verbose --state
COORDINATOR (ID) ASSIGNMENT-STRATEGY STATE #MEMBERS
10.20.30.10:9092 (0) range Stable 1
⽣产者连接的时候报了下列错误
WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (workClient)
有两个原因：1、kafka没有启动；2、连接串使⽤了⾮conf/server.properties⾥⾯的LISTENERS参数的值。

log4j-kafka配置
增加jar包依赖：
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.12</artifactId>
<version>0.11.0.3</version>
</dependency>
配置log4j2.xml，如下：
logger增加kafka appender。

<Root level="INFO" additivity="false">
<AppenderRef ref="Console"/>
<AppenderRef ref="KAFKA"/>
<AppenderRef ref="app_error"/>
</Root>
增加kafka appender。

<Appenders>
<!-- 输出错误⽇志到Kafka -->
<Kafka name="KAFKA" topic="bomp">
<ThresholdFilter level="error" onMatch="DENY" onMismatch="NEUTRAL"/>
<ThresholdFilter level="trace" onMatch="ACCEPT" onMismatch="DENY"/>
<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS}:%4p %t (%F:%L) - %m%n"/>
<Property name="bootstrap.servers">10.20.30.11:9092</Property>
</Kafka>
</Appenders>
相关异常
消费者报：
2018-09-17 14:10:07.768 WARN 130400 --- [r-finder-thread] kafka.client.ClientUtils$ : Fetching topic metadata with correlation id 0 for topics [Set(test)] from broker [BrokerEndPoint(0,10.20.30.11,9092)] failed
java.nio.channels.ClosedChannelException: null
at work.BlockingChannel.send(BlockingChannel.scala:112) ~[kafka_2.12-0.11.0.3.jar:na]
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:80) ~[kafka_2.12-0.11.0.3.jar:na]
at kafka.producer.SyncProducer.doSend(SyncProducer.scala:79) ~[kafka_2.12-0.11.0.3.jar:na]
at kafka.producer.SyncProducer.send(SyncProducer.scala:124) ~[kafka_2.12-0.11.0.3.jar:na]
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:61) [kafka_2.12-0.11.0.3.jar:na]
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:96) [kafka_2.12-0.11.0.3.jar:na]
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:72) [kafka_2.12-0.11.0.3.jar:na]
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64) [kafka_2.12-0.11.0.3.jar:na]
zk⽇志中报：
2018-10-08 14:13:28,297 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0x100147743c10000 type:setData cxid:0xc8 zxid:0x53 txntype:-1 reqpath:n/a Error Path:/config/topics/uft_trade Error:KeeperErrorCode = NoNode for
/config/topics/uft_trade
2018-10-08 14:13:28,302 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0x100147743c10000 type:create cxid:0xc9 zxid:0x54 txntype:-1 reqpath:n/a Error Path:/config/topics Error:KeeperErrorCode = NodeExists for /config/topics
解决⽅法：待排查。

spring boot kafka客户端在某虚拟机服务器（物理机⼀直运⾏未发⽣）上运⾏⼀段时间后，瞬间cpu system 80-90%，⼤量下列⽇志：
2018-10-09 13:54:57,713 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 2682ms for sessionid 0x100175687960002
2018-10-09 13:54:57,904 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 2672ms for sessionid 0x100175687960004
2018-10-09 13:54:58,621 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 2675ms for sessionid 0x100175687960003
2018-10-09 13:54:57,232 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 2700ms for sessionid 0x100175687960007
2018-10-09 13:55:09,812 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 2672ms for sessionid 0x100175687960004, closing socket connection and attempting reconn ect
2018-10-09 13:55:02,942 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 2702ms for sessionid 0x100175687960008
2018-10-09 13:55:09,755 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 2675ms for sessionid 0x100175687960003, closing socket connection and attempting reconn ect
2018-10-09 13:55:09,789 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 2682ms for sessionid 0x100175687960002, closing socket connection and attempting reconn ect
2018-10-09 13:55:18,677 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 2675ms for sessionid 0x100175687960005
2018-10-09 13:55:11,752 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 20016ms for sessionid 0x100175687960001
2018-10-09 13:55:17,709 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 2678ms for sessionid 0x100175687960006
2018-10-09 13:55:12,779 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 2700ms for sessionid 0x100175687960007, closing socket connection and attempting reconn ect
2018-10-09 13:55:20,634 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 2702ms for sessionid 0x100175687960008, closing socket connection and attempting reconn ect
2018-10-09 13:55:22,178 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 20016ms for sessionid 0x100175687960001, closing socket connection and attempting recon nect
2018-10-09 13:58:10,244 INFO ZkClient:713 - zookeeper state changed (Disconnected)
2018-10-09 13:58:10,240 INFO ZkClient:713 - zookeeper state changed (Disconnected)
2018-10-09 13:58:10,241 INFO ZkClient:713 - zookeeper state changed (Disconnected)
2018-10-09 13:58:10,240 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 2675ms for sessionid 0x100175687960005, closing socket connection and attempting reconn ect
2018-10-09 13:58:10,243 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 2678ms for sessionid 0x100175687960006, closing socket connection and attempting reconn ect
2018-10-09 13:58:11,107 INFO ZkClient:713 - zookeeper state changed (Disconnected)
2018-10-09 13:58:40,384 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 13:58:40,383 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 13:58:40,379 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 13:58:40,378 INFO ZkClient:713 - zookeeper state changed (Disconnected)
2018-10-09 13:58:40,378 INFO ZkClient:713 - zookeeper state changed (Disconnected)
2018-10-09 13:58:40,377 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 13:59:22,082 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 13:59:22,084 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 13:59:22,099 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 13:59:22,108 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 13:59:22,130 INFO ZkClient:713 - zookeeper state changed (Disconnected)
2018-10-09 13:59:23,382 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 13:59:23,412 INFO ZkClient:713 - zookeeper state changed (Expired)
2018-10-09 13:59:23,412 INFO ZkClient:713 - zookeeper state changed (Expired)
2018-10-09 13:59:23,443 INFO ZooKeeper:438 - Initiating client connection, connectString= sessionTimeout=500 watcher=org.I0Itec.zkclient.ZkClient@8646db9
2018-10-09 13:59:23,411 WARN ClientCnxn:1285 - Unable to reconnect to ZooKeeper service, session 0x100175687960001 has expired
2018-10-09 13:59:32,474 INFO ZkClient:713 - zookeeper state changed (Disconnected)
2018-10-09 13:59:23,404 WARN ClientCnxn:1285 - Unable to reconnect to ZooKeeper service, session 0x100175687960007 has expired
2018-10-09 13:59:23,390 INFO ZkClient:713 - zookeeper state changed (Expired)
2018-10-09 13:59:32,477 INFO ZooKeeper:438 - Initiating client connection, connectString= sessionTimeout=500 watcher=org.I0Itec.zkclient.ZkClient@4671e53b
2018-10-09 13:59:23,390 WARN ClientCnxn:1285 - Unable to reconnect to ZooKeeper service, session 0x100175687960008 has expired
2018-10-09 13:59:23,390 INFO ZkClient:713 - zookeeper state changed (Expired)
2018-10-09 13:59:32,477 INFO ZooKeeper:438 - Initiating client connection, connectString= sessionTimeout=500 watcher=org.I0Itec.zkclient.ZkClient@6a1aab78
2018-10-09 13:59:23,389 WARN ClientCnxn:1285 - Unable to reconnect to ZooKeeper service, session 0x100175687960004 has expired
2018-10-09 13:59:32,417 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 13:59:23,380 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 13:59:23,446 INFO ZooKeeper:438 - Initiating client connection, connectString= sessionTimeout=30000 watcher=org.I0Itec.zkclient.ZkClient@dc24521
2018-10-09 13:59:41,829 INFO ClientCnxn:1154 - Unable to reconnect to ZooKeeper service, session 0x100175687960004 has expired, closing socket connection
2018-10-09 13:59:41,832 INFO ZkClient:936 - Waiting for keeper state SyncConnected
2018-10-09 13:59:41,829 INFO ClientCnxn:1154 - Unable to reconnect to ZooKeeper service, session 0x100175687960008 has expired, closing socket connection
2018-10-09 13:59:41,831 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 13:59:41,830 INFO ClientCnxn:1154 - Unable to reconnect to ZooKeeper service, session 0x100175687960007 has expired, closing socket connection
2018-10-09 13:59:41,830 INFO ClientCnxn:1154 - Unable to reconnect to ZooKeeper service, session 0x100175687960001 has expired, closing socket connection
2018-10-09 13:59:41,860 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 13:59:42,585 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 13:59:42,810 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 13:59:42,835 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 14:00:31,813 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 48978ms for sessionid 0x100175687960002
2018-10-09 14:00:31,825 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 49644ms for sessionid 0x100175687960005
2018-10-09 14:00:31,825 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 49644ms for sessionid 0x100175687960005, closing socket connection and attempting recon
nect
2018-10-09 14:00:31,827 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 49968ms for sessionid 0x100175687960006
2018-10-09 14:00:31,827 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 49968ms for sessionid 0x100175687960006, closing socket connection and attempting recon
nect
2018-10-09 14:00:31,842 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 50011ms for sessionid 0x100175687960003
2018-10-09 14:00:31,868 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 50011ms for sessionid 0x100175687960003, closing socket connection and attempting recon
nect
2018-10-09 14:00:31,853 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 48978ms for sessionid 0x100175687960002, closing socket connection and attempting recon
nect
2018-10-09 14:00:31,885 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 14:00:31,886 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 14:00:31,887 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 14:00:31,887 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 14:00:31,907 INFO ClientCnxn:519 - EventThread shut down for session: 0x100175687960001
2018-10-09 14:00:31,907 INFO ClientCnxn:519 - EventThread shut down for session: 0x100175687960008
2018-10-09 14:00:31,908 INFO ClientCnxn:519 - EventThread shut down for session: 0x100175687960004
2018-10-09 14:00:31,944 INFO ClientCnxn:519 - EventThread shut down for session: 0x100175687960007
2018-10-09 14:00:33,391 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 14:00:33,396 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 14:00:33,424 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 1336ms for sessionid 0x0
2018-10-09 14:00:33,430 INFO ClientCnxn:1299 - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x10017568796000b, negotiated timeout = 30000
2018-10-09 14:00:33,517 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 14:00:33,516 INFO ZkClient:713 - zookeeper state changed (SyncConnected)
2018-10-09 14:00:34,399 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 14:00:34,354 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 1336ms for sessionid 0x0, closing socket connection and attempting reconnect
2018-10-09 14:00:34,433 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 14:00:34,475 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 14:00:34,476 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 14:00:34,485 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 968ms for sessionid 0x0
2018-10-09 14:00:34,488 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 968ms for sessionid 0x0, closing socket connection and attempting reconnect
2018-10-09 14:00:37,472 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 14:00:37,484 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 14:00:37,487 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 14:00:37,488 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 14:00:37,489 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 14:00:37,479 WARN ClientCnxn:1285 - Unable to reconnect to ZooKeeper service, session 0x100175687960006 has expired
2018-10-09 14:00:37,495 INFO ClientCnxn:1154 - Unable to reconnect to ZooKeeper service, session 0x100175687960006 has expired, closing socket connection
2018-10-09 14:00:37,447 INFO ZkClient:713 - zookeeper state changed (Expired)
2018-10-09 14:00:37,479 INFO ZkClient:713 - zookeeper state changed (Expired)
2018-10-09 14:00:37,519 INFO ZooKeeper:438 - Initiating client connection, connectString= sessionTimeout=500 watcher=org.I0Itec.zkclient.ZkClient@69b0fd6f
2018-10-09 14:00:37,519 INFO ZooKeeper:438 - Initiating client connection, connectString= sessionTimeout=500 watcher=org.I0Itec.zkclient.ZkClient@4a87761d
2018-10-09 14:00:37,446 WARN ClientCnxn:1285 - Unable to reconnect to ZooKeeper service, session 0x100175687960005 has expired
2018-10-09 14:00:37,519 INFO ClientCnxn:1154 - Unable to reconnect to ZooKeeper service, session 0x100175687960005 has expired, closing socket connection
2018-10-09 14:00:37,765 INFO ZkClient:713 - zookeeper state changed (Expired)
2018-10-09 14:00:37,780 INFO ZkClient:713 - zookeeper state changed (Expired)
2018-10-09 14:00:37,780 WARN ClientCnxn:1285 - Unable to reconnect to ZooKeeper service, session 0x100175687960003 has expired
2018-10-09 14:00:37,791 INFO ClientCnxn:1154 - Unable to reconnect to ZooKeeper service, session 0x100175687960003 has expired, closing socket connection
2018-10-09 14:00:38,194 INFO ZooKeeper:438 - Initiating client connection, connectString= sessionTimeout=500 watcher=org.I0Itec.zkclient.ZkClient@3aeaafa6
2018-10-09 14:00:37,995 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 507ms for sessionid 0x0
2018-10-09 14:00:52,148 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 507ms for sessionid 0x0, closing socket connection and attempting reconnect
2018-10-09 14:00:38,198 INFO ZooKeeper:438 - Initiating client connection, connectString= sessionTimeout=500 watcher=org.I0Itec.zkclient.ZkClient@491cc5c9
2018-10-09 14:00:52,141 INFO ClientCnxn:519 - EventThread shut down for session: 0x100175687960006
2018-10-09 14:00:52,128 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 14:00:52,154 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 14:00:52,126 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 14:00:52,179 INFO ClientCnxn:876 - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-10-09 14:00:38,010 WARN ClientCnxn:1285 - Unable to reconnect to ZooKeeper service, session 0x100175687960002 has expired
2018-10-09 14:00:52,231 INFO ClientCnxn:1154 - Unable to reconnect to ZooKeeper service, session 0x100175687960002 has expired, closing socket connection
2018-10-09 14:00:52,683 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 504ms for sessionid 0x0
2018-10-09 14:05:12,238 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 14:05:12,176 INFO ClientCnxn:1032 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-09 14:08:21,078 INFO ClientCnxn:519 - EventThread shut down for session: 0x100175687960002
2018-10-09 14:05:12,113 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 259911ms for sessionid 0x10017568796000b
2018-10-09 14:08:21,107 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 259911ms for sessionid 0x10017568796000b, closing socket connection and attempting reco
nnect
2018-10-09 14:05:12,098 INFO ClientCnxn:519 - EventThread shut down for session: 0x100175687960003
2018-10-09 14:00:52,677 WARN ClientCnxn:1108 - Client session timed out, have not heard from server in 501ms for sessionid 0x0
2018-10-09 14:08:21,107 INFO ClientCnxn:1156 - Client session timed out, have not heard from server in 501ms for sessionid 0x0, closing socket connection and attempting reconnect
14时00分28秒 sda 3062.38 922268.58 670.77 301.38 5.17 1.71 0.16 49.44
14时00分28秒 ol-root 3111.77 922266.41 495.79 296.54 5.29 1.70 0.16 49.43
14时00分28秒 ol-swap 22.04 2.09 174.24 8.00 0.13 5.80 0.15 0.33
14时11分16秒 sda 5432.75 1537105.34 768.61 283.07 19.06 3.53 0.17 91.53
14时11分16秒 ol-root 5513.26 1537106.56 731.82 278.93 19.55 3.54 0.17 91.52
14时11分16秒 ol-swap 5.07 4.68 35.87 8.00 0.01 2.27 0.19 0.10
14时11分16秒 DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
14时20分01秒 sda 2784.00 795332.59 462.60 285.85 10.89 3.93 0.18 50.09
14时20分01秒 ol-root 2827.44 795311.85 414.30 281.43 11.18 3.95 0.18 50.07
14时20分01秒 ol-swap 6.96 12.98 42.72 8.00 0.05 7.80 0.18 0.12
14时30分01秒 sda 3.13 12.42 59.59 23.04 0.00 0.57 0.44 0.14
但是这段时间没有东西特别在运⾏，这就⽐较奇怪了，那会⼉⼀下⼦也忘了⽤iotop看下是哪个进程所致。

上述帖⼦提到的⼏点是：
关于ZK⽇志存放，官⽹给出如下建议:
Having a dedicated log devicehas a large impact on throughput and stable latencies. It is highly recommenedto dedicate a log device and set dataLogDir to point to a directory on thatdevice, and then make sure to point dataDir to a directory not residing on thatdevice.
在ZOO.CFG中增加：
forceSync=no
默认是开启的，为避免同步延迟问题，ZK接收到数据后会⽴刻去讲当前状态信息同步到磁盘⽇志⽂件中，同步完成后才会应答。

将此项关闭后，客户端连接可以得到快速响应（这⼀点在有BMU的服务器上问题不⼤）。

再看下zk服务器的⽇志，差不多时间开始出现⼤量CancelledKeyException：
2018-10-09 13:56:36,712 [myid:] - INFO [SyncThread:0:NIOServerCnxn@1040] - Closed socket connection for client /127.0.0.1:14926 which had sessionid 0x100175687960008
2018-10-09 13:56:43,857 [myid:] - INFO [SyncThread:0:NIOServerCnxn@1040] - Closed socket connection for client /127.0.0.1:14924 which had sessionid 0x100175687960006
2018-10-09 13:56:49,783 [myid:] - INFO [SyncThread:0:NIOServerCnxn@1040] - Closed socket connection for client /127.0.0.1:14919 which had sessionid 0x100175687960001
2018-10-09 13:56:49,816 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@236] - Ignoring unexpected runtime exception
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:205)
at ng.Thread.run(Thread.java:748)
2018-10-09 13:58:54,331 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@941] - Client attempting to renew session 0x100175687960000 at /192.168.223.137:23459
2018-10-09 13:58:54,377 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@686] - Invalid session 0x100175687960000 for client /192.168.223.137:23459, probably expired
2018-10-09 13:58:54,401 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@215] - Accepted socket connection from /192.168.223.137:23485
2018-10-09 13:58:54,441 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@215] - Accepted socket connection from /192.168.223.137:23494
2018-10-09 13:58:56,314 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1040] - Closed socket connection for client /192.168.223.137:23459 which had sessionid 0x10017 5687960000
2018-10-09 13:58:56,336 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@941] - Client attempting to renew session 0x100175687960000 at /192.168.223.137:23485
2018-10-09 13:58:56,392 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@686] - Invalid session 0x100175687960000 for client /192.168.223.137:23485, probably expired
2018-10-09 13:58:57,890 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@215] - Accepted socket connection from /192.168.223.137:23497
2018-10-09 13:58:59,480 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1040] - Closed socket connection for client /192.168.223.137:23485 which had sessionid 0x10017。