We have a live Zookeeper environment (quorum size is 2) and observed a strange
behavior:
Kafka created 2 ephemeral nodes /brokers/ids/822712429 and
/brokers/ids/707577499 on 2018-03-12 03:30:36.933
The Kafka clients were long gone but as of today, the two ephemeral nodes are
still present
Troubleshooting:
1) Lists the outstanding sessions and ephemeral nodes
$ echo dump | nc $SERVER1 2181
SessionTracker dump:
org.apache.zookeeper.server.quorum.LearnerSessionTracker@6d7fd863
ephemeral nodes dump:
Sessions with Ephemerals (2):
0x162183ea9f70003:
/brokers/ids/822712429
0x162183ea9f70002:
/brokers/ids/707577499
/controller
2) stat on /brokers/ids/822712429
zk> stat /brokers/ids/822712429
czxid: 4294967344
mzxid: 4294967344
pzxid: 4294967344
ctime: 1520825436933 (2018-03-11T20:30:36.933-0700)
mtime: 1520825436933 (2018-03-11T20:30:36.933-0700)
version: 0
cversion: 0
aversion: 0
owner: 99668799174148099
datalen: 102
children: 0
3) List full connection/session details for all clients connected
$ echo cons | nc $SERVER1 2181
/10.247.114.70:30401[0](queued=0,recved=1,sent=0)
/10.248.88.235:40430[1](queued=0,recved=345,sent=345,sid=0x162183ea9f70c22,lop=PING,est=1522713395028,to=40000,lcxid=0x12,lzxid=0xffffffffffffffff,lresp=1522717802117,llat=0,minlat=0,avglat=0,maxlat=31)
$ echo cons | nc $SERVER2 2181
/10.196.18.61:28173[0](queued=0,recved=1,sent=0)
/10.247.114.69:42679[1](queued=0,recved=73800,sent=73800,sid=0x262183eaa21da96,lop=PING,est=1522651352906,to=9000,lcxid=0xe49f,lzxid=0x10004683d,lresp=1522717854847,llat=0,minlat=0,avglat=0,maxlat=1235)
4) health
$ echo mntr | nc $SERVER1 2181
zk_version 3.4.6-1569965, built on 02/20/2014 09:09 GMT
zk_avg_latency 0
zk_max_latency 443
zk_min_latency 0
zk_packets_received 11158019
zk_packets_sent 11158244
zk_num_alive_connections 2
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 344
zk_watch_count 0
zk_ephemerals_count 3
zk_approximate_data_size 36654
zk_open_file_descriptor_count 33
zk_max_file_descriptor_count 65536
5) Could not find any special exception from zookeeper logs about the two
sessions
Is this a known bug in version 3.4.6? what could be the potential cause of the
issue?
Thanks,
Daniel
