Are you able to verify, using netstat, if those connections are from the ZooKeeper process? You can compare the TCP port
Enrico Il Mer 23 Feb 2022, 14:15 Jason Grammenos <[email protected]> ha scritto: > I do have tools that could be opening connections. I have akhq (akhq.io) > running on two of the hosts [host01, host04 (two nodes for redundancy > reasons)], connecting to the kafka cluster. I also have telegraph running > the jolokia2 agent plugin to pull monitoring/stats data from the hosts jmx > (for both kafka and zookeeper) > > > Jason Grammenos | Operations and Infrastructure Analyst > Pronouns: he/him > P: 613.232.7797 x1131 > Toll-free: 866.545.3745 x1131 > [email protected] > agilitypr.com > Learn new PR tips from our free resources. > > -----Original Message----- > From: Enrico Olivelli <[email protected]> > Sent: February 23, 2022 8:08 AM > To: [email protected] > Cc: UserZooKeeper <[email protected]> > Subject: Re: Help With zookeeper follower connections > > Jason, > Do you have other tools on those machines that could open client > connections? > > Enrico > > Il Mer 23 Feb 2022, 13:30 Jason Grammenos > <[email protected]> ha scritto: > > > Hello, > > > > > > > > I have a 5 node zookeeper (+ kafka) cluster. I am trying to find out > > if the inter node connection behaviour is normal. > > > > I should have 1 zookeeper leader and 4 followers, and I expect that > > all 4 followers would open connections exclusively to the leader. > > > > So that “echo stat | nc localhost 2181” on the leader should show me 6 > > connections: 4 from the followers and 2 from itself (one via proper > > ip, one from localhost) > > > > > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > built on 05/04/2020 15:53 GMT > > > > Clients: > > > > /host02:48610[1](queued=0,recved=1455831,sent=1455833) > > > > /host03:36244[1](queued=0,recved=1459769,sent=1459793) > > > > /host05:53680[1](queued=0,recved=484,sent=484) > > > > /host01:48978[1](queued=0,recved=226,sent=226) > > > > /host04:44810[1](queued=0,recved=52,sent=52) > > > > /127.0.0.1:41434[0](queued=0,recved=1,sent=0) > > > > > > > > But instead I have ended up with a few connections on each node, > > including the followers (as seen in below output). Some followers are > > even connecting to themselves? The cluster looks stable, with only 1 > > leader and 4 followers, but I just do not understand this connection > > behaviour and would like to understand if this is normal behaviour, > > mis configuration, a bug or something else. > > > > > > > > user@host01:~$ <jason.grammenos@ca02vleventstream01:~$> echo stat | nc > > localhost 2181 > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > built on 05/04/2020 15:53 GMT > > > > Clients: > > > > /host01:50768[1](queued=0,recved=6156,sent=6156) > > > > /127.0.0.1:35508[0](queued=0,recved=1,sent=0) > > > > > > > > Latency min/avg/max: 0/0/3 > > > > Received: 9855 > > > > Sent: 9854 > > > > Connections: 2 > > > > Outstanding: 0 > > > > Zxid: 0xb00000068 > > > > Mode: follower > > > > Node count: 226 > > > > > > > > user@host02:~$ <jason.grammenos@ca02vleventstream02:~$> echo stat | nc > > localhost 2181 > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > built on 05/04/2020 15:53 GMT > > > > Clients: > > > > /127.0.0.1:49624[0](queued=0,recved=1,sent=0) > > > > > > > > Latency min/avg/max: 0/0/0 > > > > Received: 3699 > > > > Sent: 3698 > > > > Connections: 1 > > > > Outstanding: 0 > > > > Zxid: 0xb00000068 > > > > Mode: follower > > > > Node count: 226 > > > > > > > > usaer@host03:~$ <jason.grammenos@ca02vleventstream03:~$> echo stat | > > nc localhost 2181 > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > built on 05/04/2020 15:53 GMT > > > > Clients: > > > > /host03:50674[1](queued=0,recved=6136,sent=6136) > > > > /127.0.0.1:54554[0](queued=0,recved=1,sent=0) > > > > > > > > Latency min/avg/max: 0/0/22 > > > > Received: 9795 > > > > Sent: 9794 > > > > Connections: 2 > > > > Outstanding: 0 > > > > Zxid: 0xb00000068 > > > > Mode: follower > > > > Node count: 226 > > > > > > > > user@host04:~$ <jason.grammenos@ca04vleventstream04:~$> echo stat | nc > > localhost 2181 > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > built on 05/04/2020 15:53 GMT > > > > Clients: > > > > /host02:42874[1](queued=0,recved=143,sent=143) > > > > /host04:38222[1](queued=0,recved=89,sent=89) > > > > /127.0.0.1:58796[0](queued=0,recved=1,sent=0) > > > > Latency min/avg/max: 0/0/27 > > > > Received: 287 > > > > Sent: 286 > > > > Connections: 3 > > > > Outstanding: 0 > > > > Zxid: 0xb00000056 > > > > Mode: follower > > > > Node count: 226 > > > > > > > > user@host05:~$ <jason.grammenos@ca04vleventstream05:~$> echo stat | nc > > localhost 2181 > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > built on 05/04/2020 15:53 GMT > > > > Clients: > > > > /127.0.0.1:38680[0](queued=0,recved=1,sent=0) > > > > /host01:41198[1](queued=0,recved=6339,sent=6340) > > > > > > > > Latency min/avg/max: 0/0/8 > > > > Received: 10041 > > > > Sent: 10041 > > > > Connections: 2 > > > > Outstanding: 0 > > > > Zxid: 0xb00000068 > > > > Mode: leader > > > > Node count: 226 > > > > Proposal sizes last/min/max: 220/36/392 > > > > > > > > > > > > Thank you > > > > Jason > > > > > > > > Jason Grammenos | Operations and Infrastructure Analyst > > > > Pronouns: he/him > > P: 613.232.7797 x1131 > > Toll-free: 866.545.3745 x1131 > > [email protected] > > agilitypr.com > > > > Learn new PR tips from our free resources > > <https://www.agilitypr.com/resources/>. > > > > > > >
