Hello, I did some more digging after Enrico's last reply (netstat) and it appears I may be miss interpreting what the "stat" command output is telling me. Does the stat command list all connections or only connections from followers/leaders?
Jason Grammenos | Operations and Infrastructure Analyst Pronouns: he/him P: 613.232.7797 x1131 Toll-free: 866.545.3745 x1131 [email protected] agilitypr.com Learn new PR tips from our free resources. -----Original Message----- From: Enrico Olivelli <[email protected]> Sent: February 23, 2022 8:30 AM To: UserZooKeeper <[email protected]> Subject: Re: Help With zookeeper follower connections Are you able to verify, using netstat, if those connections are from the ZooKeeper process? You can compare the TCP port Enrico Il Mer 23 Feb 2022, 14:15 Jason Grammenos <[email protected]> ha scritto: > I do have tools that could be opening connections. I have akhq > (akhq.io) running on two of the hosts [host01, host04 (two nodes for > redundancy reasons)], connecting to the kafka cluster. I also have > telegraph running the jolokia2 agent plugin to pull monitoring/stats > data from the hosts jmx (for both kafka and zookeeper) > > > Jason Grammenos | Operations and Infrastructure Analyst > Pronouns: he/him > P: 613.232.7797 x1131 > Toll-free: 866.545.3745 x1131 > [email protected] > agilitypr.com > Learn new PR tips from our free resources. > > -----Original Message----- > From: Enrico Olivelli <[email protected]> > Sent: February 23, 2022 8:08 AM > To: [email protected] > Cc: UserZooKeeper <[email protected]> > Subject: Re: Help With zookeeper follower connections > > Jason, > Do you have other tools on those machines that could open client > connections? > > Enrico > > Il Mer 23 Feb 2022, 13:30 Jason Grammenos > <[email protected]> ha scritto: > > > Hello, > > > > > > > > I have a 5 node zookeeper (+ kafka) cluster. I am trying to find out > > if the inter node connection behaviour is normal. > > > > I should have 1 zookeeper leader and 4 followers, and I expect that > > all 4 followers would open connections exclusively to the leader. > > > > So that “echo stat | nc localhost 2181” on the leader should show me > > 6 > > connections: 4 from the followers and 2 from itself (one via proper > > ip, one from localhost) > > > > > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > built on 05/04/2020 15:53 GMT > > > > Clients: > > > > /host02:48610[1](queued=0,recved=1455831,sent=1455833) > > > > /host03:36244[1](queued=0,recved=1459769,sent=1459793) > > > > /host05:53680[1](queued=0,recved=484,sent=484) > > > > /host01:48978[1](queued=0,recved=226,sent=226) > > > > /host04:44810[1](queued=0,recved=52,sent=52) > > > > /127.0.0.1:41434[0](queued=0,recved=1,sent=0) > > > > > > > > But instead I have ended up with a few connections on each node, > > including the followers (as seen in below output). Some followers > > are even connecting to themselves? The cluster looks stable, with > > only 1 leader and 4 followers, but I just do not understand this > > connection behaviour and would like to understand if this is normal > > behaviour, mis configuration, a bug or something else. > > > > > > > > user@host01:~$ <jason.grammenos@ca02vleventstream01:~$> echo stat | > > nc localhost 2181 > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > built on 05/04/2020 15:53 GMT > > > > Clients: > > > > /host01:50768[1](queued=0,recved=6156,sent=6156) > > > > /127.0.0.1:35508[0](queued=0,recved=1,sent=0) > > > > > > > > Latency min/avg/max: 0/0/3 > > > > Received: 9855 > > > > Sent: 9854 > > > > Connections: 2 > > > > Outstanding: 0 > > > > Zxid: 0xb00000068 > > > > Mode: follower > > > > Node count: 226 > > > > > > > > user@host02:~$ <jason.grammenos@ca02vleventstream02:~$> echo stat | > > nc localhost 2181 > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > built on 05/04/2020 15:53 GMT > > > > Clients: > > > > /127.0.0.1:49624[0](queued=0,recved=1,sent=0) > > > > > > > > Latency min/avg/max: 0/0/0 > > > > Received: 3699 > > > > Sent: 3698 > > > > Connections: 1 > > > > Outstanding: 0 > > > > Zxid: 0xb00000068 > > > > Mode: follower > > > > Node count: 226 > > > > > > > > usaer@host03:~$ <jason.grammenos@ca02vleventstream03:~$> echo stat | > > nc localhost 2181 > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > built on 05/04/2020 15:53 GMT > > > > Clients: > > > > /host03:50674[1](queued=0,recved=6136,sent=6136) > > > > /127.0.0.1:54554[0](queued=0,recved=1,sent=0) > > > > > > > > Latency min/avg/max: 0/0/22 > > > > Received: 9795 > > > > Sent: 9794 > > > > Connections: 2 > > > > Outstanding: 0 > > > > Zxid: 0xb00000068 > > > > Mode: follower > > > > Node count: 226 > > > > > > > > user@host04:~$ <jason.grammenos@ca04vleventstream04:~$> echo stat | > > nc localhost 2181 > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > built on 05/04/2020 15:53 GMT > > > > Clients: > > > > /host02:42874[1](queued=0,recved=143,sent=143) > > > > /host04:38222[1](queued=0,recved=89,sent=89) > > > > /127.0.0.1:58796[0](queued=0,recved=1,sent=0) > > > > Latency min/avg/max: 0/0/27 > > > > Received: 287 > > > > Sent: 286 > > > > Connections: 3 > > > > Outstanding: 0 > > > > Zxid: 0xb00000056 > > > > Mode: follower > > > > Node count: 226 > > > > > > > > user@host05:~$ <jason.grammenos@ca04vleventstream05:~$> echo stat | > > nc localhost 2181 > > > > Zookeeper version: 3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, > > built on 05/04/2020 15:53 GMT > > > > Clients: > > > > /127.0.0.1:38680[0](queued=0,recved=1,sent=0) > > > > /host01:41198[1](queued=0,recved=6339,sent=6340) > > > > > > > > Latency min/avg/max: 0/0/8 > > > > Received: 10041 > > > > Sent: 10041 > > > > Connections: 2 > > > > Outstanding: 0 > > > > Zxid: 0xb00000068 > > > > Mode: leader > > > > Node count: 226 > > > > Proposal sizes last/min/max: 220/36/392 > > > > > > > > > > > > Thank you > > > > Jason > > > > > > > > Jason Grammenos | Operations and Infrastructure Analyst > > > > Pronouns: he/him > > P: 613.232.7797 x1131 > > Toll-free: 866.545.3745 x1131 > > [email protected] > > agilitypr.com > > > > Learn new PR tips from our free resources > > <https://www.agilitypr.com/resources/>. > > > > > > >
