Brian Thanks for sharing your story. It is very helpful to the community
Enrico Il Mar 9 Feb 2021, 00:51 Brian Rickabaugh <[email protected]> ha scritto: > It looks like there were a couple of things going on here: > > 1.) After a little more digging, I found that the leader of the ensemble > was listening on port 2888. Does only the leader of a ZooKeeper ensemble > listen on 2888?This seems to be the case but I didn't see it in the > documentation somewhere. I may have missed it. > > I found that only one of my servers, the leader, was listening on port > 2888. When I shut it down, another server in the ensemble started > listening on port 2888. > > 2.) While I did check that iptables was clear off all rules, it appears > that firewalld did have some rules that were blocking some of the > connectivity. When I added the appropriate rules for 2888/tcp and > 3888/tcp, the ensemble is able to communicate as expected. > > I ran in to a red herring while diagnosing the issue. I wanted to mention > it here for the record. I was receiving a > "java.net.NoRouteToHostException: No route to host" exception before > shutting down the firewall. This was a little misleading as I knew there > was a route between the VMs, they're on the same physical host and I could > SSH from one to the other. Tracing the route with traceroute proved it. > The ncat command yielded the same message, no route, when trying to connect > to these ports. It was this fact that pointed me to something going on > with a firewall somewhere. > > At any rate, problem solved. Thanks for your help, Enrico. > > Brian > > Quoting Enrico Olivelli <[email protected]>: > > > With a "grep 2888" you do not see any reference to 2888 ? > > > > Can you try with JDK11 ? > > I am sure that ZK works with JDK15 (I have used it in production) and > > JDK16-ea > > > > Enrico > > > > Il giorno lun 8 feb 2021 alle ore 07:44 Brian @ Rickabaugh.Net < > > [email protected]> ha scritto: > > > >> I am using 3 different virtual machines. One ZooKeeper server on each > >> VM. None of the hosts are listening on port 2888 and they each report > >> the > >> same error when trying to contact each other. > >> > >> I have placed the id file on each of the servers in the data directory. > >> They each appear to know their individual ids. > >> > >> I disabled SELinux and the local firewall on each of the hosts and the > >> issue remains. > >> > >> Brian > >> > >> On Feb 8, 2021, at 1:04 AM, Enrico Olivelli <[email protected]> > wrote: > >> > >> Brian > >> > >> Il Lun 8 Feb 2021, 03:50 Brian Rickabaugh <[email protected]> > ha > >> scritto: > >> > >> Hello, everyone. > >> > >> I am working on configuring a new set of 3 ZooKeeper servers and I am > >> finding that none of the servers are listening on port 2888. There are > >> no > >> relevant messages or exceptions logged during start up. > >> > >> The servers are listening on port 3888. The servers do appear to start > >> up > >> properly and connect to each other on port 3888. > >> > >> After a few seconds, I start seeing exceptions logged along the lines of > >> the following: > >> > >> 2021-02-07 21:30:16,583 [myid:1] - WARN > >> > >> > > [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Follower@129 > >> ] > >> - Exception when following the leader > >> java.io.IOException: Failed connect to box02/10.1.1.202:2888 > >> at > >> > >> > > org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:298) > >> at > >> > >> > org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:87) > >> at > >> org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1458) > >> > >> I am using ZooKeeper 3.6.2, Fedora 33 and the following Java runtime. > >> > >> Are you using 3 virtual machines/servers or are you running everything > in > >> the same machine? > >> > >> Did you write the 'id' file in the data directory for each server? > >> > >> Enrico > >> > >> openjdk version "15.0.1" 2020-10-20 > >> OpenJDK Runtime Environment 20.9 (build 15.0.1+9) > >> OpenJDK 64-Bit Server VM 20.9 (build 15.0.1+9, mixed mode, sharing) > >> > >> My ZooKeeper conf file is as follows: > >> > >> ----- > >> tickTime=2000 > >> dataDir=/opt/zookeeper-data/ > >> clientPort=2181 > >> initLimit=5 > >> syncLimit=2 > >> server.1=box00:2888:3888 > >> server.2=box01:2888:3888 > >> server.3=box02:2888:3888 > >> ----- > >> > >> Any thoughts or ideas? > >> > >> Thanks in advance, Brian > >> > > > > >
