Hi, We are using consul for our external service DNS resolution and the zookeeper is running in a Docker containerized environment. We can not use a hard IP address in our server config due to containerization. We are migrating from 3.4.12 to 3.5.5 and I had to introduce the flag (which is not needed in 3.6.1 thanks to the fix made by Mate Szalay)
>From the code, the zookeeper requires IP and port to connect for the socket connection. We used to have 0.0.0.0 set in our zoo.cfg which was supported by 3.4.12 (in general 3.4.x). Since 3.5, when the zookeeper server starts for the very first time, it sends out its "Initial Identification" to its peers. The ZK server sends out its sid and its address as its identification. Our zk server list is of the form server.1=zk.service:2888:3888;2181 The hostname zk.service is not understood by the zookeeper as it requires InetAddress (as its all internal communication, our external consul DNS resolution doesn't come into the picture). What this leads to is that the originating node is never able to complete its leader election as it cannot bind to the address, to begin with, and therefore is never able to join the cluster. Now ** quorumListenOnAllIPs** enables the 0.0.0.0 interface as it allows the zookeeper to bind on the address 0.0.0.0:3888 and also talk to its peers during leader election. It will receive the connections from all the IP's and quorum will be formed. You don't need to add this if you have a static IP config or using 3.6+ (or now 3.5.8 as well as it is also released) On Thu, May 7, 2020 at 6:30 PM blb.dev <[email protected]> wrote: > Thanks for sharing - > > I wasn't using quorumListenOnAllIPs=true, as it's listed as an Unsafe > Option > - any reason why you chose to use it? > > The bind issue is fixed by setting either clientPort or secureClientPort > only and only setting it in zoo.cfg or zoo.cfg.dynamic. > > However, my quorum still does not start up! It gives the following stack > trace: > > # bin/zkCli.sh > /usr/bin/java > Connecting to localhost:2181 > 2020-05-08 01:25:54,277 [myid:] - INFO [main:Environment@98] - Client > > environment:zookeeper.version=3.6.1--104dcb3e3fb464b30c5186d229e00af9f332524b, > built on 04/21/2020 15:01 GMT > 2020-05-08 01:25:54,281 [myid:] - INFO [main:Environment@98] - Client > environment:host.name=zoo2 > 2020-05-08 01:25:54,281 [myid:] - INFO [main:Environment@98] - Client > environment:java.version=1.8.0_252 > 2020-05-08 01:25:54,284 [myid:] - INFO [main:Environment@98] - Client > environment:java.vendor=Oracle Corporation > 2020-05-08 01:25:54,284 [myid:] - INFO [main:Environment@98] - Client > > environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre > 2020-05-08 01:25:54,284 [myid:] - INFO [main:Environment@98] - Client > > environment:java.class.path=/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/classes:/apache-zookeeper-3.6.1-bin/bin/../build/classes:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../build/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-prometheus-metrics-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-jute-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/snappy-java-1.1.7.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-log4j12-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-api-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_servlet-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_hotspot-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_common-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-unix-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-epoll-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-resolver-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-handler-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-codec-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-buffer-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/metrics-core-3.2.5.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/log4j-1.2.17.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/json-simple-1.1.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jline-2.11.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-util-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-servlet-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-server-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-security-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-io-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-http-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-databind-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-core-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-annotations-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-lang-2.6.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-cli-1.2.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/audience-annotations-0.5.0.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/conf: > 2020-05-08 01:25:54,284 [myid:] - INFO [main:Environment@98] - Client > > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > 2020-05-08 01:25:54,285 [myid:] - INFO [main:Environment@98] - Client > environment:java.io.tmpdir=/tmp > 2020-05-08 01:25:54,285 [myid:] - INFO [main:Environment@98] - Client > environment:java.compiler=<NA> > 2020-05-08 01:25:54,285 [myid:] - INFO [main:Environment@98] - Client > environment:os.name=Linux > 2020-05-08 01:25:54,285 [myid:] - INFO [main:Environment@98] - Client > environment:os.arch=amd64 > 2020-05-08 01:25:54,285 [myid:] - INFO [main:Environment@98] - Client > environment:os.version=3.10.0-327.28.3.el7.x86_64 > 2020-05-08 01:25:54,285 [myid:] - INFO [main:Environment@98] - Client > environment:user.name=root > 2020-05-08 01:25:54,285 [myid:] - INFO [main:Environment@98] - Client > environment:user.home=/root > 2020-05-08 01:25:54,285 [myid:] - INFO [main:Environment@98] - Client > environment:user.dir=/apache-zookeeper-3.6.1-bin > 2020-05-08 01:25:54,286 [myid:] - INFO [main:Environment@98] - Client > environment:os.memory.free=51MB > 2020-05-08 01:25:54,287 [myid:] - INFO [main:Environment@98] - Client > environment:os.memory.max=228MB > 2020-05-08 01:25:54,287 [myid:] - INFO [main:Environment@98] - Client > environment:os.memory.total=57MB > 2020-05-08 01:25:54,293 [myid:] - INFO [main:ZooKeeper@1005] - Initiating > client connection, connectString=localhost:2181 sessionTimeout=30000 > watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5e265ba4 > 2020-05-08 01:25:54,298 [myid:] - INFO [main:X509Util@77] - Setting -D > jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated > TLS renegotiation > 2020-05-08 01:25:54,306 [myid:] - INFO [main:ClientCnxnSocket@239] - > jute.maxbuffer value is 1048575 Bytes > 2020-05-08 01:25:54,318 [myid:] - INFO [main:ClientCnxn@1703] - > zookeeper.request.timeout value is 0. feature enabled=false > Welcome to ZooKeeper! > 2020-05-08 01:25:54,331 [myid:localhost:2181] - INFO > [main-SendThread(localhost:2181):ClientCnxn$SendThread@1154] - Opening > socket connection to server localhost/127.0.0.1:2181. > 2020-05-08 01:25:54,333 [myid:localhost:2181] - INFO > [main-SendThread(localhost:2181):ClientCnxn$SendThread@1156] - SASL config > status: Will not attempt to authenticate using SASL (unknown error) > JLine support is enabled > 2020-05-08 01:25:54,449 [myid:localhost:2181] - INFO > [main-SendThread(localhost:2181):ClientCnxn$SendThread@986] - Socket > connection established, initiating session, client: /127.0.0.1:49708, > server: localhost/127.0.0.1:2181 > 2020-05-08 01:25:54,474 [myid:localhost:2181] - WARN > [main-SendThread(localhost:2181):ClientCnxn$SendThread@1272] - Session 0x0 > for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting > reconnect except it is a SessionExpiredException. > EndOfStreamException: Unable to read additional data from server sessionid > 0x0, likely server has closed socket > at > org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75) > at > > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:348) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1262) > > I tried adding multiAddress.reachabilityCheckEnabled=true since that seemed > to paid with quorumListenOnAllIPs=true, but still the same SessionExpired > error. > > Current config: > > dataDir=/data > dataLogDir=/datalog > tickTime=2000 > initLimit=10 > syncLimit=5 > maxClientCnxns=0 > autopurge.snapRetainCount=10 > autopurge.purgeInterval=24 > standaloneEnabled=false > admin.enableServer=false > snapshot.trust.empty=true > reconfigEnabled=true > audit.enable=true > clientPort=2181 > sslQuorum=true > ssl.client.enable=true > portUnification=true > client.portUnification=true > serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > ssl.quorum.keyStore.location= > ssl.quorum.keyStore.password= > ssl.quorum.trustStore.location= > ssl.quorum.trustStore.password= > multiAddress.reachabilityCheckEnabled=true > quorumListenOnAllIPs=true > 4lw.commands.whitelist=* > dynamicConfigFile=/conf/zoo.cfg.dynamic > > where zoo.cfg.dynamic is: > > server.2=zoo2:2888:3888:participant > server.3=zoo3:2888:3888:participant > server.1=zoo1:2888:3888:participant > > > Any ideas what this issue is? I'm really at a loss trying to add SSL... > > > > > -- > Sent from: http://zookeeper-user.578899.n2.nabble.com/ >
