Please include server logs. Andor
> On 2020. May 2., at 1:36, blb.dev <[email protected]> wrote: > > Hi, during testing upgrade to 3.6.1 version and using secure quorum ssl > communication my zookeeper is not starting up. > > *config file:* > dataDir=/data > dataLogDir=/datalog > tickTime=2000 > initLimit=10 > syncLimit=5 > maxClientCnxns=0 > autopurge.snapRetainCount=10 > autopurge.purgeInterval=24 > admin.enableServer=false > snapshot.trust.empty=true > reconfigEnabled=true > audit.enable=true > clientPort=2181 > secureClientPort=2281 > sslQuorum=true > portUnification=true > serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > client.portUnification=true > clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > ssl.client.enable=true > clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > ssl.quorum.keyStore.location=/apache-zookeeper-3.6.1-bin/java/node1.ks > ssl.quorum.keyStore.password=<pw> > ssl.quorum.trustStore.location=/apache-zookeeper-3.6.1-bin/java/truststore.ks > ssl.quorum.trustStore.password=<pw> > 4lw.commands.whitelist=* > quorumListenOnAllIPs=true > server.2=<fqdn of zoo2>:2888:3888:participant > server.3=<fqdn of zoo2>:2888:3888:participant > server.1=<fqdn of zoo1>:2888:3888:participant > > *# bin/zkServer.sh status* > /usr/bin/java > ZooKeeper JMX enabled by default > Using config: /conf/zoo.cfg > Client port found: 2181. Client address: localhost. > Error contacting service. It is probably not running. > > *# cat /logs/zookeeper_audit.log * > 2020-05-01 22:06:12,213 INFO audit.Log4jAuditLogger: user=zookeeper > operation=serverStart result=success > 2020-05-01 23:08:30,859 INFO audit.Log4jAuditLogger: user=zookeeper > operation=serverStart result=success > > *# bin/zkCli.sh -server 127.0.0.1:2181* > /usr/bin/java > Connecting to 127.0.0.1:2181 > 2020-05-01 23:19:20,035 [myid:] - INFO [main:Environment@98] - Client > environment:zookeeper.version=3.6.1--104dcb3e3fb464b30c5186d229e00af9f332524b, > built on 04/21/2020 15:01 GMT > 2020-05-01 23:19:20,039 [myid:] - INFO [main:Environment@98] - Client > environment:host.name=zoo1 > 2020-05-01 23:19:20,039 [myid:] - INFO [main:Environment@98] - Client > environment:java.version=1.8.0_252 > 2020-05-01 23:19:20,042 [myid:] - INFO [main:Environment@98] - Client > environment:java.vendor=Oracle Corporation > 2020-05-01 23:19:20,042 [myid:] - INFO [main:Environment@98] - Client > environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre > 2020-05-01 23:19:20,042 [myid:] - INFO [main:Environment@98] - Client > environment:java.class.path=/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/classes:/apache-zookeeper-3.6.1-bin/bin/../build/classes:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../build/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-prometheus-metrics-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-jute-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/snappy-java-1.1.7.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-log4j12-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-api-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_servlet-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_hotspot-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_common-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-unix-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-epoll-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-resolver-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-handler-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-codec-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-buffer-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/metrics-core-3.2.5.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/log4j-1.2.17.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/json-simple-1.1.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jline-2.11.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-util-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-servlet-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-server-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-security-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-io-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-http-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-databind-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-core-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-annotations-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-lang-2.6.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-cli-1.2.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/audience-annotations-0.5.0.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/conf: > 2020-05-01 23:19:20,043 [myid:] - INFO [main:Environment@98] - Client > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > 2020-05-01 23:19:20,043 [myid:] - INFO [main:Environment@98] - Client > environment:java.io.tmpdir=/tmp > 2020-05-01 23:19:20,043 [myid:] - INFO [main:Environment@98] - Client > environment:java.compiler=<NA> > 2020-05-01 23:19:20,043 [myid:] - INFO [main:Environment@98] - Client > environment:os.name=Linux > 2020-05-01 23:19:20,043 [myid:] - INFO [main:Environment@98] - Client > environment:os.arch=amd64 > 2020-05-01 23:19:20,043 [myid:] - INFO [main:Environment@98] - Client > environment:os.version=3.10.0-327.28.3.el7.x86_64 > 2020-05-01 23:19:20,043 [myid:] - INFO [main:Environment@98] - Client > environment:user.name=root > 2020-05-01 23:19:20,043 [myid:] - INFO [main:Environment@98] - Client > environment:user.home=/root > 2020-05-01 23:19:20,044 [myid:] - INFO [main:Environment@98] - Client > environment:user.dir=/apache-zookeeper-3.6.1-bin > 2020-05-01 23:19:20,044 [myid:] - INFO [main:Environment@98] - Client > environment:os.memory.free=51MB > 2020-05-01 23:19:20,046 [myid:] - INFO [main:Environment@98] - Client > environment:os.memory.max=228MB > 2020-05-01 23:19:20,046 [myid:] - INFO [main:Environment@98] - Client > environment:os.memory.total=57MB > 2020-05-01 23:19:20,052 [myid:] - INFO [main:ZooKeeper@1005] - Initiating > client connection, connectString=127.0.0.1:2181 sessionTimeout=30000 > watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5e265ba4 > 2020-05-01 23:19:20,059 [myid:] - INFO [main:X509Util@77] - Setting -D > jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated > TLS renegotiation > 2020-05-01 23:19:20,071 [myid:] - INFO [main:ClientCnxnSocket@239] - > jute.maxbuffer value is 1048575 Bytes > 2020-05-01 23:19:20,083 [myid:] - INFO [main:ClientCnxn@1703] - > zookeeper.request.timeout value is 0. feature enabled=false > 2020-05-01 23:19:20,091 [myid:127.0.0.1:2181] - INFO > [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1154] - Opening > socket connection to server localhost/127.0.0.1:2181. > 2020-05-01 23:19:20,091 [myid:127.0.0.1:2181] - INFO > [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1156] - SASL config > status: Will not attempt to authenticate using SASL (unknown error) > Welcome to ZooKeeper! > JLine support is enabled > 2020-05-01 23:19:20,235 [myid:127.0.0.1:2181] - INFO > [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@986] - Socket > connection established, initiating session, client: /127.0.0.1:37360, > server: localhost/127.0.0.1:2181 > 2020-05-01 23:19:20,259 [myid:127.0.0.1:2181] - WARN > [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1272] - Session 0x0 > for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting > reconnect except it is a SessionExpiredException. > EndOfStreamException: Unable to read additional data from server sessionid > 0x0, likely server has closed socket > at > org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:348) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1262) > [zk: 127.0.0.1:2181(CONNECTING) 0] 2020-05-01 23:19:21,591 > [myid:127.0.0.1:2181] - INFO > [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1154] - Opening > socket connection to server localhost/127.0.0.1:2181. > 2020-05-01 23:19:21,591 [myid:127.0.0.1:2181] - INFO > [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1156] - SASL config > status: Will not attempt to authenticate using SASL (unknown error) > 2020-05-01 23:19:21,592 [myid:127.0.0.1:2181] - INFO > [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@986] - Socket > connection established, initiating session, client: /127.0.0.1:37362, > server: localhost/127.0.0.1:2181 > 2020-05-01 23:19:21,598 [myid:127.0.0.1:2181] - WARN > [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1272] - Session 0x0 > for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting > reconnect except it is a SessionExpiredException. > EndOfStreamException: Unable to read additional data from server sessionid > 0x0, likely server has closed socket > at > org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:348) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1262) > 2020-05-01 23:19:23,303 [myid:127.0.0.1:2181] - INFO > [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1154] - Opening > socket connection to server localhost/127.0.0.1:2181. > 2020-05-01 23:19:23,304 [myid:127.0.0.1:2181] - INFO > [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1156] - SASL config > status: Will not attempt to authenticate using SASL (unknown error) > 2020-05-01 23:19:23,304 [myid:127.0.0.1:2181] - INFO > [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@986] - Socket > connection established, initiating session, client: /127.0.0.1:37364, > server: localhost/127.0.0.1:2181 > 2020-05-01 23:19:23,313 [myid:127.0.0.1:2181] - WARN > [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1272] - Session 0x0 > for sever localhost/127.0.0.1:2181, Closing socket connection. Attempting > reconnect except it is a SessionExpiredException. > EndOfStreamException: Unable to read additional data from server sessionid > 0x0, likely server has closed socket > at > org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:348) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1262) > > > (it continues on like this repeating the same exception) > > Don't see /logs/zookeeper.out > > *# nc -vz zoo2 2181* > Ncat: Version 7.50 ( https://nmap.org/ncat ) > Ncat: Connected to <fqdn zoo2>:2181. > Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds. > *# nc -vz zoo3 2181* > Ncat: Version 7.50 ( https://nmap.org/ncat ) > Ncat: Connected to <fqdn zoo3>:2181. > Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds. > > *# nc -vz zoo2 2182* > Ncat: Version 7.50 ( https://nmap.org/ncat ) > Ncat: Connection refused. > *# nc -vz zoo3 2182* > Ncat: Version 7.50 ( https://nmap.org/ncat ) > Ncat: Connection refused. > > *# ps -ef | grep zookeeper* > root 13 1 0 23:08 ? 00:00:00 su -m zookeeper zkServer.sh > start-foreground > zookeep+ 14 13 0 23:08 ? 00:00:03 java > -Dzookeeper.log.dir=/logs -Dzookeeper.log.file=zookeeper--server-zoo1.log > -Dzookeeper.root.logger=DEBUG -XX:+HeapDumpOnOutOfMemoryError > -XX:OnOutOfMemoryError=kill -9 %p -cp > /apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/classes:/apache-zookeeper-3.6.1-bin/bin/../build/classes:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../build/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-prometheus-metrics-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-jute-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/snappy-java-1.1.7.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-log4j12-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-api-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_servlet-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_hotspot-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_common-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-unix-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-epoll-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-resolver-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-handler-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-codec-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-buffer-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/metrics-core-3.2.5.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/log4j-1.2.17.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/json-simple-1.1.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jline-2.11.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-util-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-servlet-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-server-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-security-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-io-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-http-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-databind-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-core-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-annotations-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-lang-2.6.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-cli-1.2.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/audience-annotations-0.5.0.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/conf: > -Xmx1000m -Xmx4g -Xms1g -Dcom.sun.management.jmxremote > -Dcom.sun.management.jmxremote.local.only=false > org.apache.zookeeper.server.quorum.QuorumPeerMain /conf/zoo.cfg > > I think the issue is around the secureClientPort not taking the connection? > What else needs to be configured to work or would it be an issue between my > machines? At a loss for how to proceed. Any debug or configuration guidance > is appreciated. > > > > > -- > Sent from: http://zookeeper-user.578899.n2.nabble.com/
