from the stacktrace is looks like the error is on Netty and Netty is only for the communication port to the client, so the quorum communication should not be affected but this error.
So you have SSL clients and SSL quorum communication ? With portUnification you r using only one single port that serves both plain and ssl protocols so you do not need to start the secureClientPort if you want only TLS then you can disable port unification and configure the secureClientPort I hope that helps Enrico Il giorno lun 4 mag 2020 alle ore 20:44 blb.dev <[email protected]> ha scritto: > So reading through the section on Upgrading existing non-TLS cluster with > no > downtime in the zookeeper admin guide, it says when upgrading from 3.4 > branch I need to start with /sslQuorum=false portUnification=true/. > > I tried this and ZK finally came up successfully. However, when I updated > the config files to /sslQuorum=true portUnification=true/ and did rolling > restart, then they do not pick up the changes giving the same port bind > error as noted. > > *What do I need to do differently to get a healthy zk when sslQuorum > communication is enabled? * > > > > *# cat /logs/zookeeper.out * > 2020-05-04 18:15:06,565 [myid:] - INFO [main:QuorumPeerConfig@173] - > Reading configuration from: /conf/zoo.cfg > 2020-05-04 18:15:06,584 [myid:] - INFO [main:QuorumPeerConfig@459] - > clientPortAddress is 0.0.0.0:2181 > 2020-05-04 18:15:06,584 [myid:] - INFO [main:QuorumPeerConfig@472] - > secureClientPortAddress is 0.0.0.0:2281 > 2020-05-04 18:15:06,588 [myid:] - INFO [main:X509Util@77] - Setting -D > jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated > TLS renegotiation > 2020-05-04 18:15:06,589 [myid:] - INFO [main:QuorumPeerConfig@479] - > observerMasterPort is not set > 2020-05-04 18:15:06,590 [myid:] - INFO [main:QuorumPeerConfig@496] - > metricsProvider.className is > org.apache.zookeeper.metrics.impl.DefaultMetricsProvider > 2020-05-04 18:15:06,605 [myid:1] - INFO [main:DatadirCleanupManager@78] - > autopurge.snapRetainCount set to 10 > 2020-05-04 18:15:06,606 [myid:1] - INFO [main:DatadirCleanupManager@79] - > autopurge.purgeInterval set to 24 > 2020-05-04 18:15:06,610 [myid:1] - INFO [main:ManagedUtil@44] - Log4j 1.2 > jmx support found and enabled. > 2020-05-04 18:15:06,614 [myid:1] - INFO > [PurgeTask:DatadirCleanupManager$PurgeTask@139] - Purge task started. > 2020-05-04 18:15:06,618 [myid:1] - INFO [PurgeTask:FileTxnSnapLog@124] - > zookeeper.snapshot.trust.empty : true > 2020-05-04 18:15:06,629 [myid:1] - INFO [main:QuorumPeerMain@151] - > Starting quorum peer > 2020-05-04 18:15:06,654 [myid:1] - INFO [PurgeTask:SnapStream@61] - > zookeeper.snapshot.compression.method = CHECKED > 2020-05-04 18:15:06,656 [myid:1] - INFO > [PurgeTask:DatadirCleanupManager$PurgeTask@145] - Purge task completed. > 2020-05-04 18:15:06,662 [myid:1] - INFO [main:ServerMetrics@62] - > ServerMetrics initialized with provider > org.apache.zookeeper.metrics.impl.DefaultMetricsProvider@1d251891 > 2020-05-04 18:15:06,780 [myid:1] - INFO [main:NettyServerCnxnFactory@483] > - > zookeeper.client.portUnification=true > 2020-05-04 18:15:06,781 [myid:1] - INFO [main:NettyServerCnxnFactory@495] > - > zookeeper.netty.advancedFlowControl.enabled = false > 2020-05-04 18:15:06,782 [myid:1] - INFO [main:NettyServerCnxnFactory@105] > - > handshakeThrottlingEnabled = false, > zookeeper.netty.server.outstandingHandshake.limit = -1 > 2020-05-04 18:15:06,840 [myid:1] - INFO [main:ServerCnxnFactory@169] - > Using org.apache.zookeeper.server.NettyServerCnxnFactory as server > connection factory > 2020-05-04 18:15:06,841 [myid:1] - WARN [main:ServerCnxnFactory@309] - > maxCnxns is not configured, using default value 0. > 2020-05-04 18:15:06,841 [myid:1] - INFO [main:NettyServerCnxnFactory@483] > - > zookeeper.client.portUnification=true > 2020-05-04 18:15:06,842 [myid:1] - INFO [main:NettyServerCnxnFactory@495] > - > zookeeper.netty.advancedFlowControl.enabled = false > 2020-05-04 18:15:06,842 [myid:1] - INFO [main:NettyServerCnxnFactory@105] > - > handshakeThrottlingEnabled = false, > zookeeper.netty.server.outstandingHandshake.limit = -1 > 2020-05-04 18:15:06,843 [myid:1] - INFO [main:ServerCnxnFactory@169] - > Using org.apache.zookeeper.server.NettyServerCnxnFactory as server > connection factory > 2020-05-04 18:15:06,843 [myid:1] - WARN [main:ServerCnxnFactory@309] - > maxCnxns is not configured, using default value 0. > 2020-05-04 18:15:06,852 [myid:1] - INFO [main:QuorumPeer@752] - > zookeeper.quorumCnxnTimeoutMs=-1 > 2020-05-04 18:15:06,856 [myid:1] - INFO [main:FileTxnSnapLog@124] - > zookeeper.snapshot.trust.empty : true > 2020-05-04 18:15:06,857 [myid:1] - INFO [main:QuorumPeer@1680] - Local > sessions disabled > 2020-05-04 18:15:06,857 [myid:1] - INFO [main:QuorumPeer@1691] - Local > session upgrading disabled > 2020-05-04 18:15:06,857 [myid:1] - INFO [main:QuorumPeer@1658] - tickTime > set to 2000 > 2020-05-04 18:15:06,857 [myid:1] - INFO [main:QuorumPeer@1702] - > minSessionTimeout set to 4000 > 2020-05-04 18:15:06,857 [myid:1] - INFO [main:QuorumPeer@1713] - > maxSessionTimeout set to 40000 > 2020-05-04 18:15:06,857 [myid:1] - INFO [main:QuorumPeer@1738] - > initLimit > set to 10 > 2020-05-04 18:15:06,858 [myid:1] - INFO [main:QuorumPeer@1920] - > syncLimit > set to 5 > 2020-05-04 18:15:06,858 [myid:1] - INFO [main:QuorumPeer@1935] - > connectToLearnerMasterLimit set to 0 > 2020-05-04 18:15:06,877 [myid:1] - INFO [main:ZookeeperBanner@42] - > 2020-05-04 18:15:06,877 [myid:1] - INFO [main:ZookeeperBanner@42] - > ______ _ > 2020-05-04 18:15:06,878 [myid:1] - INFO [main:ZookeeperBanner@42] - > |___ > / | | > 2020-05-04 18:15:06,878 [myid:1] - INFO [main:ZookeeperBanner@42] - > / / > ___ ___ | | __ ___ ___ _ __ ___ _ __ > 2020-05-04 18:15:06,878 [myid:1] - INFO [main:ZookeeperBanner@42] - / > / > / _ \ / _ \ | |/ / / _ \ / _ \ | '_ \ / _ \ | '__| > 2020-05-04 18:15:06,878 [myid:1] - INFO [main:ZookeeperBanner@42] - / > /__ > | (_) | | (_) | | < | __/ | __/ | |_) | | __/ | | > 2020-05-04 18:15:06,878 [myid:1] - INFO [main:ZookeeperBanner@42] - > /_____| \___/ \___/ |_|\_\ \___| \___| | .__/ \___| |_| > 2020-05-04 18:15:06,878 [myid:1] - INFO [main:ZookeeperBanner@42] - > > | | > 2020-05-04 18:15:06,878 [myid:1] - INFO [main:ZookeeperBanner@42] - > > |_| > 2020-05-04 18:15:06,878 [myid:1] - INFO [main:ZookeeperBanner@42] - > 2020-05-04 18:15:06,880 [myid:1] - INFO [main:Environment@98] - Server > > environment:zookeeper.version=3.6.1--104dcb3e3fb464b30c5186d229e00af9f332524b, > built on 04/21/2020 15:01 GMT > 2020-05-04 18:15:06,881 [myid:1] - INFO [main:Environment@98] - Server > environment:host.name=zoo1 > 2020-05-04 18:15:06,881 [myid:1] - INFO [main:Environment@98] - Server > environment:java.version=1.8.0_252 > 2020-05-04 18:15:06,881 [myid:1] - INFO [main:Environment@98] - Server > environment:java.vendor=Oracle Corporation > 2020-05-04 18:15:06,881 [myid:1] - INFO [main:Environment@98] - Server > > environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre > 2020-05-04 18:15:06,881 [myid:1] - INFO [main:Environment@98] - Server > > environment:java.class.path=/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/classes:/apache-zookeeper-3.6.1-bin/bin/../build/classes:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/target/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../build/lib/*.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-prometheus-metrics-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-jute-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/snappy-java-1.1.7.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-log4j12-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/slf4j-api-1.7.25.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_servlet-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_hotspot-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient_common-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/simpleclient-0.6.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-unix-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-native-epoll-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-transport-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-resolver-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-handler-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-common-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-codec-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/netty-buffer-4.1.48.Final.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/metrics-core-3.2.5.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/log4j-1.2.17.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/json-simple-1.1.1.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jline-2.11.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-util-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-servlet-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-server-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-security-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-io-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jetty-http-9.4.24.v20191120.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/javax.servlet-api-3.1.0.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-databind-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-core-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/jackson-annotations-2.10.3.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-lang-2.6.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/commons-cli-1.2.jar:/apache-zookeeper-3.6.1-bin/bin/../lib/audience-annotations-0.5.0.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-3.6.1.jar:/apache-zookeeper-3.6.1-bin/bin/../zookeeper-server/src/main/resources/lib/*.jar:/conf: > 2020-05-04 18:15:06,881 [myid:1] - INFO [main:Environment@98] - Server > > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > 2020-05-04 18:15:06,881 [myid:1] - INFO [main:Environment@98] - Server > environment:java.io.tmpdir=/tmp > 2020-05-04 18:15:06,882 [myid:1] - INFO [main:Environment@98] - Server > environment:java.compiler=<NA> > 2020-05-04 18:15:06,882 [myid:1] - INFO [main:Environment@98] - Server > environment:os.name=Linux > 2020-05-04 18:15:06,882 [myid:1] - INFO [main:Environment@98] - Server > environment:os.arch=amd64 > 2020-05-04 18:15:06,882 [myid:1] - INFO [main:Environment@98] - Server > environment:os.version=3.10.0-327.28.3.el7.x86_64 > 2020-05-04 18:15:06,882 [myid:1] - INFO [main:Environment@98] - Server > environment:user.name=root > 2020-05-04 18:15:06,882 [myid:1] - INFO [main:Environment@98] - Server > environment:user.home=/root > 2020-05-04 18:15:06,882 [myid:1] - INFO [main:Environment@98] - Server > environment:user.dir=/apache-zookeeper-3.6.1-bin > 2020-05-04 18:15:06,883 [myid:1] - INFO [main:Environment@98] - Server > environment:os.memory.free=47MB > 2020-05-04 18:15:06,883 [myid:1] - INFO [main:Environment@98] - Server > environment:os.memory.max=889MB > 2020-05-04 18:15:06,883 [myid:1] - INFO [main:Environment@98] - Server > environment:os.memory.total=57MB > 2020-05-04 18:15:06,883 [myid:1] - INFO [main:ZooKeeperServer@128] - > zookeeper.enableEagerACLCheck = false > 2020-05-04 18:15:06,883 [myid:1] - INFO [main:ZooKeeperServer@136] - > zookeeper.digest.enabled = true > 2020-05-04 18:15:06,883 [myid:1] - INFO [main:ZooKeeperServer@140] - > zookeeper.closeSessionTxn.enabled = true > 2020-05-04 18:15:06,884 [myid:1] - INFO [main:ZooKeeperServer@1434] - > zookeeper.flushDelay=0 > 2020-05-04 18:15:06,884 [myid:1] - INFO [main:ZooKeeperServer@1443] - > zookeeper.maxWriteQueuePollTime=0 > 2020-05-04 18:15:06,884 [myid:1] - INFO [main:ZooKeeperServer@1452] - > zookeeper.maxBatchSize=1000 > 2020-05-04 18:15:06,884 [myid:1] - INFO [main:ZooKeeperServer@241] - > zookeeper.intBufferStartingSizeBytes = 1024 > 2020-05-04 18:15:06,892 [myid:1] - INFO [main:WatchManagerFactory@42] - > Using org.apache.zookeeper.server.watch.WatchManager as watch manager > 2020-05-04 18:15:06,893 [myid:1] - INFO [main:WatchManagerFactory@42] - > Using org.apache.zookeeper.server.watch.WatchManager as watch manager > 2020-05-04 18:15:06,893 [myid:1] - INFO [main:ZKDatabase@132] - > zookeeper.snapshotSizeFactor = 0.33 > 2020-05-04 18:15:06,893 [myid:1] - INFO [main:ZKDatabase@152] - > zookeeper.commitLogCount=500 > 2020-05-04 18:15:06,927 [myid:1] - INFO [main:QuorumPeer@1999] - Using > TLS > encrypted quorum communication > 2020-05-04 18:15:06,927 [myid:1] - INFO [main:QuorumPeer@2007] - Port > unification enabled > 2020-05-04 18:15:06,927 [myid:1] - INFO [main:QuorumPeer@174] - > multiAddress.enabled set to false > 2020-05-04 18:15:06,927 [myid:1] - INFO [main:QuorumPeer@199] - > multiAddress.reachabilityCheckEnabled set to true > 2020-05-04 18:15:06,927 [myid:1] - INFO [main:QuorumPeer@186] - > multiAddress.reachabilityCheckTimeoutMs set to 1000 > 2020-05-04 18:15:06,927 [myid:1] - INFO [main:QuorumPeer@2461] - > QuorumPeer > communication is not secured! (SASL auth disabled) > 2020-05-04 18:15:06,927 [myid:1] - INFO [main:QuorumPeer@2486] - > quorum.cnxn.threads.size set to 20 > 2020-05-04 18:15:06,929 [myid:1] - INFO [main:FileSnap@85] - Reading > snapshot /data/version-2/snapshot.2df500000000 > 2020-05-04 18:15:06,948 [myid:1] - INFO [main:DataTree@1737] - The digest > value is empty in snapshot > 2020-05-04 18:15:06,983 [myid:1] - INFO [main:FileTxnSnapLog@363] - 16 > txns > loaded in 22 ms > 2020-05-04 18:15:06,983 [myid:1] - INFO [main:ZKDatabase@289] - Snapshot > loaded in 55 ms, highest zxid is 0x2df700000008, digest is 354107731533 > 2020-05-04 18:15:06,984 [myid:1] - INFO [main:NettyServerCnxnFactory@670] > - > binding to port 0.0.0.0/0.0.0.0:2181 > 2020-05-04 18:15:07,088 [myid:1] - ERROR [main:QuorumPeerMain@113] - > Unexpected exception, exiting abnormally > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:433) > at sun.nio.ch.Net.bind(Net.java:425) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220) > at > > io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:134) > at > > io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:550) > at > > io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334) > at > > io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506) > at > > io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491) > at > > io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973) > at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:248) > at > io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356) > at > > io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) > at > > io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) > at > > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > at > > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.lang.Thread.run(Thread.java:748) > 2020-05-04 18:15:07,091 [myid:1] - INFO [main:ZKAuditProvider@40] - > ZooKeeper audit is enabled. > 2020-05-04 18:15:07,094 [myid:1] - ERROR [main:ServiceUtils@42] - Exiting > JVM with code 1 > > > > */conf/zoo.cfg:* > > ssl.quorum.trustStore.location=/apache-zookeeper-3.6.1-bin/java/truststore.ks > 4lw.commands.whitelist=* > ssl.quorum.trustStore.password=ApolloStore > clientPort=2181 > ssl.client.enable=true > autopurge.snapRetainCount=10 > dataLogDir=/datalog > snapshot.trust.empty=true > clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > admin.enableServer=false > syncLimit=5 > tickTime=2000 > sslQuorum=true > serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > ssl.quorum.keyStore.location=/apache-zookeeper-3.6.1-bin/java/node1.ks > ssl.quorum.keyStore.password=ApolloCert1 > initLimit=10 > secureClientPort=2281 > maxClientCnxns=0 > client.portUnification=true > reconfigEnabled=true > portUnification=true > dataDir=/data > autopurge.purgeInterval=24 > audit.enable=true > dynamicConfigFile=/conf/zoo.cfg.dynamic.2df700000000 > > > > > -- > Sent from: http://zookeeper-user.578899.n2.nabble.com/ >
