I'm in IST time zone and causes the delay:-) Have you verified zk cluster by not configuring "sasl" in all these servers and started, just to rule out the possibility of any errors with quorum authentication logic?
Could you give more details: 1) Are you seeing that all Observers(4,5,6) are not able to connect to any of the quorum 1,2,3 servers ? It would be good if you could share zk logs. 2) Hope you have checked that "myid" file is correct in each server - that each server has a distinct server id. 3) Do you have firewall/security and no issues overthere ?. Make sure 2888/3888 are all open. 4) Hope /etc/hosts entries on all the nodes are fine. 5) Have you configured sasl configs in Observer nodes? Rakesh On Wed, Sep 26, 2018 at 9:19 AM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > Any help? > > On Tue, Sep 25, 2018 at 2:20 PM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > >> And observer never joining the cluster its keep saying "Cannot open >> channel to" in the logs. >> >> On Tue, Sep 25, 2018 at 8:25 AM rammohan ganapavarapu < >> rammohanga...@gmail.com> wrote: >> >>> Rakesh, >>> >>> Thank you, i have 3 floower and 3 observers in two different DC's >>> followers came up fine with SASL but for some reasons observers are not >>> coming up with the following error but i dont see any network issues, i was >>> able to telnet to 2181 and 3888 ports. >>> >>> >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue >>> size: 1 >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue >>> size: 1 >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue >>> size: 1 >>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@555] - >>> Opening channel to server 1 >>> 2018-09-24 17:55:34,151 [myid:6] - WARN >>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot >>> open channel to 1 at election address zk-server1/10.16.1.102:3888 >>> java.net.SocketTimeoutException: connect timed out >>> at java.net.PlainSocketImpl.socketConnect(Native Method) >>> at >>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) >>> at >>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) >>> at >>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) >>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >>> at java.net.Socket.connect(Socket.java:589) >>> at >>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558) >>> at >>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610) >>> at >>> org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838) >>> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957) >>> >>> >>> server.1=zk-server1:2888:3888 >>> server.2=zk-server2:2888:3888 >>> server.3=zk-server3:2888:3888 >>> server.4=zk-server4:2888:3888:observer >>> server.5=zk-server5:2888:3888:observer >>> server.6=zk-server6:2888:3888:observer >>> peerType=observer >>> >>> What could be the reason? >>> >>> Ram >>> >>> On Tue, Sep 25, 2018 at 12:12 AM Rakesh Radhakrishnan < >>> rake...@apache.org> wrote: >>> >>>> Thanks Ram for the interest on this feature. >>>> >>>> Yes, user can enable SASL for Observer nodes as well. In general, >>>> QuorumLearner will send authentication packet to peer QuorumServer. >>>> Observer is a learner which follows the same quorum authentication protocol >>>> and auth logic will work fine. >>>> >>>> FYI, hope you are referring below links for configurations, >>>> >>>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication >>>> >>>> https://blog.cloudera.com/blog/2017/01/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/ >>>> >>>> Please let us know if you are facing any issues. >>>> >>>> Thanks, >>>> Rakesh >>>> >>>> On Mon, Sep 24, 2018 at 8:31 AM rammohan ganapavarapu < >>>> rammohanga...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> Do we need to configure any thing on observer nodes for SASL >>>>> authentication? >>>>> >>>>> tcpKeepAlive=true ( this is not for sasl but just asking ) >>>>> >>>>> quorum.auth.enableSasl=true >>>>> quorum.auth.learnerRequireSasl=true >>>>> quorum.auth.serverRequireSasl=true >>>>> >>>>> What will happen if i set these properties on observers nodes as well ? >>>>> >>>>> Thanks, >>>>> Ram >>>>> >>>>
