OK, it looks to me some common networking related issue.
1) To confirm, can you remove the Observer type and simply try to join zk
server to quorum like participant?
2) Can you also confirm, hope you don't have "hostname" from the 127.0.0.1
line in /etc/hosts. Something like,
127.0.0.1 node203ea localhost localhost.localdomain localhost4
localhost4.localdomain4
::1 localhost localhost.localdomain localhost6
localhost6.localdomain6
http://ccl.cse.nd.edu/operations/condor/hostname.shtml
On Fri, Sep 28, 2018 at 10:25 PM rammohan ganapavarapu <
rammohanga...@gmail.com> wrote:
> Any thoughts on what could be the reason for observers not able to connect
> to followers/leader?
>
> Ram
>
> On Thu, Sep 27, 2018 at 1:00 PM rammohan ganapavarapu <
> rammohanga...@gmail.com> wrote:
>
>> Incase if you have not received my previous logs files.
>>
>> On Tue, Sep 25, 2018 at 8:25 AM rammohan ganapavarapu <
>> rammohanga...@gmail.com> wrote:
>>
>>> Rakesh,
>>>
>>> Thank you, i have 3 floower and 3 observers in two different DC's
>>> followers came up fine with SASL but for some reasons observers are not
>>> coming up with the following error but i dont see any network issues, i was
>>> able to telnet to 2181 and 3888 ports.
>>>
>>>
>>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
>>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
>>> size: 1
>>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
>>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
>>> size: 1
>>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
>>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@620] - Queue
>>> size: 1
>>> 2018-09-24 17:55:34,145 [myid:6] - DEBUG
>>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@555] -
>>> Opening channel to server 1
>>> 2018-09-24 17:55:34,151 [myid:6] - WARN
>>> [QuorumPeer[myid=6]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@584] - Cannot
>>> open channel to 1 at election address zk-server1/10.16.1.102:3888
>>> java.net.SocketTimeoutException: connect timed out
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at
>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>>> at
>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>>> at
>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>> at java.net.Socket.connect(Socket.java:589)
>>> at
>>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558)
>>> at
>>> org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610)
>>> at
>>> org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838)
>>> at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:957)
>>>
>>>
>>> server.1=zk-server1:2888:3888
>>> server.2=zk-server2:2888:3888
>>> server.3=zk-server3:2888:3888
>>> server.4=zk-server4:2888:3888:observer
>>> server.5=zk-server5:2888:3888:observer
>>> server.6=zk-server6:2888:3888:observer
>>> peerType=observer
>>>
>>> What could be the reason?
>>>
>>> Ram
>>>
>>> On Tue, Sep 25, 2018 at 12:12 AM Rakesh Radhakrishnan <
>>> rake...@apache.org> wrote:
>>>
>>>> Thanks Ram for the interest on this feature.
>>>>
>>>> Yes, user can enable SASL for Observer nodes as well. In general,
>>>> QuorumLearner will send authentication packet to peer QuorumServer.
>>>> Observer is a learner which follows the same quorum authentication protocol
>>>> and auth logic will work fine.
>>>>
>>>> FYI, hope you are referring below links for configurations,
>>>>
>>>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication
>>>>
>>>> https://blog.cloudera.com/blog/2017/01/hardening-apache-zookeeper-security-sasl-quorum-peer-mutual-authentication-and-authorization/
>>>>
>>>> Please let us know if you are facing any issues.
>>>>
>>>> Thanks,
>>>> Rakesh
>>>>
>>>> On Mon, Sep 24, 2018 at 8:31 AM rammohan ganapavarapu <
>>>> rammohanga...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Do we need to configure any thing on observer nodes for SASL
>>>>> authentication?
>>>>>
>>>>> tcpKeepAlive=true ( this is not for sasl but just asking )
>>>>>
>>>>> quorum.auth.enableSasl=true
>>>>> quorum.auth.learnerRequireSasl=true
>>>>> quorum.auth.serverRequireSasl=true
>>>>>
>>>>> What will happen if i set these properties on observers nodes as well ?
>>>>>
>>>>> Thanks,
>>>>> Ram
>>>>>
>>>>
