[
https://issues.apache.org/jira/browse/ZOOKEEPER-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925746#action_12925746
]
Flavio Junqueira commented on ZOOKEEPER-914:
--------------------------------------------
As Pat, I would also appreciate some more constructive comments (and behavior).
>From the Clover reports, we exercise a significant part of the QCM code, but
>it is true, though, that we don't test the cases you have been exposing. Here
>is a way I believe we can reproduce this problem (I haven't implemented it,
>but seems to make sense). The high-level idea is to make sure that if some
>server stops responding before it completes the handshake protocol, then no
>instance of QCM across all servers will block and prevent other servers from
>joining the ensemble.
Suppose we configure an ensemble with 5 servers using QuorumBase. One of the
servers will be a simple mock server, as we do in the CnxManagerTest tests. Now
here is the sequence of steps to follow:
# Start three of the servers and confirm that they accept and execute
operations;
# Start mock server and execute the protocol partially. For the read case you
mention, you can simply not send the server identifier. That will cause the
read on the other end to block and to not accept more connections;
# Start a 5th server and check if it is able to join the ensemble.
A simple fix to have it working for you soon along the lines of what we have
done to make the connection timeout configurable seems to be to set SO_TIMEOUT.
But, if you have other ideas, please lay them out. Please bear in mind that the
major modifications we should leave for ZOOKEEPER-901 because those will take
more time to develop and get into shape.
> QuorumCnxManager blocks forever
> --------------------------------
>
> Key: ZOOKEEPER-914
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-914
> Project: Zookeeper
> Issue Type: Bug
> Components: leaderElection
> Reporter: Vishal K
> Assignee: Vishal K
> Priority: Blocker
> Fix For: 3.3.3, 3.4.0
>
>
> This was a disaster. While testing our application we ran into a scenario
> where a rebooted follower could not join the cluster. Further debugging
> showed that the follower could not join because the QuorumCnxManager on the
> leader was blocked for indefinite amount of time in receiveConnect()
> "Thread-3" prio=10 tid=0x00007fa920005800 nid=0x11bb runnable
> [0x00007fa9275ed000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.FileDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
> at sun.nio.ch.IOUtil.read(IOUtil.java:206)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> - locked <0x00007fa93315f988> (a java.lang.Object)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:210)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:501)
> I had pointed out this bug along with several other problems in
> QuorumCnxManager earlier in
> https://issues.apache.org/jira/browse/ZOOKEEPER-900 and
> https://issues.apache.org/jira/browse/ZOOKEEPER-822.
> I forgot to patch this one as a part of ZOOKEEPER-822. I am working on a fix
> and a patch will be out soon.
> The problem is that QuorumCnxManager is using SocketChannel in blocking mode.
> It does a read() in receiveConnection() and a write() in initiateConnection().
> Sorry, but this is really bad programming. Also, points out to lack of
> failure tests for QuorumCnxManager.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
