[
https://issues.apache.org/jira/browse/ZOOKEEPER-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929589#action_12929589
]
Vishal K commented on ZOOKEEPER-914:
------------------------------------
Hi Flavio,
You are right. Sorry, my comment was not fair.
Regarding SO_TIMEOUT: Per my understanding, SO_TIMEOUT works only when a
channel is set in non-blocking mode using isConfigureBlocking(). If the channel
is not configured to work in non-blocking mode, setting SO_TIMEOUT has no
effect. Please let me know if you think there is a way to set timeout on the
socket after accepting the connection (without configuring the channel in
non-blocking mode). The only way I know to use SO_TIMEOUT is by using
channel.isConfigureBlocking(false). The current code in QuorumCnxManager
assumes use of blocking IO. We will have to handle partial reads/writes. Please
refer to my earlier question regarding SO_TIMEOUT for implementing non-blocking
IO.
I thought this fix was supposed to go in for 3.3.3. As I suggested earlier, one
quick fix to the problem is to use TimerTask(). Before doing blocking IO we can
start a timer for that channel (in receiveConnect() before read). Once the
timer expires, check if the read() has finished. If not, interrupt and close
the channel. I think having such a fix (or some other fix that will get around
the problem) until the real fix is in is a better approach. Let me what you
think?
If we decide to go one of the quick fixes, then we can use this JIRA for that
and use ZOOKEEPER-900 for the real fix.. Otherwise, as you suggested, we can
close this JIRA and use ZOOKEEPER-900.
-Vishal
> QuorumCnxManager blocks forever
> --------------------------------
>
> Key: ZOOKEEPER-914
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-914
> Project: Zookeeper
> Issue Type: Bug
> Components: leaderElection
> Reporter: Vishal K
> Assignee: Vishal K
> Priority: Blocker
> Fix For: 3.3.3, 3.4.0
>
>
> This was a disaster. While testing our application we ran into a scenario
> where a rebooted follower could not join the cluster. Further debugging
> showed that the follower could not join because the QuorumCnxManager on the
> leader was blocked for indefinite amount of time in receiveConnect()
> "Thread-3" prio=10 tid=0x00007fa920005800 nid=0x11bb runnable
> [0x00007fa9275ed000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.FileDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
> at sun.nio.ch.IOUtil.read(IOUtil.java:206)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> - locked <0x00007fa93315f988> (a java.lang.Object)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:210)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:501)
> I had pointed out this bug along with several other problems in
> QuorumCnxManager earlier in
> https://issues.apache.org/jira/browse/ZOOKEEPER-900 and
> https://issues.apache.org/jira/browse/ZOOKEEPER-822.
> I forgot to patch this one as a part of ZOOKEEPER-822. I am working on a fix
> and a patch will be out soon.
> The problem is that QuorumCnxManager is using SocketChannel in blocking mode.
> It does a read() in receiveConnection() and a write() in initiateConnection().
> Sorry, but this is really bad programming. Also, points out to lack of
> failure tests for QuorumCnxManager.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
