[
https://issues.apache.org/jira/browse/ZOOKEEPER-2959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16469516#comment-16469516
]
Bogdan Kanivets commented on ZOOKEEPER-2959:
--------------------------------------------
I think this is ready to merge. There are 3 PRs for 3.4, 3.5 and master.
Steps to reproduce the bug:
Start with 3 servers. Config:
{code:java}
clientPort=2181
leaderServes=yes
server.1=<server.1-ip>:2888:3888
server.2=<server.2-ip>:2888:3888
server.3=<server.3-ip>:2888:3888:observer
{code}
On server.2 block follower port from server.1 to server.2:
{code:java}
sudo iptables -A INPUT -s <server.1-ip> -p tcp --destination-port 2888 -j
DROP{code}
Start server.1, server.2 and server.3
Wait for server.2 to declare itself a leader and then fail in
waitForNewLeaderAck
{code:java}
2018-04-16 20:56:25,990 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader@361] - LEADING - LEADER
ELECTION TOOK - 3903
2018-04-16 20:56:27,275 [myid:2] - INFO
[LearnerHandler-/<server.3-ip>:29223:LearnerHandler@329] - Follower sid: 3 :
info : org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer@136ca5bc
2018-04-16 20:56:27,281 [myid:2] - INFO
[LearnerHandler-/<server.3-ip>:29223:LearnerHandler@384] - Synchronizing with
Follower sid: 3 maxCommittedLog=0x0 minCommittedLog=0x0 peerLastZxid=0x0
2018-04-16 20:56:27,281 [myid:2] - INFO
[LearnerHandler-/<server.3-ip>:29223:LearnerHandler@393] - leader and follower
are in sync, zxid=0x0
2018-04-16 20:56:27,282 [myid:2] - INFO
[LearnerHandler-/<server.3-ip>:29223:LearnerHandler@458] - Sending DIFF
2018-04-16 20:56:27,291 [myid:2] - INFO
[LearnerHandler-/<server.3-ip>:29223:LearnerHandler@518] - Received
NEWLEADER-ACK message from 3
2018-04-16 20:56:47,283 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader@502] - Shutting down
2018-04-16 20:56:47,284 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader@508] - Shutdown called
java.lang.Exception: shutdown Leader! reason: Waiting for a quorum of
followers, only synced with sids: [ 2 ]
at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:508)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:406)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:859){code}
On server.2 check that currentEpoch is incremented in currentEpoch file. This
is the bug. Epoch is incremented in getEpochToPropose because server.3 is
counted in connectingFollowers.
> ignore accepted epoch and LEADERINFO ack from observers when a newly elected
> leader computes new epoch
> ------------------------------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-2959
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2959
> Project: ZooKeeper
> Issue Type: Bug
> Affects Versions: 3.4.10, 3.5.3
> Reporter: xiangyq000
> Assignee: Bogdan Kanivets
> Priority: Blocker
>
> Once the ZooKeeper cluster finishes the election for new leader, all learners
> report their accepted epoch to the leader for the computation of new cluster
> epoch.
> org.apache.zookeeper.server.quorum.Leader#getEpochToPropose
> {code:java}
> private final HashSet<Long> connectingFollowers = new HashSet<Long>();
> public long getEpochToPropose(long sid, long lastAcceptedEpoch) throws
> InterruptedException, IOException {
> synchronized(connectingFollowers) {
> if (!waitingForNewEpoch) {
> return epoch;
> }
> if (lastAcceptedEpoch >= epoch) {
> epoch = lastAcceptedEpoch+1;
> }
> connectingFollowers.add(sid);
> QuorumVerifier verifier = self.getQuorumVerifier();
> if (connectingFollowers.contains(self.getId()) &&
>
> verifier.containsQuorum(connectingFollowers)) {
> waitingForNewEpoch = false;
> self.setAcceptedEpoch(epoch);
> connectingFollowers.notifyAll();
> } else {
> long start = Time.currentElapsedTime();
> long cur = start;
> long end = start + self.getInitLimit()*self.getTickTime();
> while(waitingForNewEpoch && cur < end) {
> connectingFollowers.wait(end - cur);
> cur = Time.currentElapsedTime();
> }
> if (waitingForNewEpoch) {
> throw new InterruptedException("Timeout while waiting for
> epoch from quorum");
> }
> }
> return epoch;
> }
> }
> {code}
> The computation will get an outcome once :
> # The leader has call method "getEpochToPropose"
> # The number of all reporters is greater than half of participants.
> The problem is, an observer server will also send its accepted epoch to the
> leader, while this procedure treat observers as participants.
> Supposed that the cluster consists of 1 leader, 2 followers and 1 observer,
> and now the leader and the observer have reported their accepted epochs while
> neither of the followers has. Thus, the connectingFollowers set consists of
> two elements, resulting in a size of 2, which is greater than half quorum,
> namely, 2. Then QuorumVerifier#containsQuorum will return true, because it
> does not check whether the elements of the parameter are participants.
> The same flaw exists in
> org.apache.zookeeper.server.quorum.Leader#waitForEpochAck
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
