[
https://issues.apache.org/jira/browse/ZOOKEEPER-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Wang updated ZOOKEEPER-4785:
-------------------------------
Description:
We had txn loss incident in production recently. The root cause is follower
writes the current epoch and sends the ACK_LD before successfully persisting
all the txns from DIFF sync, as persisting txns is handled asynchronously via
SyncRequestProcessor.
case Leader.NEWLEADER:
...
*self.setCurrentEpoch(newEpoch);*
writeToTxnLog = true;
//Anything after this needs to go to the transaction log, not applied
directly in memory
isPreZAB1_0 = false;
// ZOOKEEPER-3911: make sure sync the uncommitted logs before commit
them (ACK NEWLEADER).
sock.setSoTimeout(self.tickTime * self.syncLimit);
self.setSyncMode(QuorumPeer.SyncMode.NONE);
zk.startupWithoutServing();
if (zk instanceof FollowerZooKeeperServer) {
FollowerZooKeeperServer fzk = (FollowerZooKeeperServer) zk;
for (PacketInFlight p : packetsNotCommitted) {
fzk.logRequest(p.hdr, p.rec, p.digest);
}
packetsNotCommitted.clear();
}
writePacket(new QuorumPacket(Leader.ACK, newLeaderZxid, null, null),
true);
break;
}
was:
We had txn loss incident in production recently. The root cause is follower
writes the current epoch and sends the ACK_LD before successfully persisting
all the txns from DIFF sync, as persisting txns is handled asynchronously via
SyncRequestProcessor.
case Leader.NEWLEADER:
...
*self.setCurrentEpoch(newEpoch);*
writeToTxnLog = true;
//Anything after this needs to go to the transaction log, not applied
directly in memory
isPreZAB1_0 = false;
// ZOOKEEPER-3911: make sure sync the uncommitted logs before commit
them (ACK NEWLEADER).
sock.setSoTimeout(self.tickTime * self.syncLimit);
self.setSyncMode(QuorumPeer.SyncMode.NONE);
zk.startupWithoutServing();
if (zk instanceof FollowerZooKeeperServer) {
FollowerZooKeeperServer fzk = (FollowerZooKeeperServer) zk;
for (PacketInFlight p : packetsNotCommitted) {
* fzk.logRequest(p.hdr, p.rec, p.digest);*
}
packetsNotCommitted.clear();
}
writePacket(new QuorumPacket(Leader.ACK, newLeaderZxid, null, null),
true);
break;
}
> Txn loss due to race condition when follower DIFF sync with leader
> ------------------------------------------------------------------
>
> Key: ZOOKEEPER-4785
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-4785
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.8.0, 3.7.1, 3.8.1, 3.7.2, 3.8.2, 3.9.1
> Reporter: Li Wang
> Priority: Major
>
> We had txn loss incident in production recently. The root cause is follower
> writes the current epoch and sends the ACK_LD before successfully persisting
> all the txns from DIFF sync, as persisting txns is handled asynchronously via
> SyncRequestProcessor.
> case Leader.NEWLEADER:
> ...
> *self.setCurrentEpoch(newEpoch);*
> writeToTxnLog = true;
> //Anything after this needs to go to the transaction log, not applied
> directly in memory
> isPreZAB1_0 = false;
> // ZOOKEEPER-3911: make sure sync the uncommitted logs before commit
> them (ACK NEWLEADER).
> sock.setSoTimeout(self.tickTime * self.syncLimit);
> self.setSyncMode(QuorumPeer.SyncMode.NONE);
> zk.startupWithoutServing();
> if (zk instanceof FollowerZooKeeperServer) {
> FollowerZooKeeperServer fzk = (FollowerZooKeeperServer) zk;
> for (PacketInFlight p : packetsNotCommitted) {
> fzk.logRequest(p.hdr, p.rec, p.digest);
> }
> packetsNotCommitted.clear();
> }
> writePacket(new QuorumPacket(Leader.ACK, newLeaderZxid, null, null),
> true);
> break;
> }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
