ZhenyuLi created ZOOKEEPER-5094:
-----------------------------------
Summary: Interrupted TRUNC synchronization can leave follower
memory ahead of its transaction log
Key: ZOOKEEPER-5094
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5094
Project: ZooKeeper
Issue Type: Bug
Components: quorum
Affects Versions: 3.9.6
Reporter: ZhenyuLi
h3. Summary
A follower interrupted during TRUNC synchronization can retain an in-memory
database that is ahead of its transaction log.
During TRUNC synchronization, committed transactions received before
\{{NEWLEADER}} can be applied directly to memory. They are persisted later when
the follower takes a snapshot after receiving \{{NEWLEADER}}.
If the connection fails before \{{NEWLEADER}}, the follower can be left with:
{noformat}
in-memory lastProcessedZxid = Z2
on-disk transaction log = Z1
{noformat}
The next synchronization reports Z2 to the leader even though Z2 was never
persisted. The leader may then continue from Z3, leaving a permanent gap in the
follower's transaction log:
{noformat}
Z1 -> Z3
{noformat}
After restart, replaying this log throws \{{IllegalStateException}}, preventing
the follower from starting.
h3. Root cause
For TRUNC, \{{syncWithLeader()}} expects to persist the synchronized state in a
snapshot:
{code:java}
// For SNAP and TRUNC the snapshot is needed to save that history
boolean snapshotNeeded = true;
{code}
Before that snapshot is taken, a committed transaction is applied directly to
memory:
{code:java}
if (!writeToTxnLog) {
zk.processTxn(pif.toRequest());
}
{code}
The snapshot is written when \{{NEWLEADER}} arrives:
{code:java}
if (snapshotNeeded) {
zk.takeSnapshot(syncSnapshot);
}
{code}
However, if the connection fails before \{{NEWLEADER}}, \{{Learner.shutdown()}}
only discards the database for SNAP synchronization:
{code:java}
zk.shutdown(
self.getSyncMode().equals(QuorumPeer.SyncMode.SNAP)
);
{code}
An incomplete TRUNC synchronization therefore keeps the modified in-memory
database even though its changes were not persisted.
h3. Impact
On the next connection, \{{getLastLoggedZxid()}} reads the retained in-memory
database and reports Z2 instead of the durable Z1.
The leader assumes the follower already has Z2 and sends later transactions
starting with Z3. Once Z3 is written to disk, the transaction log contains:
{noformat}
Z1 -> Z3
{noformat}
After the follower process restarts, \{{ZKDatabase.addCommittedProposal()}}
detects the gap and throws:
{noformat}
IllegalStateException: Committed proposal cached out of order:
Z3 is not the next proposal of Z1
{noformat}
The exception occurs while loading the database, before the participant can
enter leader election or request a new snapshot. Repeated restarts fail until
its local data is repaired or cleared and resynchronized.
h3. Sequence
{noformat}
Leader Follower memory Follower disk
TRUNC(Z1)
-----------------------------> Z1 Z1
PROPOSAL(Z2)
COMMIT(Z2)
-----------------------------> Z1 -> Z2 Z1
connection fails before NEWLEADER
incomplete state retained
follower reconnects
<--------------------- reports Z2
PROPOSAL(Z3)
COMMIT(Z3)
-----------------------------> Z1 -> Z2 -> Z3 Z1 -> Z3
follower restarts
replay Z1, then Z3
IllegalStateException
startup fails
{noformat}
h3. Steps to reproduce
# Start a ZooKeeper quorum with three voting participants.
# Make one follower reconnect using TRUNC synchronization.
# Send \{{TRUNC(Z1)}}, followed by \{{PROPOSAL(Z2)}} and \{{COMMIT(Z2)}}.
# Interrupt the quorum connection before \{{NEWLEADER}}.
# Let the follower reconnect in the same JVM.
# The follower reports the in-memory Z2 even though its transaction log ends at
Z1.
# Commit Z3 and allow the follower to persist it.
# Restart the follower.
h3. Expected result
The follower discards its in-memory database when TRUNC synchronization is
interrupted. On reconnect, it reloads Z1 from disk and receives Z2 and Z3 in
order.
h3. Actual result
The follower retains the non-durable Z2 in memory, later persists Z3 after Z1,
and creates a transaction-log gap. On restart, database loading throws
\{{IllegalStateException}} and the follower cannot start.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
