Aleksander Nek created HDFS-18023:
-------------------------------------
Summary: Async edit tailing can save rolling-upgrade rollback
images inside an edit-log segment
Key: HDFS-18023
URL: https://issues.apache.org/jira/browse/HDFS-18023
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 3.6.0
Reporter: Aleksander Nek
h2. Problem
With dfs.ha.tail-edits.in-progress=true, standby replay can advance across the
segment boundary rolled by rollingUpgrade(PREPARE), or yield before
END_LOG_SEGMENT when the replay batch is small. StandbyCheckpointer then saves
fsimage_rollback at a TXID inside a segment. Rollback subsequently calls
discardSegments(imageTxId+1), which requires a segment boundary and can fail.
h2. Reproduction
# Start two HA NameNodes and three JournalNodes with RPC edit tailing enabled.
# Write a namespace entry, then invoke rollingUpgrade(PREPARE).
# Wait for rollback images and compare the image TXID with finalized journal
segment ends.
# Repeat with dfs.ha.tail-edits.max-txns-per-lock=1.
Observed before the fix: finalized segment ends at TXID 4, but the rollback
image is saved at 5 (ordinary batch) or 3 (single-transaction batch). None of
the three journals has a matching finalized boundary. The old-only baseline
reproducer fails discard starting at 6 for segment [5,7].
h2. Proposed fix and validation
Periodic standby replay stops immediately after applying the first
END_LOG_SEGMENT after PREPARE, and preserves that exact namespace until the
local rollback image exists. The checkpointer defers a rollback image until
this boundary is ready. Failover catch-up must bypass the pause; observers and
non-checkpointing standbys must continue replaying. Startup replay remains
unrestricted.
Both before-fix regressions fail with zero of three aligned journals. A focused
baseline overlay containing only this fix passes both batch sizes, real
rollback and subsequent restart, without Multi-DC code or the separate
JournalNode cache fix. The standalone rollback test uses finalized streaming
during rollback to isolate the two defects; PREPARE itself remains
asynchronous. On current trunk all six focused tests pass: both replay batch
sizes with real rollback and restart, failover and restart while the rollback
image is pending, Observer replay and disabled-checkpointer replay. Existing
HA/checkpoint regression suites are also being run before the PR is published.
The defect is present in the baseline e66f40daf3cc4ebbb5164cec1ce54b71eddec8a4
and the corresponding source paths on current trunk
c815ec8e1e98f20e3f0ae2637665d4adb9196660. No new configuration, wire or image
format is needed. This report concerns HA checkpoint boundary selection, rather
than the non-HA replay retry in HDFS-17932.
Contains content generated by OpenAI Codex.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]