[
https://issues.apache.org/jira/browse/HBASE-30335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HBASE-30335:
-----------------------------------
Labels: pull-request-available (was: )
> Master's flushedSequenceIdByRegion is not seeded with openSeqNum on region
> OPEN, allowing WAL split to write stale recovered.edits for edits already
> durable in HFiles
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-30335
> URL: https://issues.apache.org/jira/browse/HBASE-30335
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 2.6.1, 2.5.10
> Reporter: Nirdosh Kumar Yadav
> Assignee: Nirdosh Kumar Yadav
> Priority: Minor
> Labels: pull-request-available
>
> h3. Problem Statement
> When a region is drain-moved from RS-A to RS-B and RS-A crashes shortly
> afterwards while its WALs still contain (already-durable) edits for that
> region, the subsequent ServerCrashProcedure WAL split writes a stale
> recovered.edits/<seqId> file under RS-B's already-online region directory.
> The file contains edits that are already in HFiles, but downstream guards
> treat any recovered.edits/<numeric> file as an unreplayed durability risk.
> The next MergeTableRegionsProcedure (or SplitTableRegionProcedure) on that
> region trips AssignmentManagerUtil.checkClosedRegion() with "Recovered.edits
> are found in Region: … abort split/merge to prevent data loss" and leaves the
> region stuck in RIT until an operator intervenes.
> h3. Root Cause
> WALSplitter decides whether to write an edit to recovered.edits/ by comparing
> the edit's sequenceId against the region's lastFlushedSequenceId returned by
> the master (WALSplitter.java:358-360). The master's
> ServerManager.flushedSequenceIdByRegion map is populated only by RegionServer
> heartbeats (ServerManager.java:207-223). During a drain-move:
> * The source RS drops the region from onlineRegions after CLOSE and stops
> heartbeating for it.
> * The destination RS has just OPENed the region and has no new writes to
> flush, so it has nothing to report.
> * No code path refreshes the master's cache from the OPEN transition.
> Consequently, when WALSplitter later asks the master for the region's
> lastFlushedSequenceId, it gets NO_SEQNUM and falls back to -1L at
> WALSplitter.java:358-360:
> {code:java}
> if (lastFlushedSequenceId == null) { lastFlushedSequenceId = -1L; // Treat as
> "never flushed" — write everything. } {code}
> Every edit in the crashed server's WAL for that region is now above the
> fallback watermark and gets written to recovered.edits/, even though the
> region's on-disk .seqid marker already proves those edits are durable.
> The single master-side transition point to State.OPEN is
> AssignmentManager.regionOpenedWithoutPersistingToMeta()
> (AssignmentManager.java:2150), where regionNode.getOpenSeqNum() is already
> available from the RS's OPEN transition report. That value is greater than
> every durable seqId at the previous CLOSE and is exactly the watermark
> WALSplitter needs — but it is never propagated to
> ServerManager.flushedSequenceIdByRegion.
> h3. Key Observations
> * Region 112d9f08181e80aaae01e68c9229c3cb was gracefully drained from rs-132
> → rs-81 at 16:36:34.001 UTC; rs-81's openSeqNum=4997750282. The close on
> rs-132 had already flushed all edits.
> * ~25 seconds later rs-132 was declared dead as part of a broader RS drain.
> * WAL-split worker rs-34 wrote recovered.edits/0000000004997750280 at
> 16:37:12.712 UTC — 38.7 seconds after rs-81 was live on the region. The
> edit's seqId=4997750280 was already durable in HFiles (durable watermark ≥
> 4997750281).
> * The orphan sat dormant for 1h 55m until MergeTableRegionsProcedure
> pid=46990253 tripped checkClosedRegion at 18:32:33.293 UTC and the region
> stayed in RIT for 48m 56s.
> * When rs-120 eventually replayed the file after operator intervention,
> HBase logged Applied 0, skipped 1 — direct proof that the edit was already
> durable and the guard was a false-positive.
> h3. Proposed Solution
> Seed the master's flushed-sequence cache from the OPEN transition. In
> AssignmentManager.java:2150:
> {code:java}
> void regionOpenedWithoutPersistingToMeta(RegionStateNode regionNode) throws
> IOException { regionNode.transitionState(State.OPEN,
> STATES_EXPECTED_ON_OPEN); RegionInfo regionInfo = regionNode.getRegionInfo();
> regionStates.addRegionToServer(regionNode);
> regionStates.removeFromFailedOpen(regionInfo); +
> master.getServerManager().reportRegionOpen(regionInfo,
> regionNode.getOpenSeqNum()); } {code}
> Companion in ServerManager.java (adjacent to updateLastFlushedSequenceIds at
> line 207):
>
> {code:java}
> public void reportRegionOpen(RegionInfo regionInfo, long openSeqNum) { if
> (openSeqNum == HConstants.NO_SEQNUM) return; byte[] encodedRegionName =
> Bytes.toBytes(regionInfo.getEncodedName()); // openSeqNum > every durable
> seqId at CLOSE time — preserves the // "keep the max" invariant that
> updateLastFlushedSequenceIds already uses.
> flushedSequenceIdByRegion.merge(encodedRegionName, openSeqNum, Math::max); }
> {code}
> Effect. At OPEN time the master learns the correct durable watermark
> synchronously with the state transition. When WALSplitter later queries
> ids.getLastFlushedSequenceId(), it gets openSeqNum instead of NO_SEQNUM; the
> skip check openSeqNum >= edit.seqId evaluates true for every already-durable
> edit, and no recovered.edits/* file is written. The 48-min RIT class of
> failure is eliminated at its source — no relaxation of the downstream
> recovered.edits guard is required.
>
> {code:java}
> if (lastFlushedSequenceId == null || lastFlushedSequenceId ==
> HConstants.NO_SEQNUM) { Path regionDir = FSUtils.getRegionDirFromTableDir(
> CommonFSUtils.getTableDir(walRootDir, entry.getKey().getTableName()),
> encodedRegionNameAsStr); long onDiskMaxSeqId =
> WALSplitUtil.getMaxRegionSequenceId(walFS, regionDir); lastFlushedSequenceId
> = (onDiskMaxSeqId != HConstants.NO_SEQNUM) ? onDiskMaxSeqId : -1L; } {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
