nirdosh0110 commented on PR #8584:
URL: https://github.com/apache/hbase/pull/8584#issuecomment-5479391872
> I think this is an optimization for not splitting unnecessary wal edits,
but why it will lead to merge stuck? After opening a region, we should have
removed all the recovered.edits files?
Thanks @Apache9 for looking into it. Below is incident brief explanation.
I've also raised [PR8583](https://github.com/apache/hbase/pull/8583) to reopen
parent region on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS Linked to
this incident.
What Happened:
1. Region 112d9f08 was gracefully moved from rs-132 → rs-81 at 16:36:34 UTC.
2. The close on rs-132 had already flushed the region's edits, and rs-81
opened the region with openSeqNum=4997750282, establishing that the edits up to
that point were durable.
3. ~25 seconds later, rs-132 was declared dead as part of the broader
graceful RS drain.
4. The WAL split worker (rs-34) subsequently created a recovered.edits file
for this region containing edit seqId=4997750280.
5. This edit was already durable in the HFiles. The recovered edit file was
effectively stale/orphaned, but its existence was not recognized as harmless.
Where it got stuck:
At 18:32:33 UTC, MergeTableRegionsProcedure (pid=46990253) unassigned the
region successfully.
During MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, the procedure saw the
recovered.edits file and failed the check.
The merge procedure retained the region lock and did not recover
automatically, leaving the region in CLOSED/RIT for ~48m 56s.
The RIT only cleared after the HMaster failover caused the procedure to
replay and fresh top-level ASSIGN procedures reopened the regions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
