[
https://issues.apache.org/jira/browse/HBASE-30350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HBASE-30350:
-----------------------------------
Labels: pull-request-available (was: )
> Stale entry left in RegionInTransitionTracker after GCRegionProcedure purges
> region metadata
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-30350
> URL: https://issues.apache.org/jira/browse/HBASE-30350
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
> Affects Versions: 3.0.0, 4.0.0-alpha-1, 2.5.16, 2.6.7
> Reporter: Aman Poonia
> Assignee: Aman Poonia
> Priority: Major
> Labels: pull-request-available
>
> GCRegionProcedure (state {{GC_REGION_PURGE_METADATA}}) removes a GC'd
> split/merged parent region from {{AssignmentManager#regionStates}} but never
> removes the corresponding entry from the separate
> {{RegionInTransitionTracker}} map. These two maps are not kept in sync
> automatically.
> The analogous cleanup path, {{AssignmentManager#markRegionAsMerged}},
> correctly calls both {{regionStates.deleteRegion(ri)}} and
> {{regionInTransitionTracker.handleRegionDelete(ri)}} for each merge
> parent. {{GCRegionProcedure}} only calls the former.
> As a result, once a region is GC'd via {{CatalogJanitor}} ->
> {{GCRegionProcedure}}, its ghost entry remains in
> {{RegionInTransitionTracker}} indefinitely. The periodic "STUCK
> Region-In-Transition"
> chore ({{ProcExecTimeout}} thread) keeps re-reporting this already-deleted
> region as stuck on every run, producing misleading WARN logs and inflating
> RIT-age metrics/alerts. The stale entry is only
> cleared when the active master fails over, since
> {{AssignmentManager#joinCluster}} -> {{loadMeta}} rebuilds all in-memory
> assignment state fresh from {{hbase:meta}}.
> *Observed in production:* a region was GC'd (fully removed from
> {{hbase:meta}} and HDFS) but continued to be logged as "STUCK
> Region-In-Transition" for ~9 hours until the next master failover
> silently cleared it, giving a false impression of an actual 9-hour stuck
> RIT / unavailability.
> *Fix:* add a shared {{AssignmentManager#deleteRegion(RegionInfo)}} method
> that removes the region from both {{regionStates}} and
> {{regionInTransitionTracker}} together, and have both
> {{markRegionAsMerged}} and {{GCRegionProcedure}} call it instead of
> touching {{regionStates}} directly.
> Confirmed present on {{apache/master}} as of commit fb4286d3046.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
