Aman Poonia created HBASE-30350:
-----------------------------------
Summary: Stale entry left in RegionInTransitionTracker after
GCRegionProcedure purges region metadata
Key: HBASE-30350
URL: https://issues.apache.org/jira/browse/HBASE-30350
Project: HBase
Issue Type: Bug
Components: master, Region Assignment
Affects Versions: 2.6.7, 2.5.16, 3.0.0, 4.0.0-alpha-1
Reporter: Aman Poonia
Assignee: Aman Poonia
GCRegionProcedure (state {{GC_REGION_PURGE_METADATA}}) removes a GC'd
split/merged parent region from {{AssignmentManager#regionStates}} but never
removes the corresponding entry from the separate
{{RegionInTransitionTracker}} map. These two maps are not kept in sync
automatically.
The analogous cleanup path, {{AssignmentManager#markRegionAsMerged}},
correctly calls both {{regionStates.deleteRegion(ri)}} and
{{regionInTransitionTracker.handleRegionDelete(ri)}} for each merge
parent. {{GCRegionProcedure}} only calls the former.
As a result, once a region is GC'd via {{CatalogJanitor}} ->
{{GCRegionProcedure}}, its ghost entry remains in {{RegionInTransitionTracker}}
indefinitely. The periodic "STUCK Region-In-Transition"
chore ({{ProcExecTimeout}} thread) keeps re-reporting this already-deleted
region as stuck on every run, producing misleading WARN logs and inflating
RIT-age metrics/alerts. The stale entry is only
cleared when the active master fails over, since
{{AssignmentManager#joinCluster}} -> {{loadMeta}} rebuilds all in-memory
assignment state fresh from {{hbase:meta}}.
*Observed in production:* a region was GC'd (fully removed from
{{hbase:meta}} and HDFS) but continued to be logged as "STUCK
Region-In-Transition" for ~9 hours until the next master failover
silently cleared it, giving a false impression of an actual 9-hour stuck RIT
/ unavailability.
*Fix:* add a shared {{AssignmentManager#deleteRegion(RegionInfo)}} method
that removes the region from both {{regionStates}} and
{{regionInTransitionTracker}} together, and have both
{{markRegionAsMerged}} and {{GCRegionProcedure}} call it instead of touching
{{regionStates}} directly.
Confirmed present on {{apache/master}} as of commit fb4286d3046.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
