[
https://issues.apache.org/jira/browse/HBASE-28123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18063881#comment-18063881
]
Xiao Liu commented on HBASE-28123:
----------------------------------
Hi, [~bbeaudreault] and [~zhangduo]
I think the core issue lies in how \{{MetricsTableRequests}} register via
\{{createRegistryForTableRequests}} by incrementing reference counts in
\{{MetricRegistries.global()}}. When all regions shut down, the
\{{removeRegistry}} method is invoked to remove \{{metricRegistryInfo}} from
\{{MetricRegistries.global()}}. This causes a new \{{MetricRegistry}} to be
initialized when the region comes back online. This \{{MetricRegistry}} is not
registered in \{{DefaultMetricsSystem.instance()}}, as only the
\{{MetricsSourceAdapter}} containing the old \{{MetricRegistry}} was previously
registered in \{{GlobalMetricRegistriesAdapter}}. The invalid registry is not
promptly removed because the \{{doRun}} method performs a 10-second polling
check.
{code:language=bash|borderStyle=solid|theme=RDark|linenumbers=true|collapse=false}
1. Region closed
└── MetricsTableRequests.removeRegistry()
└── MetricRegistries.global().remove(info)
└── RefCountingMap.remove() → refCount--
└── When refCount == 0, MetricRegistry is removed from the map
2. Region reopens
└── MetricsTableRequests.init()
└── MetricRegistries.global().create(info)
└── RefCountingMap.put() → refCount=1, creates new MetricRegistry
instance
3. GlobalMetricRegistriesAdapter.doRun() (10 seconds later)
└── Detects registry instance for info exists, takes no action
└── Fails to update registeredSources and DefaultMetricsSystem, JMX still
points to old registry (no data updated)
{code}
> TableRequests metrics broken after altering table
> -------------------------------------------------
>
> Key: HBASE-28123
> URL: https://issues.apache.org/jira/browse/HBASE-28123
> Project: HBase
> Issue Type: Bug
> Reporter: Bryan Beaudreault
> Priority: Major
> Labels: pull-request-available
>
> Alerting a table reopens all regions. I noticed after altering one of my
> tables that afterwards the TableRequests metrics no longer update. They seem
> stuck at whatever they were prior. I'm thinking this has something to do with
> the refactor done in https://issues.apache.org/jira/browse/HBASE-27681, which
> attempts to deregister metrics when a table is removed from the server. I
> think when it gets re-registered the new metrics aren't linked to whatever
> the jmx adapter is looking at.
> [~tianhang] FYI
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
