[
https://issues.apache.org/jira/browse/PHOENIX-6699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17530952#comment-17530952
]
ASF GitHub Bot commented on PHOENIX-6699:
-----------------------------------------
joshelser commented on code in PR #1431:
URL: https://github.com/apache/phoenix/pull/1431#discussion_r863210369
##########
phoenix-core/src/main/java/org/apache/phoenix/monitoring/MetricUtil.java:
##########
@@ -38,4 +46,22 @@ public static MetricsStopWatch getMetricsStopWatch(boolean
isRequestMetricsEnabl
return new MetricsStopWatch(true);
}
+ // We need to cover the case when JmxCacheBuster has just stopped the
HBase metrics
+ // system, and not accidentally overwrite the DefaultMetricsSystem
singleton.
+ // See PHOENIX-6699
+ public static boolean isDefaultMetricsInitialized() {
+ try {
+ MetricsSystemImpl metrics = (MetricsSystemImpl)
DefaultMetricsSystem.instance();
+ Field prefixField =
MetricsSystemImpl.class.getDeclaredField("prefix");
Review Comment:
> we could just to check if we are and RS/Master process, and skip the
initialization, but I don't know a way to do that.
I'm not coming up with any static state that you could look at to try to
determine if you're in the context of a RS or Master, shy of _hoping_ that the
classpath client side doesnt' have the `HRegionServer.class` in it (or similar).
The other thing I thought of was looking at `RpcServer.getCurrentCall()`.
Normally, there is a corresponding user RPC which would result in this
returning the current RPC. But, this method would return empty during a
compaction (e.g. updating Phoenix stats) where there is no active HBase Call.
Could we somehow separate the Phoenix client-side-only metrics from the
execution pathway of the Phoenix server-side metrics? I imagine this is no
easier to do.
> Phoenix metrics overwriting DefaultMetricsSystem in RegionServers
> -----------------------------------------------------------------
>
> Key: PHOENIX-6699
> URL: https://issues.apache.org/jira/browse/PHOENIX-6699
> Project: Phoenix
> Issue Type: Bug
> Components: core
> Affects Versions: 5.2.0
> Reporter: Istvan Toth
> Assignee: Istvan Toth
> Priority: Major
>
> There is a race condition in the Hbase and Phoenix metrics handling.
> Hbase metrics is supposed to run in the daemon processes, while Phoenix
> metrics is supposed to -run on the phoenix client side.- create its own
> DefaultMetrics object on the client side, and add its metrics to the HBase
> DefaultMetrics object on the RS side.
> On an RS both HBase and Phoenix tries to register its metrics implementation
> into DefaultMetricsSystem.
>
> On the happy path, when Phoenix tries to register its metrics on the RS, it
> won't have any effect, because HBase has already registered an active Metrics
> implementation.
> However, JmxCacheBuster stops the Hbase metrics system for ~500ms every five
> seconds, and if Phoenix tries to initialize its metrics in this period, then
> it will succeed in initializing its Metrics object, and will overwrite
> theDefaultMetricsSystem object initialized by HBase.
> This disables the HBase metrics system for the RS process.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
