[
https://issues.apache.org/jira/browse/NIFI-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard resolved NIFI-16296.
-----------------------------------
Fix Version/s: 2.12.0
Resolution: Fixed
> Stale Prometheus cluster_connected_node_count metrics accumulate indefinitely
> due to unbounded connected_nodes label values
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16296
> URL: https://issues.apache.org/jira/browse/NIFI-16296
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.10.0, 2.11.0
> Reporter: Alexander Bij
> Priority: Major
> Labels: Matrices, monitoring, prometheus
> Fix For: 2.12.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The Prometheus flow metrics endpoint (/nifi-api/flow/metrics/prometheus)
> exposes cluster_connected_node_count with a connected_nodes label containing
> a human-readable ratio string, e.g. connected_nodes="3 / 4". Because this
> label's value changes over the lifetime of the process (as nodes join/leave
> the cluster) and the underlying Gauge is never cleared, every distinct ratio
> the cluster has ever reported remains permanently exposed as a separate,
> frozen time series – alongside the current, correct one.
>
> For a 4-node cluster this means a scrape can return multiple lines like:
> {noformat}
> cluster_connected_node_count{instance="<uuid>",connected_nodes="3 / 4",} 3.0
> cluster_connected_node_count{instance="<uuid>",connected_nodes="4 / 4",} 4.0
> {noformat}
> ...even though the cluster is currently fully connected (4/4). The "3 / 4"
> series is stale – a snapshot from an earlier, transient state – but is never
> removed, so it is scraped forever (until the node's JVM restarts), making it
> impossible to build a reliable Prometheus/Grafana panel for cluster health
> without manually filtering out stale label combinations you can't distinguish
> from current ones.
>
> h2. Root cause
> ClusterMetricsRegistry
> (org.apache.nifi.prometheusutil.ClusterMetricsRegistry) is a permanent
> singleton field on StandardNiFiServiceFacade:
> {code:java}
> private final ClusterMetricsRegistry clusterMetricsRegistry = new
> ClusterMetricsRegistry();
> {code}
> Its cluster_connected_node_count gauge is declared with connected_nodes as a
> {*}label{*}, not just a numeric value:
> {code:java}
> nameToGaugeMap.put("CONNECTED_NODE_COUNT", Gauge.build()
> .name("cluster_connected_node_count")
> .help("The number of connected nodes in this cluster")
> .labelNames("instance", "connected_nodes") # <<-- ISSUE HERE
> .register(registry));
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
