[
https://issues.apache.org/jira/browse/NIFI-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111897#comment-18111897
]
ASF subversion and git services commented on NIFI-16296:
--------------------------------------------------------
Commit 9823a25ea73da9de880616e3ed156915229766a6 in nifi's branch
refs/heads/main from Alexander
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=9823a25ea73 ]
NIFI-16296 Remove connected_nodes label from cluster_connected_node_count to
prevent stale Prometheus series (#11623)
> Stale Prometheus cluster_connected_node_count metrics accumulate indefinitely
> due to unbounded connected_nodes label values
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16296
> URL: https://issues.apache.org/jira/browse/NIFI-16296
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.10.0, 2.11.0
> Reporter: Alexander Bij
> Priority: Major
> Labels: Matrices, monitoring, prometheus
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The Prometheus flow metrics endpoint (/nifi-api/flow/metrics/prometheus)
> exposes cluster_connected_node_count with a connected_nodes label containing
> a human-readable ratio string, e.g. connected_nodes="3 / 4". Because this
> label's value changes over the lifetime of the process (as nodes join/leave
> the cluster) and the underlying Gauge is never cleared, every distinct ratio
> the cluster has ever reported remains permanently exposed as a separate,
> frozen time series – alongside the current, correct one.
>
> For a 4-node cluster this means a scrape can return multiple lines like:
> {noformat}
> cluster_connected_node_count{instance="<uuid>",connected_nodes="3 / 4",} 3.0
> cluster_connected_node_count{instance="<uuid>",connected_nodes="4 / 4",} 4.0
> {noformat}
> ...even though the cluster is currently fully connected (4/4). The "3 / 4"
> series is stale – a snapshot from an earlier, transient state – but is never
> removed, so it is scraped forever (until the node's JVM restarts), making it
> impossible to build a reliable Prometheus/Grafana panel for cluster health
> without manually filtering out stale label combinations you can't distinguish
> from current ones.
>
> h2. Root cause
> ClusterMetricsRegistry
> (org.apache.nifi.prometheusutil.ClusterMetricsRegistry) is a permanent
> singleton field on StandardNiFiServiceFacade:
> {code:java}
> private final ClusterMetricsRegistry clusterMetricsRegistry = new
> ClusterMetricsRegistry();
> {code}
> Its cluster_connected_node_count gauge is declared with connected_nodes as a
> {*}label{*}, not just a numeric value:
> {code:java}
> nameToGaugeMap.put("CONNECTED_NODE_COUNT", Gauge.build()
> .name("cluster_connected_node_count")
> .help("The number of connected nodes in this cluster")
> .labelNames("instance", "connected_nodes") # <<-- ISSUE HERE
> .register(registry));
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
