[
https://issues.apache.org/jira/browse/FLINK-40372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-40372:
-----------------------------------
Labels: pull-request-available (was: )
> [Kubernetes Operator] JOSDK per-resource metrics are not unregistered after
> resource cleanup
> --------------------------------------------------------------------------------------------
>
> Key: FLINK-40372
> URL: https://issues.apache.org/jira/browse/FLINK-40372
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Affects Versions: 1.13
> Reporter: liting liu
> Priority: Critical
> Labels: pull-request-available
>
> Description
> OperatorJosdkMetrics creates resource-scoped Flink metrics for every
> Kubernetes custom resource reconciled by JOSDK. The metric scope includes the
> resource namespace and name.
> When resource cleanup completes, cleanupDone() only increments a cleanup
> counter:
> @Override
> public void cleanupDone(ResourceID resourceID, Map<String, Object> metadata)
> { counter(getResourceMg(resourceID, metadata), RECONCILIATION,
> "cleanup").inc(); }
> It does not close or remove the resource metric group, nor remove the
> associated entries from resourceNsMetricGroups, resourceMetricGroups, and
> counters.
> Furthermore, calling getResourceMg() from cleanupDone() can create a new
> metric group during cleanup if one does not already exist.
> Consequently, metrics associated with deleted resource names remain
> registered for the lifetime of the Operator. Workloads that continuously
> create resources with unique names, such as periodic FlinkStateSnapshot
> resources, cause the number of metric groups, counters, and metric-scope
> strings to grow over time.
> When the SLF4J metric reporter is enabled, every reporting cycle also
> traverses and formats the accumulated metrics, increasing allocation pressure
> and reporting cost.
> This was observed with an Apache Flink Kubernetes Operator 1.13-based build.
> The same cleanup behavior is still present on Apache main at commit:
> fdfcc695daba16e4f4a8c9a6c08b7f470787de00
> The Operator eventually reported heap exhaustion while executing the metric
> reporter. The following lines are copied from the Operator logs with only
> ANSI color codes removed:
> 2026-08-12 05:54:08,290 o.a.f.r.m.MetricRegistryImpl [WARN ] Error while
> reporting metrics
> java.lang.OutOfMemoryError: Java heap space
> The source code demonstrates that resource-scoped metrics are not cleaned up.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
