[
https://issues.apache.org/jira/browse/FLINK-40372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liting liu updated FLINK-40372:
-------------------------------
Description:
Description
OperatorJosdkMetrics creates resource-scoped Flink metrics for every Kubernetes
custom resource reconciled by JOSDK. The metric scope includes the resource
namespace and name.
When resource cleanup completes, cleanupDone() only increments a cleanup
counter:
@Override
public void cleanupDone(ResourceID resourceID, Map<String, Object> metadata) {
counter(getResourceMg(resourceID, metadata), RECONCILIATION,
"cleanup").inc();
}
It does not close or remove the resource metric group, nor remove the
associated entries from resourceNsMetricGroups, resourceMetricGroups, and
counters.
Furthermore, calling getResourceMg() from cleanupDone() can create a new metric
group during cleanup if one does not already exist.
Consequently, metrics associated with deleted resource names remain registered
for the lifetime of the Operator. Workloads that continuously create resources
with unique names, such as periodic FlinkStateSnapshot resources, cause the
number of metric groups, counters, and metric-scope strings to grow over time.
When the SLF4J metric reporter is enabled, every reporting cycle also traverses
and formats the accumulated metrics, increasing allocation pressure and
reporting cost.
This was observed with an Apache Flink Kubernetes Operator 1.13-based build.
The same cleanup behavior is still present on Apache main at commit:
fdfcc695daba16e4f4a8c9a6c08b7f470787de00
The Operator eventually reported heap exhaustion while executing the metric
reporter. The following lines are copied from the Operator logs with only ANSI
color codes removed:
2026-08-12 05:54:08,290 o.a.f.r.m.MetricRegistryImpl [WARN ] Error while
reporting metrics
java.lang.OutOfMemoryError: Java heap space
The source code demonstrates that resource-scoped metrics are not cleaned up.
The production symptoms are consistent with the resulting retention, although a
heap dump is still required to quantify the retained heap and confirm the
dominant GC retention path.
was:
```
## Description
`OperatorJosdkMetrics` creates Flink metric groups and counters whose scopes
contain
the namespace and name of every reconciled Kubernetes custom resource.
After a resource is successfully deleted, `cleanupDone()` only increments a
cleanup
counter. It does not close or remove the metric group, and it does not remove
the
associated entries from the internal maps.
As resource names change over time, especially for automatically generated
`FlinkStateSnapshot` resources such as periodic savepoints, the number of
registered
metrics can grow monotonically.
This increases metric cardinality and retains metric groups, counters and their
scope
strings for resources that no longer need to be monitored. When the SLF4J
reporter is
enabled, every reporting cycle also iterates over and formats the accumulated
metrics.
## Affected versions
Observed with an Apache Flink Kubernetes Operator 1.13 based build.
The same lifecycle issue is still present on Apache `main` at:
- commit: `fdfcc695daba16e4f4a8c9a6c08b7f470787de00`
- date: 2026-08-11
JOSDK version in the 1.13 based build: 5.1.2.
## Relevant configuration
The SLF4J reporter was configured with a five-minute interval:
kubernetes.operator.metrics.reporter.slf4j.factory.class:
org.apache.flink.metrics.slf4j.Slf4jReporterFactory
kubernetes.operator.metrics.reporter.slf4j.interval: 5 MINUTE
The Operator container had a 1 GiB memory limit.
## Source analysis
`OperatorJosdkMetrics` keeps the following maps for the lifetime of the
Operator:
```java
private final Map<ResourceID, KubernetesResourceNamespaceMetricGroup>
resourceNsMetricGroups = new ConcurrentHashMap<>();
private final Map<ResourceID, KubernetesResourceMetricGroup>
resourceMetricGroups = new ConcurrentHashMap<>();
private final Map<List<String>, Histogram> histograms = new
ConcurrentHashMap<>();
private final Map<List<String>, Counter> counters = new ConcurrentHashMap<>();
```
> [Kubernetes Operator] JOSDK per-resource metrics are not unregistered after
> resource cleanup
> --------------------------------------------------------------------------------------------
>
> Key: FLINK-40372
> URL: https://issues.apache.org/jira/browse/FLINK-40372
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Affects Versions: 1.13
> Reporter: liting liu
> Priority: Critical
>
> Description
> OperatorJosdkMetrics creates resource-scoped Flink metrics for every
> Kubernetes custom resource reconciled by JOSDK. The metric scope includes the
> resource namespace and name.
> When resource cleanup completes, cleanupDone() only increments a cleanup
> counter:
> @Override
> public void cleanupDone(ResourceID resourceID, Map<String, Object> metadata) {
> counter(getResourceMg(resourceID, metadata), RECONCILIATION,
> "cleanup").inc();
> }
> It does not close or remove the resource metric group, nor remove the
> associated entries from resourceNsMetricGroups, resourceMetricGroups, and
> counters.
> Furthermore, calling getResourceMg() from cleanupDone() can create a new
> metric group during cleanup if one does not already exist.
> Consequently, metrics associated with deleted resource names remain
> registered for the lifetime of the Operator. Workloads that continuously
> create resources with unique names, such as periodic FlinkStateSnapshot
> resources, cause the number of metric groups, counters, and metric-scope
> strings to grow over time.
> When the SLF4J metric reporter is enabled, every reporting cycle also
> traverses and formats the accumulated metrics, increasing allocation pressure
> and reporting cost.
> This was observed with an Apache Flink Kubernetes Operator 1.13-based build.
> The same cleanup behavior is still present on Apache main at commit:
> fdfcc695daba16e4f4a8c9a6c08b7f470787de00
> The Operator eventually reported heap exhaustion while executing the metric
> reporter. The following lines are copied from the Operator logs with only
> ANSI color codes removed:
> 2026-08-12 05:54:08,290 o.a.f.r.m.MetricRegistryImpl [WARN ] Error while
> reporting metrics
> java.lang.OutOfMemoryError: Java heap space
> The source code demonstrates that resource-scoped metrics are not cleaned up.
> The production symptoms are consistent with the resulting retention, although
> a heap dump is still required to quantify the retained heap and confirm the
> dominant GC retention path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
