[
https://issues.apache.org/jira/browse/FLINK-40372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liting liu updated FLINK-40372:
-------------------------------
Description:
Description
OperatorJosdkMetrics creates resource-scoped Flink metrics for every Kubernetes
custom resource reconciled by JOSDK. The metric scope includes the resource
namespace and name.
When resource cleanup completes, cleanupDone() only increments a cleanup
counter:
@Override
public void cleanupDone(ResourceID resourceID, Map<String, Object> metadata)
{ counter(getResourceMg(resourceID, metadata), RECONCILIATION,
"cleanup").inc(); }
It does not close or remove the resource metric group, nor remove the
associated entries from resourceNsMetricGroups, resourceMetricGroups, and
counters.
Furthermore, calling getResourceMg() from cleanupDone() can create a new metric
group during cleanup if one does not already exist.
Consequently, metrics associated with deleted resource names remain registered
for the lifetime of the Operator. Workloads that continuously create resources
with unique names, such as periodic FlinkStateSnapshot resources, cause the
number of metric groups, counters, and metric-scope strings to grow over time.
When the SLF4J metric reporter is enabled, every reporting cycle also traverses
and formats the accumulated metrics, increasing allocation pressure and
reporting cost.
This was observed with an Apache Flink Kubernetes Operator 1.13-based build.
The same cleanup behavior is still present on Apache main at commit:
fdfcc695daba16e4f4a8c9a6c08b7f470787de00
The Operator eventually reported heap exhaustion while executing the metric
reporter. The following lines are copied from the Operator logs with only ANSI
color codes removed:
2026-08-12 05:54:08,290 o.a.f.r.m.MetricRegistryImpl [WARN ] Error while
reporting metrics
java.lang.OutOfMemoryError: Java heap space
The source code demonstrates that resource-scoped metrics are not cleaned up.
was:
Description
OperatorJosdkMetrics creates resource-scoped Flink metrics for every Kubernetes
custom resource reconciled by JOSDK. The metric scope includes the resource
namespace and name.
When resource cleanup completes, cleanupDone() only increments a cleanup
counter:
@Override
public void cleanupDone(ResourceID resourceID, Map<String, Object> metadata) {
counter(getResourceMg(resourceID, metadata), RECONCILIATION,
"cleanup").inc();
}
It does not close or remove the resource metric group, nor remove the
associated entries from resourceNsMetricGroups, resourceMetricGroups, and
counters.
Furthermore, calling getResourceMg() from cleanupDone() can create a new metric
group during cleanup if one does not already exist.
Consequently, metrics associated with deleted resource names remain registered
for the lifetime of the Operator. Workloads that continuously create resources
with unique names, such as periodic FlinkStateSnapshot resources, cause the
number of metric groups, counters, and metric-scope strings to grow over time.
When the SLF4J metric reporter is enabled, every reporting cycle also traverses
and formats the accumulated metrics, increasing allocation pressure and
reporting cost.
This was observed with an Apache Flink Kubernetes Operator 1.13-based build.
The same cleanup behavior is still present on Apache main at commit:
fdfcc695daba16e4f4a8c9a6c08b7f470787de00
The Operator eventually reported heap exhaustion while executing the metric
reporter. The following lines are copied from the Operator logs with only ANSI
color codes removed:
2026-08-12 05:54:08,290 o.a.f.r.m.MetricRegistryImpl [WARN ] Error while
reporting metrics
java.lang.OutOfMemoryError: Java heap space
The source code demonstrates that resource-scoped metrics are not cleaned up.
The production symptoms are consistent with the resulting retention, although a
heap dump is still required to quantify the retained heap and confirm the
dominant GC retention path.
> [Kubernetes Operator] JOSDK per-resource metrics are not unregistered after
> resource cleanup
> --------------------------------------------------------------------------------------------
>
> Key: FLINK-40372
> URL: https://issues.apache.org/jira/browse/FLINK-40372
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Affects Versions: 1.13
> Reporter: liting liu
> Priority: Critical
>
> Description
> OperatorJosdkMetrics creates resource-scoped Flink metrics for every
> Kubernetes custom resource reconciled by JOSDK. The metric scope includes the
> resource namespace and name.
> When resource cleanup completes, cleanupDone() only increments a cleanup
> counter:
> @Override
> public void cleanupDone(ResourceID resourceID, Map<String, Object> metadata)
> { counter(getResourceMg(resourceID, metadata), RECONCILIATION,
> "cleanup").inc(); }
> It does not close or remove the resource metric group, nor remove the
> associated entries from resourceNsMetricGroups, resourceMetricGroups, and
> counters.
> Furthermore, calling getResourceMg() from cleanupDone() can create a new
> metric group during cleanup if one does not already exist.
> Consequently, metrics associated with deleted resource names remain
> registered for the lifetime of the Operator. Workloads that continuously
> create resources with unique names, such as periodic FlinkStateSnapshot
> resources, cause the number of metric groups, counters, and metric-scope
> strings to grow over time.
> When the SLF4J metric reporter is enabled, every reporting cycle also
> traverses and formats the accumulated metrics, increasing allocation pressure
> and reporting cost.
> This was observed with an Apache Flink Kubernetes Operator 1.13-based build.
> The same cleanup behavior is still present on Apache main at commit:
> fdfcc695daba16e4f4a8c9a6c08b7f470787de00
> The Operator eventually reported heap exhaustion while executing the metric
> reporter. The following lines are copied from the Operator logs with only
> ANSI color codes removed:
> 2026-08-12 05:54:08,290 o.a.f.r.m.MetricRegistryImpl [WARN ] Error while
> reporting metrics
> java.lang.OutOfMemoryError: Java heap space
> The source code demonstrates that resource-scoped metrics are not cleaned up.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
