[
https://issues.apache.org/jira/browse/FLINK-40372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liting liu updated FLINK-40372:
-------------------------------
Description:
```
## Description
`OperatorJosdkMetrics` creates Flink metric groups and counters whose scopes
contain
the namespace and name of every reconciled Kubernetes custom resource.
After a resource is successfully deleted, `cleanupDone()` only increments a
cleanup
counter. It does not close or remove the metric group, and it does not remove
the
associated entries from the internal maps.
As resource names change over time, especially for automatically generated
`FlinkStateSnapshot` resources such as periodic savepoints, the number of
registered
metrics can grow monotonically.
This increases metric cardinality and retains metric groups, counters and their
scope
strings for resources that no longer need to be monitored. When the SLF4J
reporter is
enabled, every reporting cycle also iterates over and formats the accumulated
metrics.
## Affected versions
Observed with an Apache Flink Kubernetes Operator 1.13 based build.
The same lifecycle issue is still present on Apache `main` at:
- commit: `fdfcc695daba16e4f4a8c9a6c08b7f470787de00`
- date: 2026-08-11
JOSDK version in the 1.13 based build: 5.1.2.
## Relevant configuration
The SLF4J reporter was configured with a five-minute interval:
kubernetes.operator.metrics.reporter.slf4j.factory.class:
org.apache.flink.metrics.slf4j.Slf4jReporterFactory
kubernetes.operator.metrics.reporter.slf4j.interval: 5 MINUTE
The Operator container had a 1 GiB memory limit.
## Source analysis
`OperatorJosdkMetrics` keeps the following maps for the lifetime of the
Operator:
```java
private final Map<ResourceID, KubernetesResourceNamespaceMetricGroup>
resourceNsMetricGroups = new ConcurrentHashMap<>();
private final Map<ResourceID, KubernetesResourceMetricGroup>
resourceMetricGroups = new ConcurrentHashMap<>();
private final Map<List<String>, Histogram> histograms = new
ConcurrentHashMap<>();
private final Map<List<String>, Counter> counters = new ConcurrentHashMap<>();
```
was:
## Description
`OperatorJosdkMetrics` creates Flink metric groups and counters whose scopes
contain
the namespace and name of every reconciled Kubernetes custom resource.
After a resource is successfully deleted, `cleanupDone()` only increments a
cleanup
counter. It does not close or remove the metric group, and it does not remove
the
associated entries from the internal maps.
As resource names change over time, especially for automatically generated
`FlinkStateSnapshot` resources such as periodic savepoints, the number of
registered
metrics can grow monotonically.
This increases metric cardinality and retains metric groups, counters and their
scope
strings for resources that no longer need to be monitored. When the SLF4J
reporter is
enabled, every reporting cycle also iterates over and formats the accumulated
metrics.
## Affected versions
Observed with an Apache Flink Kubernetes Operator 1.13 based build.
The same lifecycle issue is still present on Apache `main` at:
- commit: `fdfcc695daba16e4f4a8c9a6c08b7f470787de00`
- date: 2026-08-11
JOSDK version in the 1.13 based build: 5.1.2.
## Relevant configuration
The SLF4J reporter was configured with a five-minute interval:
kubernetes.operator.metrics.reporter.slf4j.factory.class:
org.apache.flink.metrics.slf4j.Slf4jReporterFactory
kubernetes.operator.metrics.reporter.slf4j.interval: 5 MINUTE
The Operator container had a 1 GiB memory limit.
## Source analysis
`OperatorJosdkMetrics` keeps the following maps for the lifetime of the
Operator:
```java
private final Map<ResourceID, KubernetesResourceNamespaceMetricGroup>
resourceNsMetricGroups = new ConcurrentHashMap<>();
private final Map<ResourceID, KubernetesResourceMetricGroup>
resourceMetricGroups = new ConcurrentHashMap<>();
private final Map<List<String>, Histogram> histograms = new
ConcurrentHashMap<>();
private final Map<List<String>, Counter> counters = new ConcurrentHashMap<>();
> [Kubernetes Operator] JOSDK per-resource metrics are not unregistered after
> resource cleanup
> --------------------------------------------------------------------------------------------
>
> Key: FLINK-40372
> URL: https://issues.apache.org/jira/browse/FLINK-40372
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Affects Versions: 1.13
> Reporter: liting liu
> Priority: Critical
>
> ```
> ## Description
> `OperatorJosdkMetrics` creates Flink metric groups and counters whose scopes
> contain
> the namespace and name of every reconciled Kubernetes custom resource.
> After a resource is successfully deleted, `cleanupDone()` only increments a
> cleanup
> counter. It does not close or remove the metric group, and it does not remove
> the
> associated entries from the internal maps.
> As resource names change over time, especially for automatically generated
> `FlinkStateSnapshot` resources such as periodic savepoints, the number of
> registered
> metrics can grow monotonically.
> This increases metric cardinality and retains metric groups, counters and
> their scope
> strings for resources that no longer need to be monitored. When the SLF4J
> reporter is
> enabled, every reporting cycle also iterates over and formats the accumulated
> metrics.
> ## Affected versions
> Observed with an Apache Flink Kubernetes Operator 1.13 based build.
> The same lifecycle issue is still present on Apache `main` at:
> - commit: `fdfcc695daba16e4f4a8c9a6c08b7f470787de00`
> - date: 2026-08-11
> JOSDK version in the 1.13 based build: 5.1.2.
> ## Relevant configuration
> The SLF4J reporter was configured with a five-minute interval:
> kubernetes.operator.metrics.reporter.slf4j.factory.class:
> org.apache.flink.metrics.slf4j.Slf4jReporterFactory
> kubernetes.operator.metrics.reporter.slf4j.interval: 5 MINUTE
> The Operator container had a 1 GiB memory limit.
> ## Source analysis
> `OperatorJosdkMetrics` keeps the following maps for the lifetime of the
> Operator:
> ```java
> private final Map<ResourceID, KubernetesResourceNamespaceMetricGroup>
> resourceNsMetricGroups = new ConcurrentHashMap<>();
> private final Map<ResourceID, KubernetesResourceMetricGroup>
> resourceMetricGroups = new ConcurrentHashMap<>();
> private final Map<List<String>, Histogram> histograms = new
> ConcurrentHashMap<>();
> private final Map<List<String>, Counter> counters = new ConcurrentHashMap<>();
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
