[
https://issues.apache.org/jira/browse/FLINK-40372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liting liu updated FLINK-40372:
-------------------------------
Description:
our flink opeartor encounterd OOM after running for months. Our flink jobs
enable the \{{periodic}} savepoint.
There are some detail logs can be reporduced, just create a job enable the
periodic savepoint then monitor the log:
only seven \{{jm-ha-f6ndtm-savepoint-periodic-*}} resources remained in the
cluster. However, the next metrics report at \{{11:18:12 CST}} still emitted
metrics for twelve resource names. The five additional resources had already
been deleted.
h3. Seven resources still present
Raw \{{kubectl get}} output:
{{jm-ha-f6ndtm-savepoint-periodic-1786589833663 2026-08-13T02:57:13Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589864994 2026-08-13T02:57:45Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589927560 2026-08-13T02:58:47Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589943145 2026-08-13T02:59:03Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589958617 2026-08-13T02:59:18Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589974113 2026-08-13T02:59:34Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589983358 <none> ABANDONED}}
h3. Five additional resources had already been deleted
Directly querying these resources returned \{{{}NotFound{}}}:
{{Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589817875" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589849241" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589880549" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589896162" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589911865" not found}}
h3. The next reporter cycle still emitted all twelve resources
The Operator logs use UTC. Therefore, \{{03:18:12}} corresponds to
\{{{}11:18:12 CST{}}}.
Raw reporter header:
{\{2026-08-13 03:18:12,173 o.a.f.m.s.Slf4jReporter [INFO ]}}
Resource metrics emitted during that report:
{{...jm-ha-f6ndtm-savepoint-periodic-1786589911865.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589817875.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589880549.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589974113.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589927560.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589896162.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589833663.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589943145.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589983358.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
2
...jm-ha-f6ndtm-savepoint-periodic-1786589958617.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589849241.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589864994.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3}}
The following five entries belong to deleted resources:
{{1786589817875
1786589849241
1786589880549
1786589896162
1786589911865}}
The runtime sequence is therefore:
{{11:17 CST Only 7 matching resources existed; the other 5 returned NotFound
11:18:12 CST The next reporter cycle still emitted metrics for all 12
resources}}
Additionally, the retained metrics for the deleted resources include:
{\{FlinkStateSnapshot.JOSDK.Resource.Event.DELETED.Count: 1}}
This shows that the deletion events had been observed, but the corresponding
resource-specific metrics continued to be emitted afterward.
A concise description for the issue:
Five \{{jm-ha-f6ndtm-savepoint-periodic-*}} resources no longer existed in
Kubernetes and returned \{{{}NotFound{}}}. However, the subsequent SLF4J
metrics report continued to emit their resource-specific JOSDK metrics. The
reporter emitted metrics for 12 resource names while only 7 matching resources
remained in the cluster. The retained metrics also contained
\{{{}Resource.Event.DELETED.Count: 1{}}}, confirming that the deletion events
had already been observed.
That unreleased resource crash the JVM heap as long as the job enable the
periodic savepoint and operator run long enough.
h2. Heap dump evidence
A live heap dump was captured from the reproduced Operator process after a full
GC and analyzed offline with Eclipse Memory Analyzer (MAT). The attached MAT
Leak Suspects report identifies
\{{org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics}} as
*Problem Suspect 1*:
{code}
OperatorJosdkMetrics retained heap: 14,784,328 bytes (14.10 MiB)
Share of post-GC live heap: 19.35%
resourceMetricGroups retained heap: 11,742,264 bytes
counters retained heap: 1,907,984 bytes
{code}
At the time of the dump, the adapter maps contained:
{code}
resourceMetricGroups: 1,801 entries
resourceNsMetricGroups: 1,801 entries
counters: 13,554 entries
histograms: 6 entries
{code}
The \{{ResourceID}} keys retained in \{{resourceMetricGroups}} were compared
with the Flink custom resources that still existed in the corresponding
Kubernetes namespaces:
{code}
Resource metric groups retained: 1,801
Resources still present: 151
Deleted resources still retained: 1,650
{code}
Of the 1,650 stale resource entries, 1,647 were \{{FlinkStateSnapshot}}
resources and 3 were \{{FlinkDeployment}} resources. They were associated with
12,745 retained JOSDK counter entries.
The stale resource metric groups alone retained approximately 10.9 MiB. A
conservative attribution that also includes their namespace groups, counter
keys and wrappers, metric-name strings retained by the reporter, counter
objects, and map-entry overhead totals approximately *16.7 MiB*, or *22.9% of
the post-GC live Java heap* in this reproduction. This 16.7 MiB value is
derived from MAT OQL results combined with the live Kubernetes resource-name
comparison; it is not a single built-in MAT suspect value.
The SLF4J reporter retained 4,206,696 bytes. It contained 13,607 counters in
total, of which 13,554 (99.6%) were JOSDK counters. Therefore, reporter cycles
become increasingly expensive as stale resource metrics accumulate.
h2. Root cause
{\{OperatorJosdkMetrics}} keeps resource-specific metric groups and counters in
long-lived maps. Observing a CR deletion does not unregister those metrics from
the adapter and the Flink metric registry/reporter. Consequently, a subsequent
reporter cycle continues to enumerate the deleted resource, even though its
\{{Resource.Event.DELETED.Count}} metric confirms that the deletion event was
received.
Periodic savepoints make this growth unbounded because every
\{{FlinkStateSnapshot}} has a unique timestamped name. Each deleted snapshot
can therefore leave another permanently registered set of resource-specific
metrics behind. Retries and failed snapshot cleanup increase the number of
counters per resource, but they are accelerators rather than the fundamental
leak.
{code}
Periodic FlinkStateSnapshot with a unique name
-> JOSDK resource metrics registered
-> snapshot CR deleted
-> DELETED event observed
-> resource metrics remain registered
-> SLF4J reporter keeps enumerating historical resources
-> retained heap and report-generation allocations grow for the Operator Pod
lifetime
-> GC pressure and eventual Java heap space OOM
{code}
h2. Impact
This is not limited to the five deleted resources shown in the log
reproduction. In the reproduced heap, 1,650 already-deleted resources were
still retained. The retained state grows with the number of unique resources
processed during the Operator Pod lifetime and is therefore unbounded for
workloads that continuously create timestamped periodic-savepoint resources.
was:
our flink opeartor encounterd OOM after running for months. Our flink jobs
enable the {{periodic}} savepoint.
There are some detail logs can be reporduced, just create a job enable the
periodic savepoint then monitor the log:
only seven {{jm-ha-f6ndtm-savepoint-periodic-*}} resources remained in the
cluster. However, the next metrics report at {{11:18:12 CST}} still emitted
metrics for twelve resource names. The five additional resources had already
been deleted.
h3. Seven resources still present
Raw {{kubectl get}} output:
{{jm-ha-f6ndtm-savepoint-periodic-1786589833663 2026-08-13T02:57:13Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589864994 2026-08-13T02:57:45Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589927560 2026-08-13T02:58:47Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589943145 2026-08-13T02:59:03Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589958617 2026-08-13T02:59:18Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589974113 2026-08-13T02:59:34Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589983358 <none> ABANDONED}}
h3. Five additional resources had already been deleted
Directly querying these resources returned {{{}NotFound{}}}:
{{Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589817875" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589849241" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589880549" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589896162" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589911865" not found}}
h3. The next reporter cycle still emitted all twelve resources
The Operator logs use UTC. Therefore, {{03:18:12}} corresponds to {{{}11:18:12
CST{}}}.
Raw reporter header:
{{2026-08-13 03:18:12,173 o.a.f.m.s.Slf4jReporter [INFO ]}}
Resource metrics emitted during that report:
{{...jm-ha-f6ndtm-savepoint-periodic-1786589911865.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589817875.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589880549.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589974113.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589927560.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589896162.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589833663.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589943145.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589983358.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
2
...jm-ha-f6ndtm-savepoint-periodic-1786589958617.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589849241.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589864994.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3}}
The following five entries belong to deleted resources:
{{1786589817875
1786589849241
1786589880549
1786589896162
1786589911865}}
The runtime sequence is therefore:
{{11:17 CST Only 7 matching resources existed; the other 5 returned NotFound
11:18:12 CST The next reporter cycle still emitted metrics for all 12
resources}}
Additionally, the retained metrics for the deleted resources include:
{{FlinkStateSnapshot.JOSDK.Resource.Event.DELETED.Count: 1}}
This shows that the deletion events had been observed, but the corresponding
resource-specific metrics continued to be emitted afterward.
A concise description for the issue:
Five {{jm-ha-f6ndtm-savepoint-periodic-*}} resources no longer existed in
Kubernetes and returned {{{}NotFound{}}}. However, the subsequent SLF4J metrics
report continued to emit their resource-specific JOSDK metrics. The reporter
emitted metrics for 12 resource names while only 7 matching resources remained
in the cluster. The retained metrics also contained
{{{}Resource.Event.DELETED.Count: 1{}}}, confirming that the deletion events
had already been observed.
That unreleased resource crash the JVM heap as long as the job enable the
periodic savepoint and operator run long enough.
> [Kubernetes Operator] JOSDK per-resource metrics are not unregistered after
> resource cleanup
> --------------------------------------------------------------------------------------------
>
> Key: FLINK-40372
> URL: https://issues.apache.org/jira/browse/FLINK-40372
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Affects Versions: 1.13
> Reporter: liting liu
> Priority: Critical
> Labels: pull-request-available
> Attachments: flink-operator-20260813-1405-live_Leak_Suspects.zip
>
>
> our flink opeartor encounterd OOM after running for months. Our flink jobs
> enable the \{{periodic}} savepoint.
> There are some detail logs can be reporduced, just create a job enable the
> periodic savepoint then monitor the log:
> only seven \{{jm-ha-f6ndtm-savepoint-periodic-*}} resources remained in the
> cluster. However, the next metrics report at \{{11:18:12 CST}} still emitted
> metrics for twelve resource names. The five additional resources had already
> been deleted.
> h3. Seven resources still present
> Raw \{{kubectl get}} output:
>
> {{jm-ha-f6ndtm-savepoint-periodic-1786589833663 2026-08-13T02:57:13Z
> IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589864994 2026-08-13T02:57:45Z IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589927560 2026-08-13T02:58:47Z IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589943145 2026-08-13T02:59:03Z IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589958617 2026-08-13T02:59:18Z IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589974113 2026-08-13T02:59:34Z IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589983358 <none> ABANDONED}}
> h3. Five additional resources had already been deleted
> Directly querying these resources returned \{{{}NotFound{}}}:
>
> {{Error from server (NotFound): flinkstatesnapshots.flink.apache.org
> "jm-ha-f6ndtm-savepoint-periodic-1786589817875" not found
> Error from server (NotFound): flinkstatesnapshots.flink.apache.org
> "jm-ha-f6ndtm-savepoint-periodic-1786589849241" not found
> Error from server (NotFound): flinkstatesnapshots.flink.apache.org
> "jm-ha-f6ndtm-savepoint-periodic-1786589880549" not found
> Error from server (NotFound): flinkstatesnapshots.flink.apache.org
> "jm-ha-f6ndtm-savepoint-periodic-1786589896162" not found
> Error from server (NotFound): flinkstatesnapshots.flink.apache.org
> "jm-ha-f6ndtm-savepoint-periodic-1786589911865" not found}}
> h3. The next reporter cycle still emitted all twelve resources
> The Operator logs use UTC. Therefore, \{{03:18:12}} corresponds to
> \{{{}11:18:12 CST{}}}.
> Raw reporter header:
>
> {\{2026-08-13 03:18:12,173 o.a.f.m.s.Slf4jReporter [INFO ]}}
> Resource metrics emitted during that report:
>
> {{...jm-ha-f6ndtm-savepoint-periodic-1786589911865.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 4
> ...jm-ha-f6ndtm-savepoint-periodic-1786589817875.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 4
> ...jm-ha-f6ndtm-savepoint-periodic-1786589880549.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 4
> ...jm-ha-f6ndtm-savepoint-periodic-1786589974113.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3
> ...jm-ha-f6ndtm-savepoint-periodic-1786589927560.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3
> ...jm-ha-f6ndtm-savepoint-periodic-1786589896162.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 4
> ...jm-ha-f6ndtm-savepoint-periodic-1786589833663.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3
> ...jm-ha-f6ndtm-savepoint-periodic-1786589943145.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3
> ...jm-ha-f6ndtm-savepoint-periodic-1786589983358.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 2
> ...jm-ha-f6ndtm-savepoint-periodic-1786589958617.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3
> ...jm-ha-f6ndtm-savepoint-periodic-1786589849241.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 4
> ...jm-ha-f6ndtm-savepoint-periodic-1786589864994.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3}}
> The following five entries belong to deleted resources:
>
> {{1786589817875
> 1786589849241
> 1786589880549
> 1786589896162
> 1786589911865}}
> The runtime sequence is therefore:
>
> {{11:17 CST Only 7 matching resources existed; the other 5 returned NotFound
> 11:18:12 CST The next reporter cycle still emitted metrics for all 12
> resources}}
> Additionally, the retained metrics for the deleted resources include:
>
> {\{FlinkStateSnapshot.JOSDK.Resource.Event.DELETED.Count: 1}}
> This shows that the deletion events had been observed, but the corresponding
> resource-specific metrics continued to be emitted afterward.
> A concise description for the issue:
> Five \{{jm-ha-f6ndtm-savepoint-periodic-*}} resources no longer existed in
> Kubernetes and returned \{{{}NotFound{}}}. However, the subsequent SLF4J
> metrics report continued to emit their resource-specific JOSDK metrics. The
> reporter emitted metrics for 12 resource names while only 7 matching
> resources remained in the cluster. The retained metrics also contained
> \{{{}Resource.Event.DELETED.Count: 1{}}}, confirming that the deletion events
> had already been observed.
> That unreleased resource crash the JVM heap as long as the job enable the
> periodic savepoint and operator run long enough.
> h2. Heap dump evidence
> A live heap dump was captured from the reproduced Operator process after a
> full GC and analyzed offline with Eclipse Memory Analyzer (MAT). The attached
> MAT Leak Suspects report identifies
> \{{org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics}} as
> *Problem Suspect 1*:
> {code}
> OperatorJosdkMetrics retained heap: 14,784,328 bytes (14.10 MiB)
> Share of post-GC live heap: 19.35%
> resourceMetricGroups retained heap: 11,742,264 bytes
> counters retained heap: 1,907,984 bytes
> {code}
> At the time of the dump, the adapter maps contained:
> {code}
> resourceMetricGroups: 1,801 entries
> resourceNsMetricGroups: 1,801 entries
> counters: 13,554 entries
> histograms: 6 entries
> {code}
> The \{{ResourceID}} keys retained in \{{resourceMetricGroups}} were compared
> with the Flink custom resources that still existed in the corresponding
> Kubernetes namespaces:
> {code}
> Resource metric groups retained: 1,801
> Resources still present: 151
> Deleted resources still retained: 1,650
> {code}
> Of the 1,650 stale resource entries, 1,647 were \{{FlinkStateSnapshot}}
> resources and 3 were \{{FlinkDeployment}} resources. They were associated
> with 12,745 retained JOSDK counter entries.
> The stale resource metric groups alone retained approximately 10.9 MiB. A
> conservative attribution that also includes their namespace groups, counter
> keys and wrappers, metric-name strings retained by the reporter, counter
> objects, and map-entry overhead totals approximately *16.7 MiB*, or *22.9% of
> the post-GC live Java heap* in this reproduction. This 16.7 MiB value is
> derived from MAT OQL results combined with the live Kubernetes resource-name
> comparison; it is not a single built-in MAT suspect value.
> The SLF4J reporter retained 4,206,696 bytes. It contained 13,607 counters in
> total, of which 13,554 (99.6%) were JOSDK counters. Therefore, reporter
> cycles become increasingly expensive as stale resource metrics accumulate.
> h2. Root cause
> {\{OperatorJosdkMetrics}} keeps resource-specific metric groups and counters
> in long-lived maps. Observing a CR deletion does not unregister those metrics
> from the adapter and the Flink metric registry/reporter. Consequently, a
> subsequent reporter cycle continues to enumerate the deleted resource, even
> though its \{{Resource.Event.DELETED.Count}} metric confirms that the
> deletion event was received.
> Periodic savepoints make this growth unbounded because every
> \{{FlinkStateSnapshot}} has a unique timestamped name. Each deleted snapshot
> can therefore leave another permanently registered set of resource-specific
> metrics behind. Retries and failed snapshot cleanup increase the number of
> counters per resource, but they are accelerators rather than the fundamental
> leak.
> {code}
> Periodic FlinkStateSnapshot with a unique name
> -> JOSDK resource metrics registered
> -> snapshot CR deleted
> -> DELETED event observed
> -> resource metrics remain registered
> -> SLF4J reporter keeps enumerating historical resources
> -> retained heap and report-generation allocations grow for the Operator
> Pod lifetime
> -> GC pressure and eventual Java heap space OOM
> {code}
> h2. Impact
> This is not limited to the five deleted resources shown in the log
> reproduction. In the reproduced heap, 1,650 already-deleted resources were
> still retained. The retained state grows with the number of unique resources
> processed during the Operator Pod lifetime and is therefore unbounded for
> workloads that continuously create timestamped periodic-savepoint resources.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
