[
https://issues.apache.org/jira/browse/FLINK-40372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liting liu updated FLINK-40372:
-------------------------------
Description:
our flink opeartor encounterd OOM after running for months. Our flink jobs
enable the {{periodic}} savepoint.
There are some detail logs can be reporduced, just create a job enable the
periodic savepoint then monitor the log:
only seven {{jm-ha-f6ndtm-savepoint-periodic-*}} resources remained in the
cluster. However, the next metrics report at {{11:18:12 CST}} still emitted
metrics for twelve resource names. The five additional resources had already
been deleted.
h3. Seven resources still present
Raw {{kubectl get}} output:
{{jm-ha-f6ndtm-savepoint-periodic-1786589833663 2026-08-13T02:57:13Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589864994 2026-08-13T02:57:45Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589927560 2026-08-13T02:58:47Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589943145 2026-08-13T02:59:03Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589958617 2026-08-13T02:59:18Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589974113 2026-08-13T02:59:34Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589983358 <none> ABANDONED}}
h3. Five additional resources had already been deleted
Directly querying these resources returned {{{}NotFound{}}}:
{{Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589817875" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589849241" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589880549" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589896162" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589911865" not found}}
h3. The next reporter cycle still emitted all twelve resources
The Operator logs use UTC. Therefore, {{03:18:12}} corresponds to {{{}11:18:12
CST{}}}.
Raw reporter header:
{{2026-08-13 03:18:12,173 o.a.f.m.s.Slf4jReporter [INFO ]}}
Resource metrics emitted during that report:
{{...jm-ha-f6ndtm-savepoint-periodic-1786589911865.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589817875.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589880549.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589974113.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589927560.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589896162.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589833663.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589943145.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589983358.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
2
...jm-ha-f6ndtm-savepoint-periodic-1786589958617.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589849241.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589864994.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3}}
The following five entries belong to deleted resources:
{{1786589817875
1786589849241
1786589880549
1786589896162
1786589911865}}
The runtime sequence is therefore:
{{11:17 CST Only 7 matching resources existed; the other 5 returned NotFound
11:18:12 CST The next reporter cycle still emitted metrics for all 12
resources}}
Additionally, the retained metrics for the deleted resources include:
{{FlinkStateSnapshot.JOSDK.Resource.Event.DELETED.Count: 1}}
This shows that the deletion events had been observed, but the corresponding
resource-specific metrics continued to be emitted afterward.
A concise description for the issue:
Five {{jm-ha-f6ndtm-savepoint-periodic-*}} resources no longer existed in
Kubernetes and returned {{{}NotFound{}}}. However, the subsequent SLF4J metrics
report continued to emit their resource-specific JOSDK metrics. The reporter
emitted metrics for 12 resource names while only 7 matching resources remained
in the cluster. The retained metrics also contained
{{{}Resource.Event.DELETED.Count: 1{}}}, confirming that the deletion events
had already been observed.
That unreleased resource crash the JVM heap as long as the job enable the
periodic savepoint and operator run long enough.
was:
our flink opeartor encounterd OOM after running for months. Our flink jobs
enable the {{periodic}} savepoint.
There are some detail logs can be reporduced:
only seven {{jm-ha-f6ndtm-savepoint-periodic-*}} resources remained in the
cluster. However, the next metrics report at {{11:18:12 CST}} still emitted
metrics for twelve resource names. The five additional resources had already
been deleted.
h3. Seven resources still present
Raw {{kubectl get}} output:
{{jm-ha-f6ndtm-savepoint-periodic-1786589833663 2026-08-13T02:57:13Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589864994 2026-08-13T02:57:45Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589927560 2026-08-13T02:58:47Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589943145 2026-08-13T02:59:03Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589958617 2026-08-13T02:59:18Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589974113 2026-08-13T02:59:34Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589983358 <none> ABANDONED}}
h3. Five additional resources had already been deleted
Directly querying these resources returned {{{}NotFound{}}}:
{{Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589817875" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589849241" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589880549" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589896162" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589911865" not found}}
h3. The next reporter cycle still emitted all twelve resources
The Operator logs use UTC. Therefore, {{03:18:12}} corresponds to {{{}11:18:12
CST{}}}.
Raw reporter header:
{{2026-08-13 03:18:12,173 o.a.f.m.s.Slf4jReporter [INFO ]}}
Resource metrics emitted during that report:
{{...jm-ha-f6ndtm-savepoint-periodic-1786589911865.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589817875.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589880549.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589974113.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589927560.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589896162.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589833663.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589943145.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589983358.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
2
...jm-ha-f6ndtm-savepoint-periodic-1786589958617.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589849241.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589864994.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3}}
The following five entries belong to deleted resources:
{{1786589817875
1786589849241
1786589880549
1786589896162
1786589911865}}
The runtime sequence is therefore:
{{11:17 CST Only 7 matching resources existed; the other 5 returned NotFound
11:18:12 CST The next reporter cycle still emitted metrics for all 12
resources}}
Additionally, the retained metrics for the deleted resources include:
{{FlinkStateSnapshot.JOSDK.Resource.Event.DELETED.Count: 1}}
This shows that the deletion events had been observed, but the corresponding
resource-specific metrics continued to be emitted afterward.
A concise description for the issue:
{quote}Five {{jm-ha-f6ndtm-savepoint-periodic-*}} resources no longer existed
in Kubernetes and returned {{{}NotFound{}}}. However, the subsequent SLF4J
metrics report continued to emit their resource-specific JOSDK metrics. The
reporter emitted metrics for 12 resource names while only 7 matching resources
remained in the cluster. The retained metrics also contained
{{{}Resource.Event.DELETED.Count: 1{}}}, confirming that the deletion events
had already been observed.
{quote}
> [Kubernetes Operator] JOSDK per-resource metrics are not unregistered after
> resource cleanup
> --------------------------------------------------------------------------------------------
>
> Key: FLINK-40372
> URL: https://issues.apache.org/jira/browse/FLINK-40372
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Affects Versions: 1.13
> Reporter: liting liu
> Priority: Critical
> Labels: pull-request-available
>
> our flink opeartor encounterd OOM after running for months. Our flink jobs
> enable the {{periodic}} savepoint.
> There are some detail logs can be reporduced, just create a job enable the
> periodic savepoint then monitor the log:
> only seven {{jm-ha-f6ndtm-savepoint-periodic-*}} resources remained in the
> cluster. However, the next metrics report at {{11:18:12 CST}} still emitted
> metrics for twelve resource names. The five additional resources had already
> been deleted.
> h3. Seven resources still present
> Raw {{kubectl get}} output:
>
> {{jm-ha-f6ndtm-savepoint-periodic-1786589833663 2026-08-13T02:57:13Z
> IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589864994 2026-08-13T02:57:45Z IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589927560 2026-08-13T02:58:47Z IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589943145 2026-08-13T02:59:03Z IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589958617 2026-08-13T02:59:18Z IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589974113 2026-08-13T02:59:34Z IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589983358 <none> ABANDONED}}
> h3. Five additional resources had already been deleted
> Directly querying these resources returned {{{}NotFound{}}}:
>
> {{Error from server (NotFound): flinkstatesnapshots.flink.apache.org
> "jm-ha-f6ndtm-savepoint-periodic-1786589817875" not found
> Error from server (NotFound): flinkstatesnapshots.flink.apache.org
> "jm-ha-f6ndtm-savepoint-periodic-1786589849241" not found
> Error from server (NotFound): flinkstatesnapshots.flink.apache.org
> "jm-ha-f6ndtm-savepoint-periodic-1786589880549" not found
> Error from server (NotFound): flinkstatesnapshots.flink.apache.org
> "jm-ha-f6ndtm-savepoint-periodic-1786589896162" not found
> Error from server (NotFound): flinkstatesnapshots.flink.apache.org
> "jm-ha-f6ndtm-savepoint-periodic-1786589911865" not found}}
> h3. The next reporter cycle still emitted all twelve resources
> The Operator logs use UTC. Therefore, {{03:18:12}} corresponds to
> {{{}11:18:12 CST{}}}.
> Raw reporter header:
>
> {{2026-08-13 03:18:12,173 o.a.f.m.s.Slf4jReporter [INFO ]}}
> Resource metrics emitted during that report:
>
> {{...jm-ha-f6ndtm-savepoint-periodic-1786589911865.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 4
> ...jm-ha-f6ndtm-savepoint-periodic-1786589817875.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 4
> ...jm-ha-f6ndtm-savepoint-periodic-1786589880549.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 4
> ...jm-ha-f6ndtm-savepoint-periodic-1786589974113.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3
> ...jm-ha-f6ndtm-savepoint-periodic-1786589927560.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3
> ...jm-ha-f6ndtm-savepoint-periodic-1786589896162.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 4
> ...jm-ha-f6ndtm-savepoint-periodic-1786589833663.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3
> ...jm-ha-f6ndtm-savepoint-periodic-1786589943145.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3
> ...jm-ha-f6ndtm-savepoint-periodic-1786589983358.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 2
> ...jm-ha-f6ndtm-savepoint-periodic-1786589958617.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3
> ...jm-ha-f6ndtm-savepoint-periodic-1786589849241.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 4
> ...jm-ha-f6ndtm-savepoint-periodic-1786589864994.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3}}
> The following five entries belong to deleted resources:
>
> {{1786589817875
> 1786589849241
> 1786589880549
> 1786589896162
> 1786589911865}}
> The runtime sequence is therefore:
>
> {{11:17 CST Only 7 matching resources existed; the other 5 returned NotFound
> 11:18:12 CST The next reporter cycle still emitted metrics for all 12
> resources}}
> Additionally, the retained metrics for the deleted resources include:
>
> {{FlinkStateSnapshot.JOSDK.Resource.Event.DELETED.Count: 1}}
> This shows that the deletion events had been observed, but the corresponding
> resource-specific metrics continued to be emitted afterward.
> A concise description for the issue:
> Five {{jm-ha-f6ndtm-savepoint-periodic-*}} resources no longer existed in
> Kubernetes and returned {{{}NotFound{}}}. However, the subsequent SLF4J
> metrics report continued to emit their resource-specific JOSDK metrics. The
> reporter emitted metrics for 12 resource names while only 7 matching
> resources remained in the cluster. The retained metrics also contained
> {{{}Resource.Event.DELETED.Count: 1{}}}, confirming that the deletion events
> had already been observed.
> That unreleased resource crash the JVM heap as long as the job enable the
> periodic savepoint and operator run long enough.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
