[
https://issues.apache.org/jira/browse/FLINK-40372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liting liu updated FLINK-40372:
-------------------------------
Description:
our flink opeartor encounterd OOM. Our flink jobs enable the {{periodic}}
savepoint.
There are some detail logs:
only seven {{jm-ha-f6ndtm-savepoint-periodic-*}} resources remained in the
cluster. However, the next metrics report at {{11:18:12 CST}} still emitted
metrics for twelve resource names. The five additional resources had already
been deleted.
h3. Seven resources still present
Raw {{kubectl get}} output:
{{jm-ha-f6ndtm-savepoint-periodic-1786589833663 2026-08-13T02:57:13Z
IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589864994 2026-08-13T02:57:45Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589927560 2026-08-13T02:58:47Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589943145 2026-08-13T02:59:03Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589958617 2026-08-13T02:59:18Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589974113 2026-08-13T02:59:34Z IN_PROGRESS
jm-ha-f6ndtm-savepoint-periodic-1786589983358 <none> ABANDONED}}
h3. Five additional resources had already been deleted
Directly querying these resources returned {{{}NotFound{}}}:
{{Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589817875" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589849241" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589880549" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589896162" not found
Error from server (NotFound): flinkstatesnapshots.flink.apache.org
"jm-ha-f6ndtm-savepoint-periodic-1786589911865" not found}}
h3. The next reporter cycle still emitted all twelve resources
The Operator logs use UTC. Therefore, {{03:18:12}} corresponds to {{{}11:18:12
CST{}}}.
Raw reporter header:
{{2026-08-13 03:18:12,173 o.a.f.m.s.Slf4jReporter [INFO ]}}
Resource metrics emitted during that report:
{{...jm-ha-f6ndtm-savepoint-periodic-1786589911865.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589817875.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589880549.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589974113.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589927560.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589896162.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589833663.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589943145.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589983358.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
2
...jm-ha-f6ndtm-savepoint-periodic-1786589958617.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3
...jm-ha-f6ndtm-savepoint-periodic-1786589849241.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
4
...jm-ha-f6ndtm-savepoint-periodic-1786589864994.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
3}}
The following five entries belong to deleted resources:
{{1786589817875
1786589849241
1786589880549
1786589896162
1786589911865}}
The runtime sequence is therefore:
{{11:17 CST Only 7 matching resources existed; the other 5 returned NotFound
11:18:12 CST The next reporter cycle still emitted metrics for all 12
resources}}
Additionally, the retained metrics for the deleted resources include:
{{FlinkStateSnapshot.JOSDK.Resource.Event.DELETED.Count: 1}}
This shows that the deletion events had been observed, but the corresponding
resource-specific metrics continued to be emitted afterward.
A concise description for the issue or PR:
{quote}Five {{jm-ha-f6ndtm-savepoint-periodic-*}} resources no longer existed
in Kubernetes and returned {{{}NotFound{}}}. However, the subsequent SLF4J
metrics report continued to emit their resource-specific JOSDK metrics. The
reporter emitted metrics for 12 resource names while only 7 matching resources
remained in the cluster. The retained metrics also contained
{{{}Resource.Event.DELETED.Count: 1{}}}, confirming that the deletion events
had already been observed.
{quote}
was:
Description
OperatorJosdkMetrics creates resource-scoped Flink metrics for every Kubernetes
custom resource reconciled by JOSDK. The metric scope includes the resource
namespace and name.
When resource cleanup completes, cleanupDone() only increments a cleanup
counter:
@Override
public void cleanupDone(ResourceID resourceID, Map<String, Object> metadata)
{ counter(getResourceMg(resourceID, metadata), RECONCILIATION,
"cleanup").inc(); }
It does not close or remove the resource metric group, nor remove the
associated entries from resourceNsMetricGroups, resourceMetricGroups, and
counters.
Furthermore, calling getResourceMg() from cleanupDone() can create a new metric
group during cleanup if one does not already exist.
Consequently, metrics associated with deleted resource names remain registered
for the lifetime of the Operator. Workloads that continuously create resources
with unique names, such as periodic FlinkStateSnapshot resources, cause the
number of metric groups, counters, and metric-scope strings to grow over time.
When the SLF4J metric reporter is enabled, every reporting cycle also traverses
and formats the accumulated metrics, increasing allocation pressure and
reporting cost.
This was observed with an Apache Flink Kubernetes Operator 1.13-based build.
The same cleanup behavior is still present on Apache main at commit:
fdfcc695daba16e4f4a8c9a6c08b7f470787de00
The Operator eventually reported heap exhaustion while executing the metric
reporter. The following lines are copied from the Operator logs with only ANSI
color codes removed:
2026-08-12 05:54:08,290 o.a.f.r.m.MetricRegistryImpl [WARN ] Error while
reporting metrics
java.lang.OutOfMemoryError: Java heap space
The source code demonstrates that resource-scoped metrics are not cleaned up.
> [Kubernetes Operator] JOSDK per-resource metrics are not unregistered after
> resource cleanup
> --------------------------------------------------------------------------------------------
>
> Key: FLINK-40372
> URL: https://issues.apache.org/jira/browse/FLINK-40372
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Affects Versions: 1.13
> Reporter: liting liu
> Priority: Critical
> Labels: pull-request-available
>
> our flink opeartor encounterd OOM. Our flink jobs enable the {{periodic}}
> savepoint.
> There are some detail logs:
> only seven {{jm-ha-f6ndtm-savepoint-periodic-*}} resources remained in the
> cluster. However, the next metrics report at {{11:18:12 CST}} still emitted
> metrics for twelve resource names. The five additional resources had already
> been deleted.
> h3. Seven resources still present
> Raw {{kubectl get}} output:
>
> {{jm-ha-f6ndtm-savepoint-periodic-1786589833663 2026-08-13T02:57:13Z
> IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589864994 2026-08-13T02:57:45Z
> IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589927560 2026-08-13T02:58:47Z
> IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589943145 2026-08-13T02:59:03Z
> IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589958617 2026-08-13T02:59:18Z
> IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589974113 2026-08-13T02:59:34Z
> IN_PROGRESS
> jm-ha-f6ndtm-savepoint-periodic-1786589983358 <none>
> ABANDONED}}
> h3. Five additional resources had already been deleted
> Directly querying these resources returned {{{}NotFound{}}}:
>
> {{Error from server (NotFound): flinkstatesnapshots.flink.apache.org
> "jm-ha-f6ndtm-savepoint-periodic-1786589817875" not found
> Error from server (NotFound): flinkstatesnapshots.flink.apache.org
> "jm-ha-f6ndtm-savepoint-periodic-1786589849241" not found
> Error from server (NotFound): flinkstatesnapshots.flink.apache.org
> "jm-ha-f6ndtm-savepoint-periodic-1786589880549" not found
> Error from server (NotFound): flinkstatesnapshots.flink.apache.org
> "jm-ha-f6ndtm-savepoint-periodic-1786589896162" not found
> Error from server (NotFound): flinkstatesnapshots.flink.apache.org
> "jm-ha-f6ndtm-savepoint-periodic-1786589911865" not found}}
> h3. The next reporter cycle still emitted all twelve resources
> The Operator logs use UTC. Therefore, {{03:18:12}} corresponds to
> {{{}11:18:12 CST{}}}.
> Raw reporter header:
>
> {{2026-08-13 03:18:12,173 o.a.f.m.s.Slf4jReporter [INFO ]}}
> Resource metrics emitted during that report:
>
> {{...jm-ha-f6ndtm-savepoint-periodic-1786589911865.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 4
> ...jm-ha-f6ndtm-savepoint-periodic-1786589817875.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 4
> ...jm-ha-f6ndtm-savepoint-periodic-1786589880549.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 4
> ...jm-ha-f6ndtm-savepoint-periodic-1786589974113.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3
> ...jm-ha-f6ndtm-savepoint-periodic-1786589927560.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3
> ...jm-ha-f6ndtm-savepoint-periodic-1786589896162.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 4
> ...jm-ha-f6ndtm-savepoint-periodic-1786589833663.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3
> ...jm-ha-f6ndtm-savepoint-periodic-1786589943145.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3
> ...jm-ha-f6ndtm-savepoint-periodic-1786589983358.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 2
> ...jm-ha-f6ndtm-savepoint-periodic-1786589958617.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3
> ...jm-ha-f6ndtm-savepoint-periodic-1786589849241.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 4
> ...jm-ha-f6ndtm-savepoint-periodic-1786589864994.FlinkStateSnapshot.JOSDK.Resource.Event.Count:
> 3}}
> The following five entries belong to deleted resources:
>
> {{1786589817875
> 1786589849241
> 1786589880549
> 1786589896162
> 1786589911865}}
> The runtime sequence is therefore:
>
> {{11:17 CST Only 7 matching resources existed; the other 5 returned
> NotFound
> 11:18:12 CST The next reporter cycle still emitted metrics for all 12
> resources}}
> Additionally, the retained metrics for the deleted resources include:
>
> {{FlinkStateSnapshot.JOSDK.Resource.Event.DELETED.Count: 1}}
> This shows that the deletion events had been observed, but the corresponding
> resource-specific metrics continued to be emitted afterward.
> A concise description for the issue or PR:
> {quote}Five {{jm-ha-f6ndtm-savepoint-periodic-*}} resources no longer existed
> in Kubernetes and returned {{{}NotFound{}}}. However, the subsequent SLF4J
> metrics report continued to emit their resource-specific JOSDK metrics. The
> reporter emitted metrics for 12 resource names while only 7 matching
> resources remained in the cluster. The retained metrics also contained
> {{{}Resource.Event.DELETED.Count: 1{}}}, confirming that the deletion events
> had already been observed.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
