[
https://issues.apache.org/jira/browse/FLINK-40330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaitanya Waikar updated FLINK-40330:
-------------------------------------
Description:
h2. Description
We observed a Kubernetes HA cluster in which old per-job leader-information
entries accumulated in the cluster-level HA ConfigMap.
The affected ConfigMap contains *255* keys with this format:
{code:java}
org.apache.flink.k8s.leader.job-<job-id>{code}
Many of the corresponding JobIDs are no longer active, but their
leader-information entries remain in the ConfigMap. The same ConfigMap also
stores other HA metadata, so this accumulation causes it to grow continuously
toward Kubernetes’ {*}1 MiB ConfigMap limit{*}.
Source inspection shows that Flink normally attempts to remove a job’s leader
entry when the corresponding {{JobManagerRunner}} closes. However, this
deletion is conditional on the process still holding Kubernetes leadership. In
addition, the job-level HA cleanup method deletes the job-specific ConfigMap
but does not perform a fallback deletion or reconciliation of the job’s leader
entry in the cluster ConfigMap.
This means that {*}abnormal termination{*}, {*}leadership loss during
cleanup{*}, or a failed ConfigMap update may leave an orphaned leader entry
without a later reconciliation mechanism.
h2. Observed behavior
The cluster ConfigMap contains entries such as:
{code:java}
org.apache.flink.k8s.leader.job-04836a890e3bdbb2eb3c44e0332725cb:
48cd444c-3df1-4963-91aa-0df8f0bcfdb5,pekko.tcp://flink@<pod-address>:6123/user/rpc/jobmanager_55
org.apache.flink.k8s.leader.job-048ea419dbc869d34139e6f277b0d1f0:
92edfece-ac04-4300-91e3-40fbb57b6875,pekko.tcp://flink@<pod-address>:6123/user/rpc/jobmanager_185{{}}
{{}}{}
{code}
The corresponding JobIDs are no longer returned by the Dispatcher’s active-job
list and are not expected to be recovered.
The entries remain in the ConfigMap after the jobs have disappeared from the
active cluster state.
The cluster regularly stops and resubmits jobs from savepoints, generating new
JobIDs. The cluster has also experienced JobManager process failures, including
an OOMKill. At present, we cannot attribute every stale entry to a single
cleanup path, but the production ConfigMap demonstrates that orphaned entries
persist across multiple JobManager lifetimes.
h2. Expected behavior
After a job reaches a globally terminal state and its resources are cleaned up,
the following leader-information key should eventually be absent:
{code:java}
org.apache.flink.k8s.leader.job-<terminated-job-id> {code}
If immediate removal is not possible because the process has lost leadership or
terminated unexpectedly, a later active leader should *reconcile and remove
entries* for jobs that are no longer active or recoverable.
At minimum, stale per-job leader entries should not accumulate indefinitely in
the shared cluster ConfigMap.
h2. Actual behavior
Some per-job leader entries survive job termination or JobManager failover and
remain indefinitely in the shared cluster ConfigMap.
There does not appear to be a general reconciliation mechanism that compares
persisted per-job leader keys with the set of active or recoverable jobs.
The ConfigMap therefore grows over time.
*Source Code Analysis*
We could not find a mechanism that periodically or during recovery:
# reads all {{org.apache.flink.k8s.leader.job-*}} entries;
# compares them with active or recoverable jobs;
# removes entries for jobs that no longer exist.
The Kubernetes leader-election driver can parse all leader entries, but only
reconciles information for currently registered contenders. An orphaned
component ID has no registered contender that would cause its data to be
corrected or removed.
h2. Suspected failure scenarios
The following scenarios appear capable of leaving orphaned keys.
# Abrupt JobManager process termination (due to OOMKill)
# Leadership loss during cleanup
h2. Operational impact
The cluster-level ConfigMap is used for:
* Kubernetes leader-election metadata;
* Dispatcher, ResourceManager and REST endpoint leader information;
* per-job JobMaster leader information;
* execution-plan state handles;
* potentially other cluster-level HA metadata.
Kubernetes limits ConfigMap data to approximately {*}1 MiB{*}.
Unbounded stale-entry accumulation can therefore eventually prevent new
ConfigMap entries or updates from succeeding. Once the ConfigMap approaches
that limit, operations that increase its size may fail with HTTP 422 validation
errors.
This can affect:
* new JobMaster leader publication;
* execution-plan persistence;
* other HA metadata writes;
* Dispatcher operation;
* recovery and failover reliability.
This is similar in impact to oversized HA ConfigMap incidents such as
FLINK-31135, although this report concerns a potentially different accumulation
mechanism.
We are not claiming that the current ConfigMap size will immediately cause a
cluster-wide outage. The concern is that the growth is unbounded and eventually
reaches a hard Kubernetes limit.
was:
h2. Description
We observed a Kubernetes HA cluster in which old per-job leader-information
entries accumulated in the cluster-level HA ConfigMap.
The affected ConfigMap contains *255* keys with this format:
Text
org.apache.flink.k8s.leader.job-<job-id>
Many of the corresponding JobIDs are no longer active, but their
leader-information entries remain in the ConfigMap. The same ConfigMap also
stores other HA metadata, so this accumulation causes it to grow continuously
toward Kubernetes’ {*}1 MiB ConfigMap limit{*}.
Source inspection shows that Flink normally attempts to remove a job’s leader
entry when the corresponding {{JobManagerRunner}} closes. However, this
deletion is conditional on the process still holding Kubernetes leadership. In
addition, the job-level HA cleanup method deletes the job-specific ConfigMap
but does not perform a fallback deletion or reconciliation of the job’s leader
entry in the cluster ConfigMap.
This means that {*}abnormal termination{*}, {*}leadership loss during
cleanup{*}, or a failed ConfigMap update may leave an orphaned leader entry
without a later reconciliation mechanism.
h2. Observed behavior
The cluster ConfigMap contains entries such as:
YAML
org.apache.flink.k8s.leader.job-04836a890e3bdbb2eb3c44e0332725cb:
48cd444c-3df1-4963-91aa-0df8f0bcfdb5,pekko.tcp://flink@<pod-address>:6123/user/rpc/jobmanager_55
org.apache.flink.k8s.leader.job-048ea419dbc869d34139e6f277b0d1f0:
92edfece-ac04-4300-91e3-40fbb57b6875,pekko.tcp://flink@<pod-address>:6123/user/rpc/jobmanager_185{{}}
{{}}
{{{}{}}}The corresponding JobIDs are no longer returned by the Dispatcher’s
active-job list and are not expected to be recovered.
The entries remain in the ConfigMap after the jobs have disappeared from the
active cluster state.
The cluster regularly stops and resubmits jobs from savepoints, generating new
JobIDs. The cluster has also experienced JobManager process failures, including
an OOMKill. At present, we cannot attribute every stale entry to a single
cleanup path, but the production ConfigMap demonstrates that orphaned entries
persist across multiple JobManager lifetimes.
h2. Expected behavior
After a job reaches a globally terminal state and its resources are cleaned up,
the following leader-information key should eventually be absent:
org.apache.flink.k8s.leader.job-<terminated-job-id>
If immediate removal is not possible because the process has lost leadership or
terminated unexpectedly, a later active leader should *reconcile and remove
entries* for jobs that are no longer active or recoverable.
At minimum, stale per-job leader entries should not accumulate indefinitely in
the shared cluster ConfigMap.
h2. Actual behavior
Some per-job leader entries survive job termination or JobManager failover and
remain indefinitely in the shared cluster ConfigMap.
There does not appear to be a general reconciliation mechanism that compares
persisted per-job leader keys with the set of active or recoverable jobs.
The ConfigMap therefore grows over time.
*Source Code Analysis*
We could not find a mechanism that periodically or during recovery:
# reads all {{org.apache.flink.k8s.leader.job-*}} entries;
# compares them with active or recoverable jobs;
# removes entries for jobs that no longer exist.
The Kubernetes leader-election driver can parse all leader entries, but only
reconciles information for currently registered contenders. An orphaned
component ID has no registered contender that would cause its data to be
corrected or removed.
h2. Suspected failure scenarios
The following scenarios appear capable of leaving orphaned keys.
# Abrupt JobManager process termination (due to OOMKill)
# Leadership loss during cleanup
h2.
h2. Operational impact
The cluster-level ConfigMap is used for:
* Kubernetes leader-election metadata;
* Dispatcher, ResourceManager and REST endpoint leader information;
* per-job JobMaster leader information;
* execution-plan state handles;
* potentially other cluster-level HA metadata.
Kubernetes limits ConfigMap data to approximately {*}1 MiB{*}.
Unbounded stale-entry accumulation can therefore eventually prevent new
ConfigMap entries or updates from succeeding. Once the ConfigMap approaches
that limit, operations that increase its size may fail with HTTP 422 validation
errors.
This can affect:
* new JobMaster leader publication;
* execution-plan persistence;
* other HA metadata writes;
* Dispatcher operation;
* recovery and failover reliability.
This is similar in impact to oversized HA ConfigMap incidents such as
[FLINK-31135|https://issues.apache.org/jira/browse/FLINK-31135], although this
report concerns a potentially different accumulation mechanism.
We are not claiming that the current ConfigMap size will immediately cause a
cluster-wide outage. The concern is that the growth is unbounded and eventually
reaches a hard Kubernetes limit.
> [Kubernetes HA] Orphaned per-job leader entries accumulate in the cluster
> ConfigMap after interrupted cleanup or leadership loss
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40330
> URL: https://issues.apache.org/jira/browse/FLINK-40330
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, Runtime / Coordination
> Affects Versions: 2.1.3
> Environment: * *Flink version:* 2.1.3
> * *Flink release commit:* {{6cda56b084d5c337b36d2f8ed464bc92093b0a34}}
> * *Release commit date:* June 8, 2026
> * *Deployment type:* Kubernetes session cluster with HA enabled
> * *HA backend:* Kubernetes
> * *Observation date:* July 30–31, 2026
> * *Cluster age at observation:* approximately four months
> * *Cluster-level HA ConfigMap size:* approximately 290 KB
> * *Per-job leader entries:* 255
> * *Execution-plan entries:* 204
> * *Kubernetes leader transitions:* 193
> Reporter: Chaitanya Waikar
> Priority: Minor
>
> h2. Description
> We observed a Kubernetes HA cluster in which old per-job leader-information
> entries accumulated in the cluster-level HA ConfigMap.
> The affected ConfigMap contains *255* keys with this format:
>
> {code:java}
> org.apache.flink.k8s.leader.job-<job-id>{code}
> Many of the corresponding JobIDs are no longer active, but their
> leader-information entries remain in the ConfigMap. The same ConfigMap also
> stores other HA metadata, so this accumulation causes it to grow continuously
> toward Kubernetes’ {*}1 MiB ConfigMap limit{*}.
> Source inspection shows that Flink normally attempts to remove a job’s leader
> entry when the corresponding {{JobManagerRunner}} closes. However, this
> deletion is conditional on the process still holding Kubernetes leadership.
> In addition, the job-level HA cleanup method deletes the job-specific
> ConfigMap but does not perform a fallback deletion or reconciliation of the
> job’s leader entry in the cluster ConfigMap.
>
> This means that {*}abnormal termination{*}, {*}leadership loss during
> cleanup{*}, or a failed ConfigMap update may leave an orphaned leader entry
> without a later reconciliation mechanism.
> h2. Observed behavior
> The cluster ConfigMap contains entries such as:
>
> {code:java}
> org.apache.flink.k8s.leader.job-04836a890e3bdbb2eb3c44e0332725cb:
>
> 48cd444c-3df1-4963-91aa-0df8f0bcfdb5,pekko.tcp://flink@<pod-address>:6123/user/rpc/jobmanager_55
> org.apache.flink.k8s.leader.job-048ea419dbc869d34139e6f277b0d1f0:
>
> 92edfece-ac04-4300-91e3-40fbb57b6875,pekko.tcp://flink@<pod-address>:6123/user/rpc/jobmanager_185{{}}
> {{}}{}
> {code}
>
> The corresponding JobIDs are no longer returned by the Dispatcher’s
> active-job list and are not expected to be recovered.
> The entries remain in the ConfigMap after the jobs have disappeared from the
> active cluster state.
> The cluster regularly stops and resubmits jobs from savepoints, generating
> new JobIDs. The cluster has also experienced JobManager process failures,
> including an OOMKill. At present, we cannot attribute every stale entry to a
> single cleanup path, but the production ConfigMap demonstrates that orphaned
> entries persist across multiple JobManager lifetimes.
> h2. Expected behavior
> After a job reaches a globally terminal state and its resources are cleaned
> up, the following leader-information key should eventually be absent:
>
> {code:java}
> org.apache.flink.k8s.leader.job-<terminated-job-id> {code}
> If immediate removal is not possible because the process has lost leadership
> or terminated unexpectedly, a later active leader should *reconcile and
> remove entries* for jobs that are no longer active or recoverable.
> At minimum, stale per-job leader entries should not accumulate indefinitely
> in the shared cluster ConfigMap.
> h2. Actual behavior
> Some per-job leader entries survive job termination or JobManager failover
> and remain indefinitely in the shared cluster ConfigMap.
> There does not appear to be a general reconciliation mechanism that compares
> persisted per-job leader keys with the set of active or recoverable jobs.
> The ConfigMap therefore grows over time.
>
> *Source Code Analysis*
> We could not find a mechanism that periodically or during recovery:
> # reads all {{org.apache.flink.k8s.leader.job-*}} entries;
> # compares them with active or recoverable jobs;
> # removes entries for jobs that no longer exist.
> The Kubernetes leader-election driver can parse all leader entries, but only
> reconciles information for currently registered contenders. An orphaned
> component ID has no registered contender that would cause its data to be
> corrected or removed.
> h2. Suspected failure scenarios
> The following scenarios appear capable of leaving orphaned keys.
> # Abrupt JobManager process termination (due to OOMKill)
> # Leadership loss during cleanup
> h2. Operational impact
> The cluster-level ConfigMap is used for:
> * Kubernetes leader-election metadata;
> * Dispatcher, ResourceManager and REST endpoint leader information;
> * per-job JobMaster leader information;
> * execution-plan state handles;
> * potentially other cluster-level HA metadata.
> Kubernetes limits ConfigMap data to approximately {*}1 MiB{*}.
> Unbounded stale-entry accumulation can therefore eventually prevent new
> ConfigMap entries or updates from succeeding. Once the ConfigMap approaches
> that limit, operations that increase its size may fail with HTTP 422
> validation errors.
> This can affect:
> * new JobMaster leader publication;
> * execution-plan persistence;
> * other HA metadata writes;
> * Dispatcher operation;
> * recovery and failover reliability.
> This is similar in impact to oversized HA ConfigMap incidents such as
> FLINK-31135, although this report concerns a potentially different
> accumulation mechanism.
> We are not claiming that the current ConfigMap size will immediately cause a
> cluster-wide outage. The concern is that the growth is unbounded and
> eventually reaches a hard Kubernetes limit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
