[ https://issues.apache.org/jira/browse/FLINK-34007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805960#comment-17805960 ]
Matthias Pohl commented on FLINK-34007: --------------------------------------- I couldn't find anything related to ConfigMap cleanup being triggered during leadership loss in the Flink code (Flink's [KubernetesLeaderElector:82|https://github.com/apache/flink/blob/c5808b04fdce9ca0b705b6cc7a64666ab6426875/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesLeaderElector.java#L82] sets up the [callback for the leadership loss|https://github.com/apache/flink/blob/c5808b04fdce9ca0b705b6cc7a64666ab6426875/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesLeaderElector.java#L124] which is implemented by [KubernetesLeaderElectionDriver#LeaderCallbackHandlerImpl|https://github.com/apache/flink/blob/11259ef52466889157ef473f422ecced72bab169/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesLeaderElectionDriver.java#L212]). This behavior is on-par with the old (i.e. pre-FLINK-24038) version of the Flink 1.15 codebase (see 1.15 class [KubernetesLeaderElectionDriver:202|https://github.com/apache/flink/blob/bba7c417217be878fffb12efedeac50dec2a7459/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesLeaderElectionDriver.java#L202]). That's also not something I would expect. It should be handled by the LeaderElector, instead, because the LeaderElector knows the state of the leader election and can trigger a clean up before the new leader information is written to the ConfigMap entry. Flink shouldn't trigger a clean up because it doesn't know whether a new leader was already elected (in which case cleaning up the ConfigMap entry would result in losing the leadership information of the new leader). And the Flink process wouldn't be able to clean it up, anyway, because the process isn't the leader anymore. Or am I missing something here? On another note: I came across [this change|https://github.com/apache/flink/pull/22540/files#diff-0e859df42954459619211d2ec60957742b24c9fc6ce55526616fddc540f0f8ffL59-R60] in the FLINK-31997 PR (k8s client update to 6.6.2): We're changing the thread pool size from 1 to 3 essentially allowing the same internal LeaderElector being executed multiple times (because we trigger another {{KubernetesLeaderElector#run}} call when the leadership is revoked). The old version of the code use a single thread which would mean that the run call would get queued until the previous {{LeaderElector#run}} failed for whatever reason. That change sounds strange but shouldn't be the cause of this Jira issue because the change only went into 1.18 and we're experiencing this also in older versions of Flink. > Flink Job stuck in suspend state after losing leadership in HA Mode > ------------------------------------------------------------------- > > Key: FLINK-34007 > URL: https://issues.apache.org/jira/browse/FLINK-34007 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination > Affects Versions: 1.16.3, 1.17.2, 1.18.1, 1.18.2 > Reporter: Zhenqiu Huang > Priority: Major > Attachments: Debug.log, job-manager.log > > > The observation is that Job manager goes to suspend state with a failed > container not able to register itself to resource manager after timeout. > JM Log, see attached > -- This message was sent by Atlassian Jira (v8.20.10#820010)