Hello, Do I understand correctly that you are using native Kubernetes deployment in application mode; and the issue *only* happens if you set kubernetes-jobmanager-replicas [1] to a value greater than 1?
Does it happen during deployment or at some point while running the job? Could you share Flink and Kubernetes versions and HA configuration [2]? (I'm assuming you're using Kubernetes for HA, not ZK). [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/config/#kubernetes-jobmanager-replicas [2] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/ha/kubernetes_ha/ Regards, Roman On Fri, Aug 27, 2021 at 2:31 PM mejri houssem <mejrihousse...@gmail.com> wrote: > > hello i am deploying a flink application cluster with kubernetes HA mode, but > i am facing this recurrent problem and i didn't know how to solve it. > > Any help would be appreciated. > > > > this of the jobManager: > > {"@timestamp":"2021-08-27T14:19:42.447+02:00","@version":"1","message":"Exception > occurred while renewing lock: Unable to update > ConfigMapLock","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-4092-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.LockException: > Unable to update ConfigMapLock\n\tat > io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:108)\n\tat > > io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:156)\n\tat > > io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renew(LeaderElector.java:120)\n\tat > > io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$null$1(LeaderElector.java:104)\n\tat > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat > java.util.concurrent.FutureTask.run(FutureT > ask.java:266)\n\tat > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)\n\tat > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)\n\tat > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat > java.lang.Thread.run(Thread.java:748)\nCaused by: > io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: > PUT at: > https://172.31.64.1/api/v1/namespaces/flink-pushavoo-flink-rec/configmaps/elifibre-00000000000000000000000000000000-jobmanager-leader. > Message: Operation cannot be fulfilled on configmaps > \"elifibre-00000000000000000000000000000000-jobmanager-leader\": the object > has been modified; please apply your changes to the latest version and try > again. Received status: Status(apiVersion=v1, code=409, > details=StatusDetails(causes=[], gro > up=null, kind=configmaps, > name=elifibre-00000000000000000000000000000000-jobmanager-leader, > retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, > message=Operation cannot be fulfilled on configmaps > \"elifibre-00000000000000000000000000000000-jobmanager-leader\": the object > has been modified; please apply your changes to the latest version and try > again, metadata=ListMeta(_continue=null, remainingItemCount=null, > resourceVersion=null, selfLink=null, additionalProperties={}), > reason=Conflict, status=Failure, additionalProperties={}).\n\tat > io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)\n\tat > > io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)\n\tat > > io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)\n\tat > > io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)\n\tat > io.fabric8.kubernet > > es.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)\n\tat > > io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)\n\tat > > io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)\n\tat > > io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)\n\tat > > io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:26)\n\tat > > io.fabric8.kubernetes.api.model.DoneableConfigMap.done(DoneableConfigMap.java:5)\n\tat > > io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)\n\tat > > io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)\n\tat > > io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ConfigMapLock.update(ConfigMapLock.java:106)\n\t... > 10 common frames omitted\n"} >
