Hi Yun, I'm currently specifying that specific RPC address in my kubernetes charts for conveniene, should I be generating a new one for every deployment?
And yes, I am deleting the pods using those commands, I'm just noticing that the task-manager termination process is short circuited by the registration timeout check, so that instead of terminating quickly, the task-manger would wait for 5 minutes to timeout before terminating. I'm expecting it to just terminate without doing that registration timeout, is there a way to configure that? Thanks, Li On Thu, Jan 30, 2020 at 8:53 AM Yun Tang <myas...@live.com> wrote: > Hi Li > > Why you still use ’job-manager' as thejobmanager.rpc.address for the > second new cluster? If you use another rpc address, previous task managers > would not try to register with old one. > > Take flink documentation [1] for k8s as example. You can list/delete all > pods like: > > kubectl get/delete pods -l app=flink > > > By the way, the default registration timeout is 5min [2], those > taskmanager could not register to the JM will suicide after 5 minutes. > > [1] > https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html#session-cluster-resource-definitions > [2] > https://github.com/apache/flink/blob/7e1a0f446e018681cb537dd936ae54388b5a7523/flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java#L158 > > Best > Yun Tang > > ------------------------------ > *From:* Li Peng <li.p...@doordash.com> > *Sent:* Thursday, January 30, 2020 9:24 > *To:* user <user@flink.apache.org> > *Subject:* Task-manager kubernetes pods take a long time to terminate > > Hey folks, I'm deploying a Flink cluster via kubernetes, and starting each > task manager with taskmanager.sh. I noticed that when I tell kubectl to > delete the deployment, the job-manager pod usually terminates very quickly, > but any task-manager that doesn't get terminated before the job-manager, > usually gets stuck in this loop: > > 2020-01-29 09:18:47,867 INFO > org.apache.flink.runtime.taskexecutor.TaskExecutor - Could not > resolve ResourceManager address > akka.tcp://flink@job-manager:6123/user/resourcemanager, > retrying in 10000 ms: Could not connect to rpc endpoint under address > akka.tcp://flink@job-manager:6123/user/resourcemanager > > It then does this for about 10 minutes(?), and then shuts down. If I'm > deploying a new cluster, this pod will try to register itself with the new > job manager before terminating lter. This isn't a troubling issue as far as > I can tell, but I find it annoying that I sometimes have to force delete > the pods. > > Any easy ways to just have the task managers terminate gracefully and > quickly? > > Thanks, > Li >
