[jira] [Updated] (FLINK-12382) HA + ResourceManager exception: Fencing token not set
[ https://issues.apache.org/jira/browse/FLINK-12382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henrik updated FLINK-12382: --- Description: I'm testing zetcd + session jobs in k8s, and testing what happens when I kill both the job-cluster and task-manager at the same time, but maintain ZK/zetcd up and running. Then I get this stacktrace, that's completely non-actionable for me, and also resolves itself. I expect a number of retries, and if this exception is part of the protocol signalling to retry, then it should not be printed as a log entry. This might be related to an older bug: [https://jira.apache.org/jira/browse/FLINK-7734] {code:java} [tm] 2019-05-01 11:32:01,641 ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at ResourceManager failed due to an error [tm] java.util.concurrent.CompletionException: org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token not set: Ignoring message RemoteFencedMessage(b8fb92460bdd5c250de1415e6cf04d4e, RemoteRpcInvocation(registerTaskExecutor(String, ResourceID, int, HardwareDescription, Time))) sent to akka.tcp://flink@analytics-job:6123/user/resourcemanager because the fencing token is null. [tm] at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) [tm] at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) [tm] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) [tm] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) [tm] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [tm] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) [tm] at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:815) [tm] at akka.dispatch.OnComplete.internal(Future.scala:258) [tm] at akka.dispatch.OnComplete.internal(Future.scala:256) [tm] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186) [tm] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183) [tm] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36) [tm] at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:74) [tm] at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44) [tm] at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252) [tm] at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:534) [tm] at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:97) [tm] at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:982) [tm] at akka.actor.Actor$class.aroundReceive(Actor.scala:502) [tm] at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:446) [tm] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) [tm] at akka.actor.ActorCell.invoke(ActorCell.scala:495) [tm] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [tm] at akka.dispatch.Mailbox.run(Mailbox.scala:224) [tm] at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [tm] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [tm] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [tm] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [tm] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [tm] Caused by: org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token not set: Ignoring message RemoteFencedMessage(b8fb92460bdd5c250de1415e6cf04d4e, RemoteRpcInvocation(registerTaskExecutor(String, ResourceID, int, HardwareDescription, Time))) sent to akka.tcp://flink@analytics-job:6123/user/resourcemanager because the fencing token is null. [tm] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:63) [tm] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:147) [tm] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40) [tm] at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165) [tm] at akka.actor.Actor$class.aroundReceive(Actor.scala:502) [tm] at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95) [tm] ... 9 more [tm] 2019-05-01 11:32:01,650 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Pausing and re-attempting registration in 1 ms [tm] 2019-05-01 11:32:03,070 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - The heartbeat of JobManager with id 3642aa576f132fecd6811ae0d314c2b5 timed out. [tm] 2019-05-01 11:32:03,070 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Close Jo
[jira] [Updated] (FLINK-12382) HA + ResourceManager exception: Fencing token not set
[ https://issues.apache.org/jira/browse/FLINK-12382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henrik updated FLINK-12382: --- Description: I'm testing zetcd + session jobs in k8s, and testing what happens when I kill both the job-cluster and task-manager at the same time, but maintain ZK/zetcd up and running. Then I get this stacktrace, that's completely non-actionable for me, and also resolves itself. I expect a number of retries, and if this exception is part of the protocol signalling to retry, then it should not be printed as a log entry. This might be related to an older bug: [https://jira.apache.org/jira/browse/FLINK-7734] {code:java} [tm] 2019-05-01 11:32:01,641 ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at ResourceManager failed due to an error [tm] java.util.concurrent.CompletionException: org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token not set: Ignoring message RemoteFencedMessage(b8fb92460bdd5c250de1415e6cf04d4e, RemoteRpcInvocation(registerTaskExecutor(String, ResourceID, int, HardwareDescription, Time))) sent to akka.tcp://flink@analytics-job:6123/user/resourcemanager because the fencing token is null. [tm] at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) [tm] at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) [tm] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) [tm] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) [tm] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [tm] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) [tm] at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:815) [tm] at akka.dispatch.OnComplete.internal(Future.scala:258) [tm] at akka.dispatch.OnComplete.internal(Future.scala:256) [tm] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186) [tm] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183) [tm] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36) [tm] at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:74) [tm] at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44) [tm] at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252) [tm] at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:534) [tm] at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:97) [tm] at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:982) [tm] at akka.actor.Actor$class.aroundReceive(Actor.scala:502) [tm] at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:446) [tm] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) [tm] at akka.actor.ActorCell.invoke(ActorCell.scala:495) [tm] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [tm] at akka.dispatch.Mailbox.run(Mailbox.scala:224) [tm] at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [tm] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [tm] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [tm] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [tm] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [tm] Caused by: org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token not set: Ignoring message RemoteFencedMessage(b8fb92460bdd5c250de1415e6cf04d4e, RemoteRpcInvocation(registerTaskExecutor(String, ResourceID, int, HardwareDescription, Time))) sent to akka.tcp://flink@analytics-job:6123/user/resourcemanager because the fencing token is null. [tm] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:63) [tm] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:147) [tm] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40) [tm] at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165) [tm] at akka.actor.Actor$class.aroundReceive(Actor.scala:502) [tm] at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95) [tm] ... 9 more [tm] 2019-05-01 11:32:01,650 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Pausing and re-attempting registration in 1 ms [tm] 2019-05-01 11:32:03,070 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - The heartbeat of JobManager with id 3642aa576f132fecd6811ae0d314c2b5 timed out. [tm] 2019-05-01 11:32:03,070 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Close Jo
[jira] [Updated] (FLINK-12382) HA + ResourceManager exception: Fencing token not set
[ https://issues.apache.org/jira/browse/FLINK-12382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaël Renoux updated FLINK-12382: Attachment: jobmanager_stdout > HA + ResourceManager exception: Fencing token not set > - > > Key: FLINK-12382 > URL: https://issues.apache.org/jira/browse/FLINK-12382 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.8.0 > Environment: Same all all previous bugs filed by myself, today, but > this time with HA with zetcd. >Reporter: Henrik >Priority: Major > Attachments: jobmanager_log, jobmanager_stdout > > > I'm testing zetcd + session jobs in k8s, and testing what happens when I kill > both the job-cluster and task-manager at the same time, but maintain ZK/zetcd > up and running. > Then I get this stacktrace, that's completely non-actionable for me, and also > resolves itself. I expect a number of retries, and if this exception is part > of the protocol signalling to retry, then it should not be printed as a log > entry. > This might be related to an older bug: > [https://jira.apache.org/jira/browse/FLINK-7734] > {code:java} > [tm] 2019-05-01 11:32:01,641 ERROR > org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration > at ResourceManager failed due to an error > [tm] java.util.concurrent.CompletionException: > org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token > not set: Ignoring message > RemoteFencedMessage(b8fb92460bdd5c250de1415e6cf04d4e, > RemoteRpcInvocation(registerTaskExecutor(String, ResourceID, int, > HardwareDescription, Time))) sent to > akka.tcp://flink@analytics-job:6123/user/resourcemanager because the fencing > token is null. > [tm] at > java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) > [tm] at > java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) > [tm] at > java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) > [tm] at > java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) > [tm] at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > [tm] at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) > [tm] at > org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:815) > [tm] at akka.dispatch.OnComplete.internal(Future.scala:258) > [tm] at akka.dispatch.OnComplete.internal(Future.scala:256) > [tm] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186) > [tm] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183) > [tm] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36) > [tm] at > org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:74) > [tm] at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44) > [tm] at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252) > [tm] at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:534) > [tm] at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:97) > [tm] at > akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:982) > [tm] at akka.actor.Actor$class.aroundReceive(Actor.scala:502) > [tm] at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:446) > [tm] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) > [tm] at akka.actor.ActorCell.invoke(ActorCell.scala:495) > [tm] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) > [tm] at akka.dispatch.Mailbox.run(Mailbox.scala:224) > [tm] at akka.dispatch.Mailbox.exec(Mailbox.scala:234) > [tm] at > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > [tm] at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > [tm] at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > [tm] at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > [tm] Caused by: > org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token > not set: Ignoring message > RemoteFencedMessage(b8fb92460bdd5c250de1415e6cf04d4e, > RemoteRpcInvocation(registerTaskExecutor(String, ResourceID, int, > HardwareDescription, Time))) sent to > akka.tcp://flink@analytics-job:6123/user/resourcemanager because the fencing > token is null. > [tm] at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:63) > [tm] at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:147) > [tm] at > org.apache.fl
[jira] [Updated] (FLINK-12382) HA + ResourceManager exception: Fencing token not set
[ https://issues.apache.org/jira/browse/FLINK-12382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaël Renoux updated FLINK-12382: Attachment: jobmanager_log > HA + ResourceManager exception: Fencing token not set > - > > Key: FLINK-12382 > URL: https://issues.apache.org/jira/browse/FLINK-12382 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.8.0 > Environment: Same all all previous bugs filed by myself, today, but > this time with HA with zetcd. >Reporter: Henrik >Priority: Major > Attachments: jobmanager_log, jobmanager_stdout > > > I'm testing zetcd + session jobs in k8s, and testing what happens when I kill > both the job-cluster and task-manager at the same time, but maintain ZK/zetcd > up and running. > Then I get this stacktrace, that's completely non-actionable for me, and also > resolves itself. I expect a number of retries, and if this exception is part > of the protocol signalling to retry, then it should not be printed as a log > entry. > This might be related to an older bug: > [https://jira.apache.org/jira/browse/FLINK-7734] > {code:java} > [tm] 2019-05-01 11:32:01,641 ERROR > org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration > at ResourceManager failed due to an error > [tm] java.util.concurrent.CompletionException: > org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token > not set: Ignoring message > RemoteFencedMessage(b8fb92460bdd5c250de1415e6cf04d4e, > RemoteRpcInvocation(registerTaskExecutor(String, ResourceID, int, > HardwareDescription, Time))) sent to > akka.tcp://flink@analytics-job:6123/user/resourcemanager because the fencing > token is null. > [tm] at > java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) > [tm] at > java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) > [tm] at > java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) > [tm] at > java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) > [tm] at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > [tm] at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) > [tm] at > org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:815) > [tm] at akka.dispatch.OnComplete.internal(Future.scala:258) > [tm] at akka.dispatch.OnComplete.internal(Future.scala:256) > [tm] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186) > [tm] at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183) > [tm] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36) > [tm] at > org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:74) > [tm] at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44) > [tm] at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252) > [tm] at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:534) > [tm] at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:97) > [tm] at > akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:982) > [tm] at akka.actor.Actor$class.aroundReceive(Actor.scala:502) > [tm] at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:446) > [tm] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) > [tm] at akka.actor.ActorCell.invoke(ActorCell.scala:495) > [tm] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) > [tm] at akka.dispatch.Mailbox.run(Mailbox.scala:224) > [tm] at akka.dispatch.Mailbox.exec(Mailbox.scala:234) > [tm] at > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > [tm] at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > [tm] at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > [tm] at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > [tm] Caused by: > org.apache.flink.runtime.rpc.exceptions.FencingTokenException: Fencing token > not set: Ignoring message > RemoteFencedMessage(b8fb92460bdd5c250de1415e6cf04d4e, > RemoteRpcInvocation(registerTaskExecutor(String, ResourceID, int, > HardwareDescription, Time))) sent to > akka.tcp://flink@analytics-job:6123/user/resourcemanager because the fencing > token is null. > [tm] at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:63) > [tm] at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:147) > [tm] at > org.apache.flink