[
https://issues.apache.org/jira/browse/FLINK-6063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15927836#comment-15927836
]
Dawid Wysakowicz commented on FLINK-6063:
-----------------------------------------
Hi Razvan,
I've had a look at your logs and I think the TMs do connect to the new leader
JM. See those lines:
```
2017-03-16 10:11:04,020 INFO org.apache.flink.runtime.taskmanager.TaskManager -
TaskManager akka://flink/user/taskmanager disconnects from JobManager
akka.tcp://[email protected]:45164/user/jobmanager: Old JobManager lost its
leadership.
2017-03-16 10:11:04,020 INFO org.apache.flink.runtime.taskmanager.TaskManager -
Disassociating from JobManager
2017-03-16 10:11:04,025 INFO org.apache.flink.runtime.blob.BlobCache - Shutting
down BlobCache
2017-03-16 10:11:04,042 INFO org.apache.flink.runtime.taskmanager.TaskManager -
Trying to register at JobManager akka.tcp://[email protected]:34987/user/jobmanager
(attempt 1, timeout: 500 milliseconds)
2017-03-16 10:11:04,174 INFO org.apache.flink.runtime.taskmanager.TaskManager -
Successful registration at JobManager
(akka.tcp://[email protected]:34987/user/jobmanager), starting network stack and
library cache.
2017-03-16 10:11:04,174 INFO org.apache.flink.runtime.taskmanager.TaskManager -
Determined BLOB server address to be /1.2.3.5:42030. Starting BLOB cache.
2017-03-16 10:11:04,175 INFO org.apache.flink.runtime.blob.BlobCache - Created
BLOB cache storage directory /tmp/blobStore-92bf7fe1-bab0-498c-90bf-6ec44ec6cb1e
```
The further logs with failed association to old remote system are due to Akka
system and have no impact on Flink cluster (if I am wrong [~till.rohrmann]
please correct me). Anyway after some time they stop retrying and all
communication happen between TMs and the new leader. Try to open webUI before
and after killing JM, you will see that you are redirected to proper JMs and
TMs will be connected correctly in both cases.
I believe everything is working fine, just maybe the logs could be a liitle bit
clearer.
> HA Configuration doesn't work with Flink 1.2
> --------------------------------------------
>
> Key: FLINK-6063
> URL: https://issues.apache.org/jira/browse/FLINK-6063
> Project: Flink
> Issue Type: Bug
> Components: JobManager
> Affects Versions: 1.2.0
> Reporter: Razvan
> Priority: Critical
>
> I have a setup with flink 1.2 cluster, made up of 3 JobManagers and 2
> TaskManagers. I start the Zookeeper Quorum from JobManager1, I get
> confirmation Zookeeper starts on the other 2 JobManagers then I start a Flink
> job on this JobManager1.
>
> The flink-conf.yaml is the same on all 5 VMs (also everything else related
> to flink because I copied the folder across all VMs as suggested in
> tutorials) this means jobmanager.rpc.address: points to JobManager1
> everywhere.
> If I turn off the VM running JobManager1 I would expect Zookeeper to say one
> of the remaining JobManagers is the leader and the TaskManagers should
> reconnect to it. Instead a new leader is elected but the slaves keep
> connecting to the old master
> 2017-03-15 10:28:28,655 INFO org.apache.flink.core.fs.FileSystem
> - Ensuring all FileSystem streams are closed for Async
> calls on Source: Custom Source -> Flat Map (1/1)
> 2017-03-15 10:28:38,534 WARN akka.remote.ReliableDeliverySupervisor
> - Association with remote system
> [akka.tcp://[email protected]:44779] has failed, address is now gated for [5000]
> ms. Reason: [Disassociated]
> 2017-03-15 10:28:46,606 WARN akka.remote.ReliableDeliverySupervisor
> - Association with remote system
> [akka.tcp://[email protected]:44779] has failed, address is now gated for [5000]
> ms. Reason: [Association failed with [akka.tcp://[email protected]:44779]] Caused
> by: [Connection refused: /1.2.3.4:44779]
> 2017-03-15 10:28:52,431 WARN akka.remote.ReliableDeliverySupervisor
> - Association with remote system
> [akka.tcp://[email protected]:44779] has failed, address is now gated for [5000]
> ms. Reason: [Association failed with [akka.tcp://[email protected]:44779]] Caused
> by: [Connection refused: /1.2.3.4:44779]
> 2017-03-15 10:29:02,435 WARN akka.remote.ReliableDeliverySupervisor
> - Association with remote system
> [akka.tcp://[email protected]:44779] has failed, address is now gated for [5000]
> ms. Reason: [Association failed with [akka.tcp://[email protected]:44779]] Caused
> by: [Connection refused: /1.2.3.4:44779]
> 2017-03-15 10:29:10,489 INFO
> org.apache.flink.runtime.taskmanager.TaskManager - TaskManager
> akka://flink/user/taskmanager disconnects from JobManager
> akka.tcp://[email protected]:44779/user/jobmanager: Old JobManager lost its
> leadership.
> 2017-03-15 10:29:10,490 INFO
> org.apache.flink.runtime.taskmanager.TaskManager - Cancelling
> all computations and discarding all cached data.
> 2017-03-15 10:29:10,491 INFO org.apache.flink.runtime.taskmanager.Task
> - Attempting to fail task externally Source: Custom Source
> -> Flat Map (1/1) (75fd495cc6acfd72fbe957e60e513223).
> 2017-03-15 10:29:10,491 INFO org.apache.flink.runtime.taskmanager.Task
> - Source: Custom Source -> Flat Map (1/1)
> (75fd495cc6acfd72fbe957e60e513223) switched from RUNNING to FAILED.
> java.lang.Exception: TaskManager akka://flink/user/taskmanager
> disconnects from JobManager akka.tcp://[email protected]:44779/user/jobmanager:
> Old JobManager lost its leadership.
> at
> org.apache.flink.runtime.taskmanager.TaskManager.handleJobManagerDisconnect(TaskManager.scala:1074)
> at
> org.apache.flink.runtime.taskmanager.TaskManager.org$apache$flink$runtime$taskmanager$TaskManager$$handleJobManagerLeaderAddress(TaskManager.scala:1426)
> at
> org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:286)
> at
> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
> at
> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
> at
> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
> at
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
> at
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> at
> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
> at
> org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:122)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
> at akka.dispatch.Mailbox.run(Mailbox.scala:220)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2017-03-15 10:29:10,512 INFO org.apache.flink.runtime.taskmanager.Task
> - Triggering cancellation of task code Source: Custom
> Source -> Flat Map (1/1) (75fd495cc6acfd72fbe957e60e513223).
> 2017-03-15 10:29:10,515 INFO org.apache.flink.runtime.taskmanager.Task
> - Attempting to fail task externally Flat Map (1/1)
> (dd555e0437867c3180a1ecaf0a9f4d04).
> 2017-03-15 10:29:10,515 INFO org.apache.flink.runtime.taskmanager.Task
> - Flat Map (1/1) (dd555e0437867c3180a1ecaf0a9f4d04)
> switched from RUNNING to FAILED.
> java.lang.Exception: TaskManager akka://flink/user/taskmanager
> disconnects from JobManager akka.tcp://[email protected]:44779/user/jobmanager:
> Old JobManager lost its leadership.
> at
> org.apache.flink.runtime.taskmanager.TaskManager.handleJobManagerDisconnect(TaskManager.scala:1074)
> at
> org.apache.flink.runtime.taskmanager.TaskManager.org$apache$flink$runtime$taskmanager$TaskManager$$handleJobManagerLeaderAddress(TaskManager.scala:1426)
> at
> org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:286)
> at
> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
> at
> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
> at
> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
> at
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
> at
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> at
> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
> at
> org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:122)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
> at akka.dispatch.Mailbox.run(Mailbox.scala:220)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2017-03-15 10:29:10,516 INFO org.apache.flink.runtime.taskmanager.Task
> - Triggering cancellation of task code Flat Map (1/1)
> (dd555e0437867c3180a1ecaf0a9f4d04).
> 2017-03-15 10:29:10,516 INFO
> org.apache.flink.runtime.taskmanager.TaskManager -
> Disassociating from JobManager
> 2017-03-15 10:29:10,525 INFO org.apache.flink.runtime.blob.BlobCache
> - Shutting down BlobCache
> 2017-03-15 10:29:10,542 WARN akka.remote.ReliableDeliverySupervisor
> - Association with remote system
> [akka.tcp://[email protected]:44779] has failed, address is now gated for [5000]
> ms. Reason: [Association failed with [akka.tcp://[email protected]:44779]] Caused
> by: [Connection refused: /1.2.3.4:44779]
> 2017-03-15 10:29:10,546 INFO org.apache.flink.runtime.taskmanager.Task
> - Freeing task resources for Source: Custom Source -> Flat
> Map (1/1) (75fd495cc6acfd72fbe957e60e513223).
> 2017-03-15 10:29:10,548 INFO org.apache.flink.runtime.taskmanager.Task
> - Freeing task resources for Flat Map (1/1)
> (dd555e0437867c3180a1ecaf0a9f4d04).
> 2017-03-15 10:29:10,551 INFO org.apache.flink.core.fs.FileSystem
> - Ensuring all FileSystem streams are closed for Flat Map
> (1/1)
> 2017-03-15 10:29:10,552 INFO
> org.apache.flink.runtime.taskmanager.TaskManager - Trying to
> register at JobManager akka.tcp://[email protected]:43893/user/jobmanager
> (attempt 1, timeout: 500 milliseconds)
> 2017-03-15 10:29:10,567 INFO org.apache.flink.core.fs.FileSystem
> - Ensuring all FileSystem streams are closed for Source:
> Custom Source -> Flat Map (1/1)
> 2017-03-15 10:29:10,632 INFO
> org.apache.flink.runtime.taskmanager.TaskManager - Successful
> registration at JobManager (akka.tcp://[email protected]:43893/user/jobmanager),
> starting network stack and library cache.
> 2017-03-15 10:29:10,633 INFO
> org.apache.flink.runtime.taskmanager.TaskManager - Determined
> BLOB server address to be /1.2.3.5:42830. Starting BLOB cache.
> 2017-03-15 10:29:10,633 INFO org.apache.flink.runtime.blob.BlobCache
> - Created BLOB cache storage directory
> /tmp/blobStore-d97e08db-d2f1-4f00-a7d1-30c2f5823934
> 2017-03-15 10:29:15,551 WARN akka.remote.ReliableDeliverySupervisor
> - Association with remote system
> [akka.tcp://[email protected]:44779] has failed, address is now gated for [5000]
> ms. Reason: [Association failed with [akka.tcp://[email protected]:44779]] Caused
> by: [Connection refused: /1.2.3.4:44779]
> 2017-03-15 10:29:20,571 WARN akka.remote.ReliableDeliverySupervisor
> - Association with remote system
> [akka.tcp://[email protected]:44779] has failed, address is now gated for [5000]
> ms. Reason: [Association failed with [akka.tcp://[email protected]:44779]] Caused
> by: [Connection refused: /1.2.3.4:44779]
> 2017-03-15 10:29:25,582 WARN akka.remote.ReliableDeliverySupervisor
> - Association with remote system
> [akka.tcp://[email protected]:44779] has failed, address is now gated for [5000]
> ms. Reason: [Association failed with [akka.tcp://[email protected]:44779]] Caused
> by: [Connection refused: /1.2.3.4:44779]
> 2017-03-15 10:29:30,592 WARN akka.remote.ReliableDeliverySupervisor
> - Association with remote system
> [akka.tcp://[email protected]:44779] has failed, address is now gated for [5000]
> ms. Reason: [Association failed with [akka.tcp://[email protected]:44779]] Caused
> by: [Connection refused: /1.2.3.4:44779]
> I modified the original IPs to 1.2.3.4 for JobManager1 and 1.2.3.5 for
> JobManager2 for confidentiality.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
