[
https://issues.apache.org/jira/browse/FLINK-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611669#comment-15611669
]
ASF GitHub Bot commented on FLINK-3347:
---------------------------------------
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2696
That's a very good addition, we need something like that.
After an offline discussion with @tillrohrmann we came to the following
conclusion:
There is a tricky problem with that pure appraoch: When the JobManager
fails, all TaskManagers will "quarantine" that JobManager's actor system after
they detected the failure. That means they exit and restart. Effectively, a
JobManager failure results in all TaskManagers restarting.
That is a bit heavy.
Instead, we'll adjust this to do the following:
- TaskManagers must not watch the JobManager via Akka. That way,
JobManager failures do not cause any quarantining on the TaskManager side.
- The JobManager keeps watching the TaskManagers via Akka, so TaskManager
failures (false positives) still result in TaskManager quarantine, which means
the TaskManager need to restart when a TM-JM link breaks
How do TaskManagers then detect JobManager failure?
- TaskManagers send heartbeats to the JobManager anyways (accumulators,
in the future task status reconciliation). The TaskManagers use that to detect
JobManager failures.
- In high availability setups, TaskManagers notice JobManager failure
also via ZooKeeper
- In addition, in flip-6 the resource manager tells TaskManagers about
JobManager container failures
> TaskManager (or its ActorSystem) need to restart in case they notice
> quarantine
> -------------------------------------------------------------------------------
>
> Key: FLINK-3347
> URL: https://issues.apache.org/jira/browse/FLINK-3347
> Project: Flink
> Issue Type: Improvement
> Components: TaskManager
> Affects Versions: 0.10.1
> Reporter: Stephan Ewen
> Assignee: Till Rohrmann
> Priority: Critical
> Fix For: 1.0.0, 1.2.0, 1.1.4
>
>
> There are cases where Akka quarantines remote actor systems. In that case, no
> further communication is possible with that actor system unless one of the
> two actor systems is restarted.
> The result is that a TaskManager is up and available, but cannot register at
> the JobManager (Akka refuses connection because of the quarantined state),
> making the TaskManager a useless process.
> I suggest to let the TaskManager restart itself once it notices that either
> it quarantined the JobManager, or the JobManager quarantined it.
> It is possible to recognize that by listening to certain events in the actor
> system event stream:
> http://stackoverflow.com/questions/32471088/akka-cluster-detecting-quarantined-state
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
