[
https://issues.apache.org/jira/browse/FLINK-14010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925778#comment-16925778
]
Till Rohrmann commented on FLINK-14010:
---------------------------------------
Thanks for reporting this issue [~Tison]. I think your analysis is correct that
we currently do not react properly to {{onShutdownRequest}} signals. What
happens is that we only terminate the {{ResourceManager}} but not the other
Flink components ({{Dispatcher}} and {{JobMasters}}).
Before looking into a potential solutions let's answer your questions because
they are important for understanding the bigger picture.
The two components {{Dispatcher}} and {{ResourceManager}} have been designed to
run independently of the other component, potentially also in a separate
process. The current implementation couples the {{Dispatcher}} with the
{{JobMasters}} but there should be nothing fundamental preventing from
decoupling them. The main thing one needs to do is to add logic to start
{{JobMaster}} processes somewhere. So long story short, the idea is that all
components can run independently (modulo the existing {{Dispatcher}}
implementation).
At the moment we run all these components in one process because it was easier
and less work to implement it this way. This is reflected in the
{{DispatcherResourceManagerComponent}}. Hence, for the current implementation
we can assume that if the {{ResourceManager}} terminates, then the other
components should shut down as well. This could be implemented by registering
in the {{DispatcherResourceManagerComponent}} a callback on the
{{ResourceManager's}} termination future which triggers the closing of the
{{DispatcherResourceManagerComponent}}.
Concerning using a single leader election service, I would say that this is not
feasible if we still want to keep the option to deploy the components in
separate processes eventually.
Would you have time to work on the fix for this problem [~Tison]?
> Dispatcher & JobManagers don't give up leadership when AM is shut down
> ----------------------------------------------------------------------
>
> Key: FLINK-14010
> URL: https://issues.apache.org/jira/browse/FLINK-14010
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Runtime / Coordination
> Affects Versions: 1.7.2, 1.8.1, 1.9.0, 1.10.0
> Reporter: TisonKun
> Priority: Critical
>
> In YARN deployment scenario, YARN RM possibly launches a new AM for the job
> even if the previous AM does not terminated, for example, when AMRM heartbeat
> timeout. This is a common case that RM will send a shutdown request to the
> previous AM and expect the AM shutdown properly.
> However, currently in {{YARNResourceManager}}, we handle this request in
> {{onShutdownRequest}} which simply close the {{YARNResourceManager}} *but not
> Dispatcher and JobManagers*. Thus, Dispatcher and JobManager launched in new
> AM cannot be granted leadership properly. Visually,
> on previous AM: Dispatcher leader, JM leaders
> on new AM: ResourceManager leader
> since on client side or in per-job mode, JobManager address and port are
> configured as the new AM, the whole cluster goes into an unrecoverable
> inconsistent status: client all queries the dispatcher on new AM who is now
> the leader. Briefly, Dispatcher and JobManagers on previous AM do not give up
> their leadership properly.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
