[
https://issues.apache.org/jira/browse/FLINK-14010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925460#comment-16925460
]
TisonKun commented on FLINK-14010:
----------------------------------
CC [~StephanEwen] [~till.rohrmann] [~xiaogang.shi]
Here comes a high-level problem, do we explicitly constrain Dispatcher,
ResourceManager and JobManagers run on one process?
1. the usage of reference to {{JobManagerGateway}} in Dispatcher already infers
that we require this.
2. back to the design of FLIP-6, we have a global singleton of Dispatcher, and
for each job, launch a JobManager and ResourceManager. The implementation
diverges quite a lot. Could you please provide any background?
3. if we explicitly constrain as above, we actually need not to start leader
election services per components, actually, we can use the abstraction and
layout as below:
- start a leader election service per dispatcher-resource-manager component, in
cluster entrypoint level. It will participant the election and all metadata
commits are delegate to this service.
- all cluster level components that need to publish their address, such as
Dispatcher, ResourceManager and WebMonitor publish their address via this
leader election service.
- Actors can be started as {{PermanentlyFencedRpcEndpoint}} and thus we survive
from handling a lot of mutable shared state among leadership epoch.
Specifically, cluster entrypoint acts as DispatcherRunner and so on, like
JobManagerRunner to JobMaster. See also [this
branch|https://github.com/tillrohrmann/flink/commits/removeSuspendFromJobMaster].
- back to this issue, cluster entrypoint({{YARNClusterEntrypoint}} maybe)
reacts to AMRM request and thus all components can be required to shutdown
properly.
> Dispatcher & JobManagers don't give up leadership when AM is shut down
> ----------------------------------------------------------------------
>
> Key: FLINK-14010
> URL: https://issues.apache.org/jira/browse/FLINK-14010
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Runtime / Coordination
> Affects Versions: 1.7.2, 1.8.1, 1.9.0, 1.10.0
> Reporter: TisonKun
> Priority: Critical
>
> In YARN deployment scenario, YARN RM possibly launches a new AM for the job
> even if the previous AM does not terminated, for example, when AMRM heartbeat
> timeout. This is a common case that RM will send a shutdown request to the
> previous AM and expect the AM shutdown properly.
> However, currently in {{YARNResourceManager}}, we handle this request in
> {{onShutdownRequest}} which simply close the {{YARNResourceManager}} *but not
> Dispatcher and JobManagers*. Thus, Dispatcher and JobManager launched in new
> AM cannot be granted leadership properly. Visually,
> on previous AM: Dispatcher leader, JM leaders
> on new AM: ResourceManager leader
> since on client side or in per-job mode, JobManager address and port are
> configured as the new AM, the whole cluster goes into an unrecoverable
> inconsistent status: client all queries the dispatcher on new AM who is now
> the leader. Briefly, Dispatcher and JobManagers on previous AM do not give up
> their leadership properly.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
