[
https://issues.apache.org/jira/browse/FLINK-18451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150280#comment-17150280
]
Till Rohrmann commented on FLINK-18451:
---------------------------------------
I think it is correct that the problem you are describing can occur [~Ming Li].
Having a persistent record of cluster resources will allow you to wait for them
but at some point you will have to mark them dead in order to make progress.
Hence, one could also simply introduce a delay before starting the new master.
If the delay is larger than the configured heartbeat timeout, then the problem
should not occur. Differently said, if you configure the heartbeat timeout to
be lower than the recovery of a {{JobMaster}}, then this problem should also
not occur.
For exactly-once end-to-end pipelines, this problem is also solved on a
different level: Sinks will only output results once a checkpoint is completed.
In the case of a failing {{JobMaster}} the rogue TMs will no longer complete a
checkpoint and hence no longer output elements to external systems. For at
least once processing guarantees this is different, though.
Is this a problem you have observed while running Flink or is this more of a
theoretical thought [~Ming Li]?
> Flink HA on yarn may appear TaskManager double running when HA is restored
> --------------------------------------------------------------------------
>
> Key: FLINK-18451
> URL: https://issues.apache.org/jira/browse/FLINK-18451
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.9.0
> Reporter: ming li
> Priority: Major
> Labels: high-availability
>
> We found that when NodeManager is lost, the new JobManager will be restored
> by Yarn's ResourceManager, and the Leader node will be registered on
> Zookeeper. The original TaskManager will find the new JobManager through
> Zookeeper and close the old JobManager connection. At this time, all tasks of
> the TaskManager will fail. The new JobManager will directly perform job
> recovery and recover from the latest checkpoint.
> However, during the recovery process, when a TaskManager is abnormally
> connected to Zookeeper, it is not registered with the new JobManager in time.
> Before the following timeout:
> 1. Connect with Zookeeper
> 2. Heartbeat with JobManager/ResourceManager
> Task will continue to run (assuming that Task can run independently in
> TaskManager). Assuming that HA recovers fast enough, some Task double runs
> will occur at this time.
> Do we need to make a persistent record of the cluster resources we allocated
> during the runtime, and use it to judge all Task stops when HA is restored?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
