[
https://issues.apache.org/jira/browse/FLINK-10413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626835#comment-16626835
]
Zhu Zhu commented on FLINK-10413:
---------------------------------
[~Tison] glad you have reported this case and committed the fix. We can have a
sync first next time.
> requestPartitionState messages overwhelms JM RPC main thread
> ------------------------------------------------------------
>
> Key: FLINK-10413
> URL: https://issues.apache.org/jira/browse/FLINK-10413
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination
> Affects Versions: 1.7.0
> Reporter: Zhu Zhu
> Assignee: vinoyang
> Priority: Major
>
> We tried to benchmark the job scheduling performance with a 2000x2000
> ALL-to-ALL streaming(EAGER) job. The input data is empty so the tasks
> finishes soon after started.
> In this case we see slow RPC responses and TM/RM heartbeats to JM will
> finally timeout.
> We find ~2,000,000 requestPartitionState messages triggered by
> triggerPartitionProducerStateCheck in a short time, which overwhelms JM RPC
> main thread. This is due to downstream tasks can be started earlier than
> upstream tasks in EAGER scheduling.
>
> We's suggest no partition producer state check to avoid this issue. The task
> can just keep waiting for a while and retrying if the partition does not
> exist. There are two cases when the partition does not exist:
> # the partition is not started yet
> # the partition is failed
> In case 1, retry works. In case 2, a task failover will soon happen and
> cancel the downstream tasks as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
