[
https://issues.apache.org/jira/browse/FLINK-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224723#comment-17224723
]
Zhu Zhu commented on FLINK-15325:
---------------------------------
I'd like to resume this discussion because it has been the cause of imbalanced
load problem in our latest TPCDS benchmark. It also means this problem can
happen in real production scenarios.
In cases (e.g. broadcast join with one small input) that there is a small
source with parallelism=1, we can see its consumer tasks will be deployed to
the same machine of that source task until all the slots on that machine are
assigned, while the other machines can have more than 50% available slots
unassigned. That source task machine then will have a heavy load and tasks on
it will be much slower than tasks on other machines.
We tried a local patch to disable input location preferences and 10TB TPCDS
benchmark turned out to be ~5% faster (~500s less time v.s. total 9000s).
So I think it would be good to have an config option (e.g.
jobmanager.enable-input-location-preferences) to enable or disable input
location preferences. If it is disabled,
{{DefaultPreferredLocationsRetriever#getPreferredLocationsBasedOnInputs()}}
should always return an empty set. To be safe, we can keep it "true" by default.
And we will actually not completely lost the benefit of input locality because
we now have deterministic slot sharing. The input locality of FORWARD
connections, which contributes to the major benefits of input locality, will be
respected when computing {{ExecutionSlotSharingGroup}} (when the FORWARD edge
is PIPELINED). Thus only the input locality across pipelined regions will be
ignored.
What do you think? [~trohrmann]
> Input location preference which affects task distribution may make certain
> job performance worse
> -------------------------------------------------------------------------------------------------
>
> Key: FLINK-15325
> URL: https://issues.apache.org/jira/browse/FLINK-15325
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
> Affects Versions: 1.10.0
> Reporter: Zhu Zhu
> Priority: Major
> Attachments: D58ADB03-7187-46B1-B077-91E5005FD463.png
>
>
> When running TPC-DS jobs in a session cluster, we observed that sometimes
> tasks are not evenly distributed in TMs. The root cause turned out to be that
> the downstream tasks tend to be TM or host local with its input tasks. This
> helps to reduce network shuffle.
> However, in certain cases, like the topology presented in the attached image,
> jamming the input task's TM and machine with downstream tasks would affect
> the performance. In this case, respecting input location preferences is
> causing troubles more than bringing benefits.
> So I'm wondering whether we should introduce a config so that users can
> disable input location preferences?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
