the community is currently working on Flink's scheduler component [1]
That sounds great! I agree that spreading tasks across the nodes is not
always desirable but it would be nice to give users an option to provide
hints to the scheduler. The location aware bulk scheduling you mentioned
would be useful.
Today, there is already the option to assign Resources to a
StreamTransformation. From a quick test, it seems like those resource
specifications are not honored yet.
-Max
On 13.10.18 01:41, Thomas Weise wrote:
Hi Till,
Thanks for the pointer, glad that this is being worked on.
It almost looks like the non deterministic distribution behavior started
with 1.5.x (?) and that surprised us.
https://issues.apache.org/jira/browse/BEAM-5713
I agree that there is no one strategy that fits every use case. If an
application is limited by a resource per machine that the scheduler does
not understand (like let's say CPU or disk I/O), then it would be nice to
have a way to hint that round-robin distribution is desired (or achieve the
same through anti-affinity or resource constraints).
Thanks,
Thomas
On Fri, Oct 12, 2018 at 2:06 AM Till Rohrmann <trohrm...@apache.org> wrote:
Hi Max,
the community is currently working on Flink's scheduler component [1]. One
of the things we want to enable in the future is bulk scheduling. With
this, it should also be possible to add strategies how to distribute tasks
across multiple TMs (spreading vs. co-locating).
In general, I'm not 100% sure whether spreading out tasks is always the
best strategy. Especially if you have a network heavy job co-locating tasks
on the same TM could have benefits over spreading the tasks out.
[1] https://issues.apache.org/jira/browse/FLINK-10429
Cheers,
Till
On Thu, Oct 11, 2018 at 8:16 PM Maximilian Michels <m...@apache.org> wrote:
Hi everyone,
I've recently come across a cluster scheduling problem users are facing.
Clusters where TaskManagers have more slots than the parallelism
(#tm_slots > job_parallelism), tend to schedule all job tasks on a
single TaskManager.
This is not good for spreading load and has been discussed in FLINK-1003
[1] and the other duplicate JIRA issues.
I know that this is not really an issue if the cluster is created
exclusively for the Job, or if the number of slots per Taskmanager is
smaller than the parallelism. However, this seems like a rather easy
improvement to the Scheduler which would have a huge impact on
performance.
On the JIRA issue page it has been mentioned that this was put on hold
to work on dynamic scaling first.
Now that the basic building blocks for dynamic scaling are in place, do
you think it would be possible to tackle FLINK-1003?
Thanks,
Max
[1] https://issues.apache.org/jira/browse/FLINK-1003
