[
https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568726#comment-14568726
]
Saisai Shao commented on SPARK-4352:
------------------------------------
Hi [~sandyr], I have a proposal based on ratio to calculate the node locality
which can cover all the situation, even in the run-time of dynamic allocation,
say if we have 300 tasks, 200 tasks with node a, b, c; and 100 tasks with node
a, b, d. So the ratio of node locality is 300 : 300 : 200 : 100.
Now we need to allocate 10 executors, so according to the ratio distribution,
we will calculate out the best distribution of 10 executors based on the ratio
above:
300 * 10 / 300 : 300 * 10 / 300 : 200 * 10 / 300 : 100 * 10 / 300 = 10 : 10 : 7
: 4, floor to get the integer.
and requests:
4 executors: <a, b, c, d>
3 executors: <a, b, c>
3 executors: <a, b>
The probability of a, b is highest, and d is lowest, basicly follow the
distribution of data.
If we request for 1 executor, this would be {{300 * 1 / 300 : 300 * 1 / 300 :
200 * 1 / 300 : 100 * 1 / 300 = 1 : 1 : 1 : 1 }}, so each node has a equal
chance to allocate the executor.
If {{task number <= executor number * cores}} which means resource is over
demanded, both above method and this ratio based method is OK, since they will
by chance be the same, but ratio based implementation do not need to consider
this special case, the algorithm is same for all the situation.
If currently we already have some nodes with executors allocated, say for
example on nodes a, b, c, d, currently is 3 : 3 : 0 : 0, and we still need to
request for 10 executors, originally the ratio is 3 : 3 : 2 : 1, so we will get
10 executors on node a, b, c, d which is 3 : 3 : 2 : 2 by equal probability.
And we already have 3 executors on a and b, so actually we only need 4
executors with <c, d> to satisfy the ratio, and finally left 6 for <a, b, c, d>
to equally increase the executor number (since now the probability is already
satisfied).
What do you think about this algorithm, it's fairly general, one concern is
that it does not take the core numbers into consideration.
> Incorporate locality preferences in dynamic allocation requests
> ---------------------------------------------------------------
>
> Key: SPARK-4352
> URL: https://issues.apache.org/jira/browse/SPARK-4352
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
> Affects Versions: 1.2.0
> Reporter: Sandy Ryza
> Assignee: Saisai Shao
> Priority: Critical
> Attachments: Supportpreferrednodelocationindynamicallocation.pdf
>
>
> Currently, achieving data locality in Spark is difficult unless an
> application takes resources on every node in the cluster.
> preferredNodeLocalityData provides a sort of hacky workaround that has been
> broken since 1.0.
> With dynamic executor allocation, Spark requests executors in response to
> demand from the application. When this occurs, it would be useful to look at
> the pending tasks and communicate their location preferences to the cluster
> resource manager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
