[
https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16286032#comment-16286032
]
Julien Cuquemelle commented on SPARK-22683:
-------------------------------------------
I'd like to point out that when refering to "my use case", it is actually a
wide range of job sizes, ranging from tens to thousands splits in the data
partitioning and between 400 to 9000 seconds of wall clock time.
Maybe I didn't emphasize enough in my last answer that the figures I presented
using schedulerBacklogTimeout needed tuning for a specific job, and that the
resulting value of the parameter will not be optimal for other job settings. My
proposal allows to have a sensible default over the whole range of jobs.
This is also true of the maxExecutors parameters, each job needs to be tuned
using this parameter.
Regarding "there's nothing inherently more efficient about the scheduling
policy you're proposing", I might add a few information: so far I have aimed at
having the same latency as the legacy MR job, which yield the 6 tasksPerSlot
figure and a 30% resource usage decrease.
Here are the figures with other values of taskPerSlot, in terms of gain wrt the
corresponding MR jobs:
||taskPerCore||VcoreGain wrt MR (%)||WallTimeGain wrt MR (%)||
|1.0|-114.329984|43.777071|
|2.0|5.373456|37.610840|
|3.0|20.785099|28.528394|
|4.0|26.501597|18.516913|
|6.0|30.852756|-0.991918|
|8.0|31.147028|-14.633520|
It seems to me the 2 tasks per slot is particularly interesting, as it allows
to go from a doubling of used resources to a usage similar to MR with a very
low impact on the latency.
I bet that this resource usage reduction might not be limited to our use case
...
> Allow tuning the number of dynamically allocated executors wrt task number
> --------------------------------------------------------------------------
>
> Key: SPARK-22683
> URL: https://issues.apache.org/jira/browse/SPARK-22683
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.0, 2.2.0
> Reporter: Julien Cuquemelle
> Labels: pull-request-available
>
> let's say an executor has spark.executor.cores / spark.task.cpus taskSlots
> The current dynamic allocation policy allocates enough executors
> to have each taskSlot execute a single task, which minimizes latency,
> but wastes resources when tasks are small regarding executor allocation
> overhead.
> By adding the tasksPerExecutorSlot, it is made possible to specify how many
> tasks
> a single slot should ideally execute to mitigate the overhead of executor
> allocation.
> PR: https://github.com/apache/spark/pull/19881
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
