[
https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16286058#comment-16286058
]
Sean Owen commented on SPARK-22683:
-----------------------------------
This is still optimizing for a particular type of workload. Imagine a series of
infrequent jobs which keep spawning 20 short-running tasks, and 10 executor
slots currently available. It's probably sensible to not spin up more executors
at all, as it won't help execution time much at all. At the other end, imagine
a single job with 20 very long-running tasks. Not spawning more executors is a
loss -- almost 2x execution time. A time-based adaptive parameter would let you
handle both pretty well, but your new config would not spawn more executors,
ever.
You could still argue it either way about which lever is better, but we already
have a lever like this to pull. I do not see a good reason to add this.
I'll leave it open another week for other opinions.
> Allow tuning the number of dynamically allocated executors wrt task number
> --------------------------------------------------------------------------
>
> Key: SPARK-22683
> URL: https://issues.apache.org/jira/browse/SPARK-22683
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.0, 2.2.0
> Reporter: Julien Cuquemelle
> Labels: pull-request-available
>
> let's say an executor has spark.executor.cores / spark.task.cpus taskSlots
> The current dynamic allocation policy allocates enough executors
> to have each taskSlot execute a single task, which minimizes latency,
> but wastes resources when tasks are small regarding executor allocation
> overhead.
> By adding the tasksPerExecutorSlot, it is made possible to specify how many
> tasks
> a single slot should ideally execute to mitigate the overhead of executor
> allocation.
> PR: https://github.com/apache/spark/pull/19881
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
