[
https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-22683:
------------------------------
Target Version/s: (was: 2.1.1, 2.2.0)
The overhead of small tasks doesn't change if you over-commit tasks with
respect to task slots. I think this isn't really a solution, and the app needs
to look at ways to make fewer, larger tasks.
There's overhead to adding yet another knob to turn here, and its interaction
with other settings isn't obvious. This concept isn't present elsewhere in
Spark.
You will also kind of get this effect anyway; if tasks are finishing very
quickly, and locality wait is at all positive, you'll find tasks tend to favor
older executors with cached data, and the newer ones, dynamically allocated,
may get few or no tasks and deallocate anyway. Allocation only happens when the
task backlog builds up.
> Allow tuning the number of dynamically allocated executors wrt task number
> --------------------------------------------------------------------------
>
> Key: SPARK-22683
> URL: https://issues.apache.org/jira/browse/SPARK-22683
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.0, 2.2.0
> Reporter: Julien Cuquemelle
> Priority: Minor
>
> let's say an executor has spark.executor.cores / spark.task.cpus taskSlots
> The current dynamic allocation policy allocates enough executors
> to have each taskSlot execute a single task, which minimizes latency,
> but wastes resources when tasks are small regarding executor allocation
> overhead.
> By adding the tasksPerExecutorSlot, it is made possible to specify how many
> tasks
> a single slot should ideally execute to mitigate the overhead of executor
> allocation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
