Julien Cuquemelle created SPARK-22683:
-----------------------------------------
Summary: Allow tuning the number of dynamically allocated
executors wrt task number
Key: SPARK-22683
URL: https://issues.apache.org/jira/browse/SPARK-22683
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.2.0, 2.1.0
Reporter: Julien Cuquemelle
Priority: Minor
let's say an executor has spark.executor.cores / spark.task.cpus taskSlots
The current dynamic allocation policy allocates enough executors
to have each taskSlot execute a single task, which minimizes latency,
but wastes resources when tasks are small regarding executor allocation
overhead.
By adding the tasksPerExecutorSlot, it is made possible to specify how many
tasks
a single slot should ideally execute to mitigate the overhead of executor
allocation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
