Holden Karau created SPARK-44951:
------------------------------------
Summary: Improve Spark Dynamic Allocation
Key: SPARK-44951
URL: https://issues.apache.org/jira/browse/SPARK-44951
Project: Spark
Issue Type: Improvement
Components: Kubernetes, Spark Core, YARN
Affects Versions: 4.0.0
Reporter: Holden Karau
For Spark 4 we should aim to improve Spark's dynamic allocation. Some potential
ideas here includes the following:
* Plug-gable DEA algorithms
* How to reduce wastage on the RM side? Sometimes the driver asks for some
units of resources. But when RM provisions them, the driver cancels it.
* Support for "warm" executor pools which are not tied to a particular driver
but start and wait for a driver to connect to them to "claim" them.
* More explicit Cost Vs AppRunTime confiugration: A good DEA algo should allow
the developer to choose between cost and runtime. Sometimes developers might be
ok to pay higher costs for faster execution.
* Use previous run information to inform future runs
* Better selection of executors to be scaled down
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
