Xiaochang Wu created SPARK-36699:
------------------------------------
Summary: Reuse compatible executors for stage-level scheduling
Key: SPARK-36699
URL: https://issues.apache.org/jira/browse/SPARK-36699
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.1.2
Reporter: Xiaochang Wu
Fix For: 3.3.0
Current stage-level scheduling allocated separated set of executors for
different executor profiles. This approach simplified implementation, however
is a waste of executor resources when the existing executors have enough
resources to run the following tasks.
We proposed to reuse executors by defining a "compatible" executor concept: two
executors binding to different resource profiles are compatible only when the
executorResources (cores in particular if not defining custom resources) are
the same, but taskResources can be different. Users defining profiles should
make sure the different taskResources are properly specified against the same
executorResources.
The typical user scenario is for different stages, user wants to use different
core number for the task with same executor resources. For instance in CPU
machine learning scenario, to achieve the best performance, given the same
executor resources, when in ETL stage, user will allocate 1 core per task and
many tasks, and in the following CPU training stage, user will use more cores
per task and less tasks. Reusing executors allows better CPU resource
utilization and better performance.
The first PR will focus on reusing executors with same cores without custom
resources. A SparkConf option is defined to change the default behavior which
is not reusing executors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
