Xiaochang Wu created SPARK-36699: ------------------------------------ Summary: Reuse compatible executors for stage-level scheduling Key: SPARK-36699 URL: https://issues.apache.org/jira/browse/SPARK-36699 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.1.2 Reporter: Xiaochang Wu Fix For: 3.3.0
Current stage-level scheduling allocated separated set of executors for different executor profiles. This approach simplified implementation, however is a waste of executor resources when the existing executors have enough resources to run the following tasks. We proposed to reuse executors by defining a "compatible" executor concept: two executors binding to different resource profiles are compatible only when the executorResources (cores in particular if not defining custom resources) are the same, but taskResources can be different. Users defining profiles should make sure the different taskResources are properly specified against the same executorResources. The typical user scenario is for different stages, user wants to use different core number for the task with same executor resources. For instance in CPU machine learning scenario, to achieve the best performance, given the same executor resources, when in ETL stage, user will allocate 1 core per task and many tasks, and in the following CPU training stage, user will use more cores per task and less tasks. Reusing executors allows better CPU resource utilization and better performance. The first PR will focus on reusing executors with same cores without custom resources. A SparkConf option is defined to change the default behavior which is not reusing executors. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org