Thomas Graves created SPARK-30391:
-------------------------------------
Summary: ExecutorAllocationManager requestTotalExecutors in
removeExecutors may not be needed
Key: SPARK-30391
URL: https://issues.apache.org/jira/browse/SPARK-30391
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.4, 3.0.0
Reporter: Thomas Graves
in the ExecutorAllocationManager.removeExecutors
[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L459]
there is a call to requestTotalExecutors that I don't think is needed anymore
because SPARK-23365 changed the killExecutors call to not adjust the target
number. We should investigate if we can remove it.
{color:#808080}
{color}{color:#808080}
{color}{color:#808080}// [SPARK-21834] killExecutors api reduces the target
number of executors.
{color}{color:#808080}// So we need to update the target with desired value.
{color}client.requestTotalExecutors({color:#660e7a}numLocalityAwareTasksPerResourceProfileId{color}.toMap,
{color:#660e7a}rpIdToHostToLocalTaskCount{color},
{color:#660e7a}numExecutorsTargetPerResourceProfileId{color}.toMap)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
