Repository: spark Updated Branches: refs/heads/master 235d28333 -> 6949a9c5c
[SPARK-21834] Incorrect executor request in case of dynamic allocation ## What changes were proposed in this pull request? killExecutor api currently does not allow killing an executor without updating the total number of executors needed. In case of dynamic allocation is turned on and the allocator tries to kill an executor, the scheduler reduces the total number of executors needed ( see https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L635) which is incorrect because the allocator already takes care of setting the required number of executors itself. ## How was this patch tested? Ran a job on the cluster and made sure the executor request is correct Author: Sital Kedia <ske...@fb.com> Closes #19081 from sitalkedia/skedia/oss_fix_executor_allocation. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6949a9c5 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6949a9c5 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6949a9c5 Branch: refs/heads/master Commit: 6949a9c5c6120fdde1b63876ede661adbd1eb15e Parents: 235d283 Author: Sital Kedia <ske...@fb.com> Authored: Wed Aug 30 14:19:13 2017 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Wed Aug 30 14:19:13 2017 -0700 ---------------------------------------------------------------------- .../main/scala/org/apache/spark/ExecutorAllocationManager.scala | 3 +++ 1 file changed, 3 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/6949a9c5/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala index 3350326..7a5fb9a 100644 --- a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala +++ b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala @@ -446,6 +446,9 @@ private[spark] class ExecutorAllocationManager( } else { client.killExecutors(executorIdsToBeRemoved) } + // [SPARK-21834] killExecutors api reduces the target number of executors. + // So we need to update the target with desired value. + client.requestTotalExecutors(numExecutorsTarget, localityAwareTasks, hostToLocalTaskCount) // reset the newExecutorTotal to the existing number of executors newExecutorTotal = numExistingExecutors if (testing || executorsRemoved.nonEmpty) { --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
