[
https://issues.apache.org/jira/browse/SPARK-28403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17022527#comment-17022527
]
Thomas Graves commented on SPARK-28403:
---------------------------------------
so after looking at the pr for this, this logic may have been an attempt to get
executors on different hosts. The speculation logic in the scheduler is such
that it will only run a speculative task on a different host then the current
running task.
> Executor Allocation Manager can add an extra executor when speculative tasks
> ----------------------------------------------------------------------------
>
> Key: SPARK-28403
> URL: https://issues.apache.org/jira/browse/SPARK-28403
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.3.0
> Reporter: Thomas Graves
> Priority: Major
>
> It looks like SPARK-19326 added a bug in the execuctor allocation maanger
> where it adds an extra executor when it shouldn't when we have pending
> speculative tasks but the target number didn't change.
> [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L377]
> It doesn't look like this is necessary since it already added in the
> pendingSpeculative tasks.
> See the questioning of this on the PR at:
> https://github.com/apache/spark/pull/18492/files#diff-b096353602813e47074ace09a3890d56R379
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
