[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297453#comment-16297453
]
Nan Zhu commented on SPARK-22765:
---------------------------------
I took a look at the code, one of the possibilities is as following:
we add the new executor id
https://github.com/apache/spark/blob/a233fac0b8bf8229d938a24f2ede2d9d8861c284/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L598
and immediately filter idle executors
https://github.com/apache/spark/blob/a233fac0b8bf8229d938a24f2ede2d9d8861c284/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L419
Right after this step, since executorIdToTaskIds hasn't contained executor id,
the newly added executor is applied with onExecutorIdle
in the method of onExecutorIdle:
https://github.com/apache/spark/blob/a233fac0b8bf8229d938a24f2ede2d9d8861c284/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L479
the newly added executor is also vulnerable to be assigned with a removeTimes
value, since removeTimes and executorsPendingToRemove would not contain the
newly added executor ID. Based on the calculation method, the removeTime would
be 60s after. Then we will remove the executor after 60s
(https://github.com/apache/spark/blob/a233fac0b8bf8229d938a24f2ede2d9d8861c284/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L266).
The potential fix is to update executorIdToTaskIds before we filter idle
executors (but SPARK-21656 should prevent this happening....still trying to
figure out why SPARK-21656 not working)
> Create a new executor allocation scheme based on that of MR
> -----------------------------------------------------------
>
> Key: SPARK-22765
> URL: https://issues.apache.org/jira/browse/SPARK-22765
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler
> Affects Versions: 1.6.0
> Reporter: Xuefu Zhang
>
> Many users migrating their workload from MR to Spark find a significant
> resource consumption hike (i.e, SPARK-22683). While this might not be a
> concern for users that are more performance centric, for others conscious
> about cost, such hike creates a migration obstacle. This situation can get
> worse as more users are moving to cloud.
> Dynamic allocation make it possible for Spark to be deployed in multi-tenant
> environment. With its performance-centric design, its inefficiency has also
> unfortunately shown up, especially when compared with MR. Thus, it's believed
> that MR-styled scheduler still has its merit. Based on our research, the
> inefficiency associated with dynamic allocation comes in many aspects such as
> executor idling out, bigger executors, many stages (rather than 2 stages only
> in MR) in a spark job, etc.
> Rather than fine tuning dynamic allocation for efficiency, the proposal here
> is to add a new, efficiency-centric scheduling scheme based on that of MR.
> Such a MR-based scheme can be further enhanced and be more adapted to Spark
> execution model. This alternative is expected to offer good performance
> improvement (compared to MR) still with similar to or even better efficiency
> than MR.
> Inputs are greatly welcome!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
