[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297316#comment-16297316
]
Xuefu Zhang commented on SPARK-22765:
-------------------------------------
Actually I meant across parallel stages (those connected to union
transformation), not serial stages that I care much less. The following is what
I meant (in Hive on Spark context):
{code}
Status: Running (Hive on Spark job[4])
--------------------------------------------------------------------------------------
STAGES ATTEMPT STATUS TOTAL COMPLETED RUNNING PENDING
FAILED
--------------------------------------------------------------------------------------
Stage-10 .. 0 RUNNING 340 126 214 0
0
Stage-11 0 PENDING 201 0 0 201
0
Stage-12 0 PENDING 191 0 0 191
0
Stage-13 . 0 RUNNING 178 33 145 0
0
Stage-14 0 PENDING 115 0 0 115
0
Stage-15 0 PENDING 105 0 0 105
0
Stage-16 0 PENDING 592 0 0 592
0
Stage-17 0 PENDING 191 0 0 191
0
Stage-4 ....... 0 RUNNING 178 157 21 0
4
Stage-5 0 PENDING 115 0 0 115
0
Stage-6 0 PENDING 105 0 0 105
0
Stage-7 ..... 0 RUNNING 340 232 108 0
0
Stage-8 0 PENDING 201 0 0 201
0
Stage-9 0 PENDING 191 0 0 191
0
{code}
> Create a new executor allocation scheme based on that of MR
> -----------------------------------------------------------
>
> Key: SPARK-22765
> URL: https://issues.apache.org/jira/browse/SPARK-22765
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler
> Affects Versions: 1.6.0
> Reporter: Xuefu Zhang
>
> Many users migrating their workload from MR to Spark find a significant
> resource consumption hike (i.e, SPARK-22683). While this might not be a
> concern for users that are more performance centric, for others conscious
> about cost, such hike creates a migration obstacle. This situation can get
> worse as more users are moving to cloud.
> Dynamic allocation make it possible for Spark to be deployed in multi-tenant
> environment. With its performance-centric design, its inefficiency has also
> unfortunately shown up, especially when compared with MR. Thus, it's believed
> that MR-styled scheduler still has its merit. Based on our research, the
> inefficiency associated with dynamic allocation comes in many aspects such as
> executor idling out, bigger executors, many stages (rather than 2 stages only
> in MR) in a spark job, etc.
> Rather than fine tuning dynamic allocation for efficiency, the proposal here
> is to add a new, efficiency-centric scheduling scheme based on that of MR.
> Such a MR-based scheme can be further enhanced and be more adapted to Spark
> execution model. This alternative is expected to offer good performance
> improvement (compared to MR) still with similar to or even better efficiency
> than MR.
> Inputs are greatly welcome!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
