[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289613#comment-16289613
]
Thomas Graves commented on SPARK-22765:
---------------------------------------
why doesn't idle timeout very small < 5 seconds work? If there is no work to
be done it should exit soon after task finishes similar to MR. Note that tez
added container reuse as well which is similar to spark scheme.
Basically I think you are proposing a change that essentially add a config that
does not reuse containers.
I'm not sure I agree with this and that it will help resource utilization.
Especially when looking at the whole ecosystem. Without reuse you have to go
back to yarn to ask for more, which depending on cluster usage could cause
significant overhead to wait for more containers. You are bringing up and
killing processes, you are re-downloading things into distributed cache, etc.
So from vcore/memory per second on yarn it might be better but it affects other
things as well. Just something to keep in mind.
You are seeing this on very short running tasks?
> Create a new executor allocation scheme based on that of MR
> -----------------------------------------------------------
>
> Key: SPARK-22765
> URL: https://issues.apache.org/jira/browse/SPARK-22765
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler
> Affects Versions: 1.6.0
> Reporter: Xuefu Zhang
>
> Many users migrating their workload from MR to Spark find a significant
> resource consumption hike (i.e, SPARK-22683). While this might not be a
> concern for users that are more performance centric, for others conscious
> about cost, such hike creates a migration obstacle. This situation can get
> worse as more users are moving to cloud.
> Dynamic allocation make it possible for Spark to be deployed in multi-tenant
> environment. With its performance-centric design, its inefficiency has also
> unfortunately shown up, especially when compared with MR. Thus, it's believed
> that MR-styled scheduler still has its merit. Based on our research, the
> inefficiency associated with dynamic allocation comes in many aspects such as
> executor idling out, bigger executors, many stages (rather than 2 stages only
> in MR) in a spark job, etc.
> Rather than fine tuning dynamic allocation for efficiency, the proposal here
> is to add a new, efficiency-centric scheduling scheme based on that of MR.
> Such a MR-based scheme can be further enhanced and be more adapted to Spark
> execution model. This alternative is expected to offer good performance
> improvement (compared to MR) still with similar to or even better efficiency
> than MR.
> Inputs are greatly welcome!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
