[
https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178554#comment-14178554
]
Dag Liodden commented on SPARK-3174:
------------------------------------
Hey guys, glad to see some progress on this!
I believe a very common use case here is to not do scaling based on the
internal needs of the job, but also external factors. For instance, we're using
the fair scheduler and want to have Spark jobs opportunistically leverage
available containers, but it's equally important that the executor count is
scaled down if the job does over its fair share during the job lifetime. I'm
not sure that's being covered here?
I'm sure there will be classes of jobs that heavily leverage RDD caching where
"heuristic yielding" will be very hard to implement with deterministic
performance impacts, but "regular" MR jobs should be pretty straight-forward.
Pluggable algorithms definitely sound like a viable approach for this and for
"regular" MR it could be as simple as regularly check the current target
container count and create or drain + kill executors accordingly?
Dag
> Provide elastic scaling within a Spark application
> --------------------------------------------------
>
> Key: SPARK-3174
> URL: https://issues.apache.org/jira/browse/SPARK-3174
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
> Affects Versions: 1.0.2
> Reporter: Sandy Ryza
> Assignee: Andrew Or
> Attachments: SPARK-3174design.pdf, SparkElasticScalingDesignB.pdf,
> dynamic-scaling-executors-10-6-14.pdf
>
>
> A common complaint with Spark in a multi-tenant environment is that
> applications have a fixed allocation that doesn't grow and shrink with their
> resource needs. We're blocked on YARN-1197 for dynamically changing the
> resources within executors, but we can still allocate and discard whole
> executors.
> It would be useful to have some heuristics that
> * Request more executors when many pending tasks are building up
> * Discard executors when they are idle
> See the latest design doc for more information.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
