[
https://issues.apache.org/jira/browse/SPARK-9092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14635942#comment-14635942
]
Sandy Ryza commented on SPARK-9092:
-----------------------------------
I had a brief discussion with [~andrewor14] about this offline and wanted to
move the discussion public.
--num-executors and dynamic allocation are fundamentally at odds with each
other in the sense that neither makes sense in the context of the other. This
means that essentially one needs to override the other.
My position is that it makes more sense for --num-executors to override dynamic
allocation than the other way around. I.e. if --num-executors is set, behave
as if dynamic allocation were disabled. The advantages of this are:
* Cluster operators can turn on dynamic allocation as the default cluster
setting without impacting existing applications. The precedent set by existing
big data processing frameworks (MR, Impala, Tez) is that users can depend on
the framework to determine how many resources to acquire from the cluster
manager, so I think it's reasonable that most clusters would want to move to
dynamic allocation as the default. In a Cloudera setting, we plan to
eventually enable dynamic allocation as the factory default for all clusters,
but we'd like to minimize the extent to which we change the behavior of
existing apps.
* --num-executors is conceptually a more specific property, and specific
properties tend to override more general ones.
> Make --num-executors compatible with dynamic allocation
> -------------------------------------------------------
>
> Key: SPARK-9092
> URL: https://issues.apache.org/jira/browse/SPARK-9092
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 1.2.0
> Reporter: Niranjan Padmanabhan
>
> Currently when you enable dynamic allocation, you can't use --num-executors
> or the property spark.executor.instances. If we are to enable dynamic
> allocation by default, we should make these work so that existing workloads
> don't fail
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
