[
https://issues.apache.org/jira/browse/SPARK-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342376#comment-15342376
]
Andrew Or commented on SPARK-15917:
-----------------------------------
(1) Yes, right now `spark.executor.instances` doesn't do anything in standalone
mode even if it's set so we should support it.
(2) There are several options here. Cores and number of executors are
inherently conflicting things so ideally we should disallow the setting of both
of them. We could throw an exception with a good error message, but that would
fail existing apps that do have both of them set. We could just log a warning,
but there's a high chance that people just won't see the warning.
Both options are fine but I'm slightly in favor of throwing an exception when
conflicting configs are set. You might have to dig into the internal scheduling
code in Master.scala to support num instances.
> Define the number of executors in standalone mode with an easy-to-use property
> ------------------------------------------------------------------------------
>
> Key: SPARK-15917
> URL: https://issues.apache.org/jira/browse/SPARK-15917
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, Spark Shell, Spark Submit
> Affects Versions: 1.6.1
> Reporter: Jonathan Taws
> Priority: Minor
>
> After stumbling across a few StackOverflow posts around the issue of using a
> fixed number of executors in standalone mode (non-YARN), I was wondering if
> we could not add an easier way to set this parameter than having to resort to
> some calculations based on the number of cores and the memory you have
> available on your worker.
> For example, let's say I have 8 cores and 30GB of memory available :
> - If no option is passed, one executor will be spawned with 8 cores and 1GB
> of memory allocated.
> - However, if I want to have only *2* executors, and to use 2 cores and 10GB
> of memory per executor, I will end up with *3* executors (as the available
> memory will limit the number of executors) instead of the 2 I was hoping for.
> Sure, I can set {{spark.cores.max}} as a workaround to get exactly what I
> want, but would it not be easier to add a {{--num-executors}}-like option to
> standalone mode to be able to really fine-tune the configuration ? This
> option is already available in YARN mode.
> From my understanding, I don't see any other option lying around that can
> help achieve this.
> This seems to be slightly disturbing for newcomers, and standalone mode is
> probably the first thing anyone will use to just try out Spark or test some
> configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
