Jonathan Taws created SPARK-15917:
-------------------------------------
Summary: Define the number of executors in standalone mode with an
easy-to-use property
Key: SPARK-15917
URL: https://issues.apache.org/jira/browse/SPARK-15917
Project: Spark
Issue Type: Improvement
Components: Spark Core, Spark Shell, Spark Submit
Affects Versions: 1.6.1
Reporter: Jonathan Taws
Priority: Minor
After stumbling across a few StackOverflow posts around the issue of using a
fixed number of executors in standalone mode (non-YARN), I was wondering if we
could not add an easier way to set this parameter than having to resort to some
calculations based on the number of cores and the memory you have available on
your worker.
For example, let's say I have 8 cores and 30GB of memory available.
If no option is passed, one executor will be spawned with 8 cores and 1GB of
memory allocated.
However, let's say I want to have only *2* executors, and to use 2 cores and
10GB of memory per executor, I will end up with *3* executors (as the available
memory will limit the number of executors) instead of the 2 I was hoping for.
Sure, I can set {{spark.cores.max}} as a workaround to get exactly what I want,
but would it not be easier to add a {{--num-executors}}-like option to
standalone mode to be able to really fine-tune the configuration ? This option
is already available in YARN mode.
>From my understanding, I don't see any other option lying around that can help
>achieve this.
This seems to be slightly disturbing for newcomers, and standalone mode is
probably the first thing anyone will use to just try out Spark or test some
configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
