[ https://issues.apache.org/jira/browse/SPARK-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15341360#comment-15341360 ]
Jonathan Taws edited comment on SPARK-15917 at 6/21/16 8:24 AM: ---------------------------------------------------------------- If I run the following command : {{spark-shell \--master <master-url> \--conf spark.cores.max=2 \--conf spark.executor.instances=2 \--executor-cores 2 \--executor-memory 4g}}, I get only one executor, as there are only 2 cores available (same with {{--num-executors}} instead of {{spark.executor.instances}}). {{spark.cores.max}} does its job right here, and it just seems like the executor instances property is just being discarded here. If anything, adding a support for a {{--num-executors}}-like property in standalone would allow to be consistent across deployment methods and, in my experience, make the beginner experience slightly easier. was (Author: jonathantaws): If I run the following command : {{spark-shell --master <master-url> --conf spark.cores.max=2 --conf spark.executor.instances=2 --executor-cores 2 --executor-memory 4g}}, I get only one executor, as there are only 2 cores available (same with {{--num-executors}} instead of {{spark.executor.instances}}). {{spark.cores.max}} does its job right here, and it just seems like the executor instances property is just being discarded here. If anything, adding a support for a {{--num-executors}}-like property in standalone would allow to be consistent across deployment methods and, in my experience, make the beginner experience slightly easier. > Define the number of executors in standalone mode with an easy-to-use property > ------------------------------------------------------------------------------ > > Key: SPARK-15917 > URL: https://issues.apache.org/jira/browse/SPARK-15917 > Project: Spark > Issue Type: Improvement > Components: Spark Core, Spark Shell, Spark Submit > Affects Versions: 1.6.1 > Reporter: Jonathan Taws > Priority: Minor > > After stumbling across a few StackOverflow posts around the issue of using a > fixed number of executors in standalone mode (non-YARN), I was wondering if > we could not add an easier way to set this parameter than having to resort to > some calculations based on the number of cores and the memory you have > available on your worker. > For example, let's say I have 8 cores and 30GB of memory available : > - If no option is passed, one executor will be spawned with 8 cores and 1GB > of memory allocated. > - However, if I want to have only *2* executors, and to use 2 cores and 10GB > of memory per executor, I will end up with *3* executors (as the available > memory will limit the number of executors) instead of the 2 I was hoping for. > Sure, I can set {{spark.cores.max}} as a workaround to get exactly what I > want, but would it not be easier to add a {{--num-executors}}-like option to > standalone mode to be able to really fine-tune the configuration ? This > option is already available in YARN mode. > From my understanding, I don't see any other option lying around that can > help achieve this. > This seems to be slightly disturbing for newcomers, and standalone mode is > probably the first thing anyone will use to just try out Spark or test some > configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org