[ https://issues.apache.org/jira/browse/SPARK-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15342188#comment-15342188 ]
Marcelo Vanzin commented on SPARK-15917: ---------------------------------------- Like Sean, standalone mode is not really my area. Maybe [~andrewor14]. > Define the number of executors in standalone mode with an easy-to-use property > ------------------------------------------------------------------------------ > > Key: SPARK-15917 > URL: https://issues.apache.org/jira/browse/SPARK-15917 > Project: Spark > Issue Type: Improvement > Components: Spark Core, Spark Shell, Spark Submit > Affects Versions: 1.6.1 > Reporter: Jonathan Taws > Priority: Minor > > After stumbling across a few StackOverflow posts around the issue of using a > fixed number of executors in standalone mode (non-YARN), I was wondering if > we could not add an easier way to set this parameter than having to resort to > some calculations based on the number of cores and the memory you have > available on your worker. > For example, let's say I have 8 cores and 30GB of memory available : > - If no option is passed, one executor will be spawned with 8 cores and 1GB > of memory allocated. > - However, if I want to have only *2* executors, and to use 2 cores and 10GB > of memory per executor, I will end up with *3* executors (as the available > memory will limit the number of executors) instead of the 2 I was hoping for. > Sure, I can set {{spark.cores.max}} as a workaround to get exactly what I > want, but would it not be easier to add a {{--num-executors}}-like option to > standalone mode to be able to really fine-tune the configuration ? This > option is already available in YARN mode. > From my understanding, I don't see any other option lying around that can > help achieve this. > This seems to be slightly disturbing for newcomers, and standalone mode is > probably the first thing anyone will use to just try out Spark or test some > configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org