[jira] [Comment Edited] (SPARK-15917) Define the number of executors in standalone mode with an easy-to-use property
[
https://issues.apache.org/jira/browse/SPARK-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15493395#comment-15493395
]
Jonathan Taws edited comment on SPARK-15917 at 9/15/16 4:05 PM:
Hi Andrew,
Your 2 suggestions make a lot of sense, I'll work on it straight away. Give me
a few days to try them out.
By the way, do you have any pointers to where I could look into to produce a
meaning warning when there isn't enough resources to comply with all the
parameters ?
was (Author: jonathantaws):
Hi Andrew,
Your 2 suggestions make a lot of sense, I'll work on it straight away. Give me
a few days to try them out.
> Define the number of executors in standalone mode with an easy-to-use property
> --
>
> Key: SPARK-15917
> URL: https://issues.apache.org/jira/browse/SPARK-15917
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, Spark Shell, Spark Submit
>Affects Versions: 1.6.1
>Reporter: Jonathan Taws
>Priority: Minor
>
> After stumbling across a few StackOverflow posts around the issue of using a
> fixed number of executors in standalone mode (non-YARN), I was wondering if
> we could not add an easier way to set this parameter than having to resort to
> some calculations based on the number of cores and the memory you have
> available on your worker.
> For example, let's say I have 8 cores and 30GB of memory available :
> - If no option is passed, one executor will be spawned with 8 cores and 1GB
> of memory allocated.
> - However, if I want to have only *2* executors, and to use 2 cores and 10GB
> of memory per executor, I will end up with *3* executors (as the available
> memory will limit the number of executors) instead of the 2 I was hoping for.
> Sure, I can set {{spark.cores.max}} as a workaround to get exactly what I
> want, but would it not be easier to add a {{--num-executors}}-like option to
> standalone mode to be able to really fine-tune the configuration ? This
> option is already available in YARN mode.
> From my understanding, I don't see any other option lying around that can
> help achieve this.
> This seems to be slightly disturbing for newcomers, and standalone mode is
> probably the first thing anyone will use to just try out Spark or test some
> configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-15917) Define the number of executors in standalone mode with an easy-to-use property
[
https://issues.apache.org/jira/browse/SPARK-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15341360#comment-15341360
]
Jonathan Taws edited comment on SPARK-15917 at 6/21/16 8:25 AM:
If I run the following command : {code}
spark-shell --master --conf spark.cores.max=2 --conf
spark.executor.instances=2 --executor-cores 2 --executor-memory 4g
{code}
I get only one executor, as there are only 2 cores available (same with
{{--num-executors}} instead of {{spark.executor.instances}}).
{{spark.cores.max}} does its job right here, and it just seems like the
executor instances property is just being discarded here.
If anything, adding a support for a {{--num-executors}}-like property in
standalone would allow to be consistent across deployment methods and, in my
experience, make the beginner experience slightly easier.
was (Author: jonathantaws):
If I run the following command : {{spark-shell --master --conf
spark.cores.max=2 --conf spark.executor.instances=2 --executor-cores 2
--executor-memory 4g}}, I get only one executor, as there are only 2 cores
available (same with {{--num-executors}} instead of
{{spark.executor.instances}}).
{{spark.cores.max}} does its job right here, and it just seems like the
executor instances property is just being discarded here.
If anything, adding a support for a {{--num-executors}}-like property in
standalone would allow to be consistent across deployment methods and, in my
experience, make the beginner experience slightly easier.
> Define the number of executors in standalone mode with an easy-to-use property
> --
>
> Key: SPARK-15917
> URL: https://issues.apache.org/jira/browse/SPARK-15917
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, Spark Shell, Spark Submit
>Affects Versions: 1.6.1
>Reporter: Jonathan Taws
>Priority: Minor
>
> After stumbling across a few StackOverflow posts around the issue of using a
> fixed number of executors in standalone mode (non-YARN), I was wondering if
> we could not add an easier way to set this parameter than having to resort to
> some calculations based on the number of cores and the memory you have
> available on your worker.
> For example, let's say I have 8 cores and 30GB of memory available :
> - If no option is passed, one executor will be spawned with 8 cores and 1GB
> of memory allocated.
> - However, if I want to have only *2* executors, and to use 2 cores and 10GB
> of memory per executor, I will end up with *3* executors (as the available
> memory will limit the number of executors) instead of the 2 I was hoping for.
> Sure, I can set {{spark.cores.max}} as a workaround to get exactly what I
> want, but would it not be easier to add a {{--num-executors}}-like option to
> standalone mode to be able to really fine-tune the configuration ? This
> option is already available in YARN mode.
> From my understanding, I don't see any other option lying around that can
> help achieve this.
> This seems to be slightly disturbing for newcomers, and standalone mode is
> probably the first thing anyone will use to just try out Spark or test some
> configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-15917) Define the number of executors in standalone mode with an easy-to-use property
[
https://issues.apache.org/jira/browse/SPARK-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15341360#comment-15341360
]
Jonathan Taws edited comment on SPARK-15917 at 6/21/16 8:24 AM:
If I run the following command : {{spark-shell --master --conf
spark.cores.max=2 --conf spark.executor.instances=2 --executor-cores 2
--executor-memory 4g}}, I get only one executor, as there are only 2 cores
available (same with {{--num-executors}} instead of
{{spark.executor.instances}}).
{{spark.cores.max}} does its job right here, and it just seems like the
executor instances property is just being discarded here.
If anything, adding a support for a {{--num-executors}}-like property in
standalone would allow to be consistent across deployment methods and, in my
experience, make the beginner experience slightly easier.
was (Author: jonathantaws):
If I run the following command : {{spark-shell \--master \--conf
spark.cores.max=2 \--conf spark.executor.instances=2 \--executor-cores 2
\--executor-memory 4g}}, I get only one executor, as there are only 2 cores
available (same with {{--num-executors}} instead of
{{spark.executor.instances}}).
{{spark.cores.max}} does its job right here, and it just seems like the
executor instances property is just being discarded here.
If anything, adding a support for a {{--num-executors}}-like property in
standalone would allow to be consistent across deployment methods and, in my
experience, make the beginner experience slightly easier.
> Define the number of executors in standalone mode with an easy-to-use property
> --
>
> Key: SPARK-15917
> URL: https://issues.apache.org/jira/browse/SPARK-15917
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, Spark Shell, Spark Submit
>Affects Versions: 1.6.1
>Reporter: Jonathan Taws
>Priority: Minor
>
> After stumbling across a few StackOverflow posts around the issue of using a
> fixed number of executors in standalone mode (non-YARN), I was wondering if
> we could not add an easier way to set this parameter than having to resort to
> some calculations based on the number of cores and the memory you have
> available on your worker.
> For example, let's say I have 8 cores and 30GB of memory available :
> - If no option is passed, one executor will be spawned with 8 cores and 1GB
> of memory allocated.
> - However, if I want to have only *2* executors, and to use 2 cores and 10GB
> of memory per executor, I will end up with *3* executors (as the available
> memory will limit the number of executors) instead of the 2 I was hoping for.
> Sure, I can set {{spark.cores.max}} as a workaround to get exactly what I
> want, but would it not be easier to add a {{--num-executors}}-like option to
> standalone mode to be able to really fine-tune the configuration ? This
> option is already available in YARN mode.
> From my understanding, I don't see any other option lying around that can
> help achieve this.
> This seems to be slightly disturbing for newcomers, and standalone mode is
> probably the first thing anyone will use to just try out Spark or test some
> configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-15917) Define the number of executors in standalone mode with an easy-to-use property
[
https://issues.apache.org/jira/browse/SPARK-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15341360#comment-15341360
]
Jonathan Taws edited comment on SPARK-15917 at 6/21/16 8:24 AM:
If I run the following command : {{spark-shell \--master \--conf
spark.cores.max=2 \--conf spark.executor.instances=2 \--executor-cores 2
\--executor-memory 4g}}, I get only one executor, as there are only 2 cores
available (same with {{--num-executors}} instead of
{{spark.executor.instances}}).
{{spark.cores.max}} does its job right here, and it just seems like the
executor instances property is just being discarded here.
If anything, adding a support for a {{--num-executors}}-like property in
standalone would allow to be consistent across deployment methods and, in my
experience, make the beginner experience slightly easier.
was (Author: jonathantaws):
If I run the following command : {{spark-shell --master --conf
spark.cores.max=2 --conf spark.executor.instances=2 --executor-cores 2
--executor-memory 4g}}, I get only one executor, as there are only 2 cores
available (same with {{--num-executors}} instead of
{{spark.executor.instances}}).
{{spark.cores.max}} does its job right here, and it just seems like the
executor instances property is just being discarded here.
If anything, adding a support for a {{--num-executors}}-like property in
standalone would allow to be consistent across deployment methods and, in my
experience, make the beginner experience slightly easier.
> Define the number of executors in standalone mode with an easy-to-use property
> --
>
> Key: SPARK-15917
> URL: https://issues.apache.org/jira/browse/SPARK-15917
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, Spark Shell, Spark Submit
>Affects Versions: 1.6.1
>Reporter: Jonathan Taws
>Priority: Minor
>
> After stumbling across a few StackOverflow posts around the issue of using a
> fixed number of executors in standalone mode (non-YARN), I was wondering if
> we could not add an easier way to set this parameter than having to resort to
> some calculations based on the number of cores and the memory you have
> available on your worker.
> For example, let's say I have 8 cores and 30GB of memory available :
> - If no option is passed, one executor will be spawned with 8 cores and 1GB
> of memory allocated.
> - However, if I want to have only *2* executors, and to use 2 cores and 10GB
> of memory per executor, I will end up with *3* executors (as the available
> memory will limit the number of executors) instead of the 2 I was hoping for.
> Sure, I can set {{spark.cores.max}} as a workaround to get exactly what I
> want, but would it not be easier to add a {{--num-executors}}-like option to
> standalone mode to be able to really fine-tune the configuration ? This
> option is already available in YARN mode.
> From my understanding, I don't see any other option lying around that can
> help achieve this.
> This seems to be slightly disturbing for newcomers, and standalone mode is
> probably the first thing anyone will use to just try out Spark or test some
> configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
