RE: Setting executors per worker - Standalone
I don't think you can do that in the Standalone mode before 1.5. The best you can do is to have multi workers per box. One worker can and will only start one executor, before Spark 1.5. What you can do is to set "SPARK_WORKER_INSTANCES", which control how many worker instances you can start per box. Yong Date: Mon, 28 Sep 2015 20:47:18 -0700 Subject: Re: Setting executors per worker - Standalone From: james.p...@gmail.com To: zjf...@gmail.com CC: user@spark.apache.org Thanks for your reply. Setting it as --conf spark.executor.cores=1 when I start spark-shell (as an example application) indeed sets the number of cores per executor as 1 (which is 4 before), but I still have 1 executor per worker. What I am really looking for is having 1 worker with 4 executor (each with one core) per machine when I run my application. Based one the documentation it seems it is feasible, but it is not clear as how. Thanks. On Mon, Sep 28, 2015 at 8:46 PM, Jeff Zhang <zjf...@gmail.com> wrote: use "--executor-cores 1" you will get 4 executors per worker since you have 4 cores per worker On Tue, Sep 29, 2015 at 8:24 AM, James Pirz <james.p...@gmail.com> wrote: Hi, I am using speak 1.5 (standalone mode) on a cluster with 10 nodes while each machine has 12GB of RAM and 4 cores. On each machine I have one worker which is running one executor that grabs all 4 cores. I am interested to check the performance with "one worker but 4 executors per machine - each with one core". I can see that "running multiple executors per worker in Standalone mode" is possible based on the closed issue: https://issues.apache.org/jira/browse/SPARK-1706 But I can not find a way to do that. "SPARK_EXECUTOR_INSTANCES" is only available for the Yarn mode, and in the standalone mode I can just set "SPARK_WORKER_INSTANCES" and "SPARK_WORKER_CORES" and "SPARK_WORKER_MEMORY". Any hint or suggestion would be great. -- Best Regards Jeff Zhang
Re: Setting executors per worker - Standalone
Thanks for your help. You were correct about the memory settings. Previously I had following config: --executor-memory 8g --conf spark.executor.cores=1 Which was really conflicting, as in spark-env.sh I had: export SPARK_WORKER_CORES=4 export SPARK_WORKER_MEMORY=8192m So the memory budget per worker was not enough to launch several executors. By switching to: --executor-memory 2g --conf spark.executor.cores=1 Now I can see that on each machine I have one worker, with 4 executors. Thanks again for your help. On Tue, Sep 29, 2015 at 1:30 AM, Robin Eastwrote: > I’m currently testing this exact setup - it work for me using both —conf > spark.exeuctors.cores=1 and —executor-cores 1. Do you have some memory > settings that need to be adjusted as well? Or do you accidentally have > —total-executor-cores set as well? You should be able to tell from looking > at the environment tab on the Application UI > > --- > Robin East > *Spark GraphX in Action* Michael Malak and Robin East > Manning Publications Co. > http://www.manning.com/books/spark-graphx-in-action > > > > > > On 29 Sep 2015, at 04:47, James Pirz wrote: > > Thanks for your reply. > > Setting it as > > --conf spark.executor.cores=1 > > when I start spark-shell (as an example application) indeed sets the > number of cores per executor as 1 (which is 4 before), but I still have 1 > executor per worker. What I am really looking for is having 1 worker with 4 > executor (each with one core) per machine when I run my application. Based > one the documentation it seems it is feasible, but it is not clear as how. > > Thanks. > > On Mon, Sep 28, 2015 at 8:46 PM, Jeff Zhang wrote: > >> use "--executor-cores 1" you will get 4 executors per worker since you >> have 4 cores per worker >> >> >> >> On Tue, Sep 29, 2015 at 8:24 AM, James Pirz wrote: >> >>> Hi, >>> >>> I am using speak 1.5 (standalone mode) on a cluster with 10 nodes while >>> each machine has 12GB of RAM and 4 cores. On each machine I have one worker >>> which is running one executor that grabs all 4 cores. I am interested to >>> check the performance with "one worker but 4 executors per machine - each >>> with one core". >>> >>> I can see that "running multiple executors per worker in Standalone >>> mode" is possible based on the closed issue: >>> >>> https://issues.apache.org/jira/browse/SPARK-1706 >>> >>> But I can not find a way to do that. "SPARK_EXECUTOR_INSTANCES" is only >>> available for the Yarn mode, and in the standalone mode I can just set >>> "SPARK_WORKER_INSTANCES" and "SPARK_WORKER_CORES" and "SPARK_WORKER_MEMORY". >>> >>> Any hint or suggestion would be great. >>> >>> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > > >
Re: Setting executors per worker - Standalone
use "--executor-cores 1" you will get 4 executors per worker since you have 4 cores per worker On Tue, Sep 29, 2015 at 8:24 AM, James Pirzwrote: > Hi, > > I am using speak 1.5 (standalone mode) on a cluster with 10 nodes while > each machine has 12GB of RAM and 4 cores. On each machine I have one worker > which is running one executor that grabs all 4 cores. I am interested to > check the performance with "one worker but 4 executors per machine - each > with one core". > > I can see that "running multiple executors per worker in Standalone mode" > is possible based on the closed issue: > > https://issues.apache.org/jira/browse/SPARK-1706 > > But I can not find a way to do that. "SPARK_EXECUTOR_INSTANCES" is only > available for the Yarn mode, and in the standalone mode I can just set > "SPARK_WORKER_INSTANCES" and "SPARK_WORKER_CORES" and "SPARK_WORKER_MEMORY". > > Any hint or suggestion would be great. > > -- Best Regards Jeff Zhang
Re: Setting executors per worker - Standalone
Thanks for your reply. Setting it as --conf spark.executor.cores=1 when I start spark-shell (as an example application) indeed sets the number of cores per executor as 1 (which is 4 before), but I still have 1 executor per worker. What I am really looking for is having 1 worker with 4 executor (each with one core) per machine when I run my application. Based one the documentation it seems it is feasible, but it is not clear as how. Thanks. On Mon, Sep 28, 2015 at 8:46 PM, Jeff Zhangwrote: > use "--executor-cores 1" you will get 4 executors per worker since you > have 4 cores per worker > > > > On Tue, Sep 29, 2015 at 8:24 AM, James Pirz wrote: > >> Hi, >> >> I am using speak 1.5 (standalone mode) on a cluster with 10 nodes while >> each machine has 12GB of RAM and 4 cores. On each machine I have one worker >> which is running one executor that grabs all 4 cores. I am interested to >> check the performance with "one worker but 4 executors per machine - each >> with one core". >> >> I can see that "running multiple executors per worker in Standalone mode" >> is possible based on the closed issue: >> >> https://issues.apache.org/jira/browse/SPARK-1706 >> >> But I can not find a way to do that. "SPARK_EXECUTOR_INSTANCES" is only >> available for the Yarn mode, and in the standalone mode I can just set >> "SPARK_WORKER_INSTANCES" and "SPARK_WORKER_CORES" and "SPARK_WORKER_MEMORY". >> >> Any hint or suggestion would be great. >> >> > > > -- > Best Regards > > Jeff Zhang >
Setting executors per worker - Standalone
Hi, I am using speak 1.5 (standalone mode) on a cluster with 10 nodes while each machine has 12GB of RAM and 4 cores. On each machine I have one worker which is running one executor that grabs all 4 cores. I am interested to check the performance with "one worker but 4 executors per machine - each with one core". I can see that "running multiple executors per worker in Standalone mode" is possible based on the closed issue: https://issues.apache.org/jira/browse/SPARK-1706 But I can not find a way to do that. "SPARK_EXECUTOR_INSTANCES" is only available for the Yarn mode, and in the standalone mode I can just set "SPARK_WORKER_INSTANCES" and "SPARK_WORKER_CORES" and "SPARK_WORKER_MEMORY". Any hint or suggestion would be great.