Thanks for your help. You were correct about the memory settings. Previously I had following config:
--executor-memory 8g --conf spark.executor.cores=1 Which was really conflicting, as in spark-env.sh I had: export SPARK_WORKER_CORES=4 export SPARK_WORKER_MEMORY=8192m So the memory budget per worker was not enough to launch several executors. By switching to: --executor-memory 2g --conf spark.executor.cores=1 Now I can see that on each machine I have one worker, with 4 executors. Thanks again for your help. On Tue, Sep 29, 2015 at 1:30 AM, Robin East <robin.e...@xense.co.uk> wrote: > I’m currently testing this exact setup - it work for me using both —conf > spark.exeuctors.cores=1 and —executor-cores 1. Do you have some memory > settings that need to be adjusted as well? Or do you accidentally have > —total-executor-cores set as well? You should be able to tell from looking > at the environment tab on the Application UI > > ------------------------------------------------------------------------------- > Robin East > *Spark GraphX in Action* Michael Malak and Robin East > Manning Publications Co. > http://www.manning.com/books/spark-graphx-in-action > > > > > > On 29 Sep 2015, at 04:47, James Pirz <james.p...@gmail.com> wrote: > > Thanks for your reply. > > Setting it as > > --conf spark.executor.cores=1 > > when I start spark-shell (as an example application) indeed sets the > number of cores per executor as 1 (which is 4 before), but I still have 1 > executor per worker. What I am really looking for is having 1 worker with 4 > executor (each with one core) per machine when I run my application. Based > one the documentation it seems it is feasible, but it is not clear as how. > > Thanks. > > On Mon, Sep 28, 2015 at 8:46 PM, Jeff Zhang <zjf...@gmail.com> wrote: > >> use "--executor-cores 1" you will get 4 executors per worker since you >> have 4 cores per worker >> >> >> >> On Tue, Sep 29, 2015 at 8:24 AM, James Pirz <james.p...@gmail.com> wrote: >> >>> Hi, >>> >>> I am using speak 1.5 (standalone mode) on a cluster with 10 nodes while >>> each machine has 12GB of RAM and 4 cores. On each machine I have one worker >>> which is running one executor that grabs all 4 cores. I am interested to >>> check the performance with "one worker but 4 executors per machine - each >>> with one core". >>> >>> I can see that "running multiple executors per worker in Standalone >>> mode" is possible based on the closed issue: >>> >>> https://issues.apache.org/jira/browse/SPARK-1706 >>> >>> But I can not find a way to do that. "SPARK_EXECUTOR_INSTANCES" is only >>> available for the Yarn mode, and in the standalone mode I can just set >>> "SPARK_WORKER_INSTANCES" and "SPARK_WORKER_CORES" and "SPARK_WORKER_MEMORY". >>> >>> Any hint or suggestion would be great. >>> >>> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > > >
