Re: Setting the number of executors in standalone mode
Hi Mohammed, thanks a lot for the reply. Ok, so from what I understand I cannot control the number of executors per worker in standalone cluster mode. Is that correct? BR On 20 February 2015 at 17:46, Mohammed Guller moham...@glassbeam.com wrote: SPARK_WORKER_MEMORY=8g Will allocate 8GB memory to Spark on each worker node. Nothing to do with # of executors. Mohammed *From:* Yiannis Gkoufas [mailto:johngou...@gmail.com] *Sent:* Friday, February 20, 2015 4:55 AM *To:* user@spark.apache.org *Subject:* Setting the number of executors in standalone mode Hi there, I try to increase the number of executors per worker in the standalone mode and I have failed to achieve that. I followed a bit the instructions of this thread: http://stackoverflow.com/questions/26645293/spark-configuration-memory-instance-cores and did that: spark.executor.memory 1g SPARK_WORKER_MEMORY=8g hoping to get 8 executors per worker but its still 1. And the option num-executors is not available in the standalone mode. Thanks a lot!
RE: Setting the number of executors in standalone mode
SPARK_WORKER_MEMORY=8g Will allocate 8GB memory to Spark on each worker node. Nothing to do with # of executors. Mohammed From: Yiannis Gkoufas [mailto:johngou...@gmail.com] Sent: Friday, February 20, 2015 4:55 AM To: user@spark.apache.org Subject: Setting the number of executors in standalone mode Hi there, I try to increase the number of executors per worker in the standalone mode and I have failed to achieve that. I followed a bit the instructions of this thread: http://stackoverflow.com/questions/26645293/spark-configuration-memory-instance-cores and did that: spark.executor.memory 1g SPARK_WORKER_MEMORY=8g hoping to get 8 executors per worker but its still 1. And the option num-executors is not available in the standalone mode. Thanks a lot!
RE: Setting the number of executors in standalone mode
ASFAIK, in stand-alone mode, each Spark application gets one executor on each worker. You could run multiple workers on a machine though. Mohammed From: Yiannis Gkoufas [mailto:johngou...@gmail.com] Sent: Friday, February 20, 2015 9:48 AM To: Mohammed Guller Cc: user@spark.apache.org Subject: Re: Setting the number of executors in standalone mode Hi Mohammed, thanks a lot for the reply. Ok, so from what I understand I cannot control the number of executors per worker in standalone cluster mode. Is that correct? BR On 20 February 2015 at 17:46, Mohammed Guller moham...@glassbeam.commailto:moham...@glassbeam.com wrote: SPARK_WORKER_MEMORY=8g Will allocate 8GB memory to Spark on each worker node. Nothing to do with # of executors. Mohammed From: Yiannis Gkoufas [mailto:johngou...@gmail.commailto:johngou...@gmail.com] Sent: Friday, February 20, 2015 4:55 AM To: user@spark.apache.orgmailto:user@spark.apache.org Subject: Setting the number of executors in standalone mode Hi there, I try to increase the number of executors per worker in the standalone mode and I have failed to achieve that. I followed a bit the instructions of this thread: http://stackoverflow.com/questions/26645293/spark-configuration-memory-instance-cores and did that: spark.executor.memory 1g SPARK_WORKER_MEMORY=8g hoping to get 8 executors per worker but its still 1. And the option num-executors is not available in the standalone mode. Thanks a lot!
Re: Setting the number of executors in standalone mode
Hi, Currently, there is only one executor per worker. There is jira ticket to relax this: https://issues.apache.org/jira/browse/SPARK-1706 But, if you want to use more cores, maybe, you can try increasing SPARK_WORKER_INSTANCES. It increases the number of workers per machine. Take a look here: http://spark.apache.org/docs/1.2.0/spark-standalone.html Hope this help! Kelvin On Fri, Feb 20, 2015 at 10:08 AM, Mohammed Guller moham...@glassbeam.com wrote: ASFAIK, in stand-alone mode, each Spark application gets one executor on each worker. You could run multiple workers on a machine though. Mohammed *From:* Yiannis Gkoufas [mailto:johngou...@gmail.com] *Sent:* Friday, February 20, 2015 9:48 AM *To:* Mohammed Guller *Cc:* user@spark.apache.org *Subject:* Re: Setting the number of executors in standalone mode Hi Mohammed, thanks a lot for the reply. Ok, so from what I understand I cannot control the number of executors per worker in standalone cluster mode. Is that correct? BR On 20 February 2015 at 17:46, Mohammed Guller moham...@glassbeam.com wrote: SPARK_WORKER_MEMORY=8g Will allocate 8GB memory to Spark on each worker node. Nothing to do with # of executors. Mohammed *From:* Yiannis Gkoufas [mailto:johngou...@gmail.com] *Sent:* Friday, February 20, 2015 4:55 AM *To:* user@spark.apache.org *Subject:* Setting the number of executors in standalone mode Hi there, I try to increase the number of executors per worker in the standalone mode and I have failed to achieve that. I followed a bit the instructions of this thread: http://stackoverflow.com/questions/26645293/spark-configuration-memory-instance-cores and did that: spark.executor.memory 1g SPARK_WORKER_MEMORY=8g hoping to get 8 executors per worker but its still 1. And the option num-executors is not available in the standalone mode. Thanks a lot!
