To run multiple workers with Spark’s standalone mode, set
SPARK_WORKER_INSTANCES and SPARK_WORKER_CORES in conf/spark-env.sh. For
example, if you have 16 cores and want 2 workers, you could add
export SPARK_WORKER_INSTANCES=2
export SPARK_WORKER_CORES=8
Matei
On Apr 3, 2014, at 12:38 PM,
@Mayur...I am hitting ulimits on the cluster if I go beyond 4 core per
worker and I don't think I can change the ulimit due to sudo issues etc...
If I have more workers, in ALS, I can go for 20 blocks (right now I am
running 10 blocks on 10 nodes with 4 cores each and now I can go upto 20
blocks
Hi Folks
I'm looking to buy some gear to run Spark. I'm quite well versed in Hadoop
Server design but there does not seem to be much Spark related collateral
around infrastructure guidelines (or at least I haven't been able to find
them). My current thinking for server design is something
I would suggest to start with cloud hosting if you can, depending on your
usecase, memory requirement may vary a lot .
Regards
Mayur
On Apr 2, 2014 3:59 PM, Matei Zaharia matei.zaha...@gmail.com wrote:
Hey Steve,
This configuration sounds pretty good. The one thing I would consider is
having
Hi Matei,
How can I run multiple Spark workers per node ? I am running 8 core 10 node
cluster but I do have 8 more cores on each nodeSo having 2 workers per
node will definitely help my usecase.
Thanks.
Deb
On Wed, Apr 2, 2014 at 3:58 PM, Matei Zaharia matei.zaha...@gmail.comwrote:
Hey
