Re: worker_instances vs worker_cores

2014-10-20 Thread Andrew Ash
Hi Anny, SPARK_WORKER_INSTANCES is the number of copies of spark workers running on a single box. If you change the number you change how the hardware you have is split up (useful for breaking large servers into 32GB heaps each which perform better) but doesn't change the amount of hardware you

Re: worker_instances vs worker_cores

2014-10-20 Thread Anny Chen
Thanks a lot Andrew! Yeah I actually realized that later. I made a silly mistake here. On Mon, Oct 20, 2014 at 6:03 PM, Andrew Ash and...@andrewash.com wrote: Hi Anny, SPARK_WORKER_INSTANCES is the number of copies of spark workers running on a single box. If you change the number you change