how to set SPARK_WORKER_INSTANCES and SPARK_WORKER_CORES otpimally

[Chen Jin]

Hi all,

>From spark document, we can set the number of workers by
SPARK_WORKER_INSTANCES and the max number of cores that worker can
take by using SPARK_WORKER_CORES, if I have 5 8-core machine, which
one would perform better between
(a)
   SPARK_WORKER_INSTANCES = 8
   SPARK_WORKER_CORES = 1

and
(b)
   SPARK_WORKER_INSTANCES = 1
   SPARK_WORKER_CORES = 8

(a) gives us 40 workers with each core per worker (b) gives 8 workers
while each worker has eight cores. Any advice on which better would
lead to better performance?

Thanks a lot,

-chen