It's not only about cores. Keep in mind spark.executor.cores also affects available memeory for each task:
From http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/ The memory available to each task is (spark.executor.memory * spark.shuffle.memoryFraction *spark.shuffle.safetyFraction)/ spark.executor.cores. Memory fraction and safety fraction default to 0.2 and 0.8 respectively. I'd test spark.executor.cores with 2,4,8 and 16 and see what makes your job run faster.. -- Ruslan Dautkhanov On Wed, May 27, 2015 at 6:46 PM, Mulugeta Mammo <mulugeta.abe...@gmail.com> wrote: > My executor has the following spec (lscpu): > > CPU(s): 16 > Core(s) per socket: 4 > Socket(s): 2 > Thread(s) per code: 2 > > The CPU count is obviously 4*2*2 = 16. My question is what value is Spark > expecting in SPARK_EXECUTOR_CORES ? The CPU count (16) or total # of cores > (2 * 2 = 4) ? > > Thanks >
