Thanks for the valuable information. The blog states: "The cores property controls the number of concurrent tasks an executor can run. --executor-cores 5 means that each executor can run a maximum of five tasks at the same time. "
So, I guess the max number of executor-cores I can assign is the CPU count (which includes the number of threads per core), not just the number of cores. I just want to be sure the "cores" term Spark is using. Thanks On Thu, May 28, 2015 at 11:16 AM, Ruslan Dautkhanov <dautkha...@gmail.com> wrote: > It's not only about cores. Keep in mind spark.executor.cores also affects > available memeory for each task: > > From > http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/ > > The memory available to each task is (spark.executor.memory * > spark.shuffle.memoryFraction *spark.shuffle.safetyFraction)/ > spark.executor.cores. Memory fraction and safety fraction default to 0.2 > and 0.8 respectively. > > I'd test spark.executor.cores with 2,4,8 and 16 and see what makes your > job run faster.. > > > -- > Ruslan Dautkhanov > > On Wed, May 27, 2015 at 6:46 PM, Mulugeta Mammo <mulugeta.abe...@gmail.com > > wrote: > >> My executor has the following spec (lscpu): >> >> CPU(s): 16 >> Core(s) per socket: 4 >> Socket(s): 2 >> Thread(s) per code: 2 >> >> The CPU count is obviously 4*2*2 = 16. My question is what value is Spark >> expecting in SPARK_EXECUTOR_CORES ? The CPU count (16) or total # of cores >> (2 * 2 = 4) ? >> >> Thanks >> > >
