: Thursday, May 28, 2015 7:24 PM
To: Ruslan Dautkhanov
Cc: user
Subject: Re: Value for SPARK_EXECUTOR_CORES
Thanks for the valuable information. The blog states:
"The cores property controls the number of concurrent tasks an executor can
run. --executor-cores 5 means that each executor can
Thanks for the valuable information. The blog states:
"The cores property controls the number of concurrent tasks an executor can
run. --executor-cores 5 means that each executor can run a maximum of five
tasks at the same time. "
So, I guess the max number of executor-cores I can assign is the C
It's not only about cores. Keep in mind spark.executor.cores also affects
available memeory for each task:
From
http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/
The memory available to each task is (spark.executor.memory *
spark.shuffle.memoryFraction *spark.shuffl