If some of the operations required involve shuffling and partitioning, it might mean that the data set is skewed to specific partitions which will create hot spotting on certain executors.
-adrian From: Khaled Ammar Date: Tuesday, November 3, 2015 at 11:43 PM To: "user@spark.apache.org<mailto:user@spark.apache.org>" Subject: Why some executors are lazy? Hi, I'm using the most recent Spark version on a standalone setup of 16+1 machines. While running GraphX workloads, I found that some executors are lazy? They *rarely* participate in computation. This causes some other executors to do their work. This behavior is consistent in all iterations and even in the data loading step. Only two specific executors do not participate in most computations. Does any one know how to fix that? More details: Each machine has 4 cores. I set number of partitions to be 3*16. Each executor was supposed to do 3 tasks, but few of them end up working on 4 task instead, which causes delay in computation. -- Thanks, -Khaled
