Are you doing a join/groupBy such operation? In that case i would suspect that the keys are not evenly distributed and that's why few of the tasks are spending way too much time doing the actual processing. You might want to look into custom partitioners <http://stackoverflow.com/questions/23127329/how-to-define-custom-partitioner-for-spark-rdds-of-equally-sized-partition-where> to avoid these scenarios.
Thanks Best Regards On Sat, Aug 29, 2015 at 12:17 AM, Muler <mulugeta.abe...@gmail.com> wrote: > I have a 7 node cluster running in standalone mode (1 executor per node, > 100g/executor, 18 cores/executor) > > Attached is the Task status for two of my nodes. I'm not clear why some of > my tasks are taking too long: > > 1. [node sk5, green] task 197 took 35 mins while task 218 took less > than 2 mins. But if you look into the size of output size/records they have > almost same size. Even more strange, the size of shuffle spill for memory > and disk is 0 for task 197 and yet it is taking a long time > > Same issue for my other node (sk3, red) > > Can you please explain what is going on? > > Thanks, > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org >
