are there other processes on sk3? or more generally are you sharing resources with somebody else, virtualization etc
does your transformation consumes other services?(e.g. reading from s3, so it can happen that s3 latency plays the role...) can it be that task per some key will take longer than same task on other key(I mean your business logic...) I see that some tasks take ~1min and other ~1h which is strange On 28 August 2015 at 21:47, Muler <mulugeta.abe...@gmail.com> wrote: > I have a 7 node cluster running in standalone mode (1 executor per node, > 100g/executor, 18 cores/executor) > > Attached is the Task status for two of my nodes. I'm not clear why some of > my tasks are taking too long: > > 1. [node sk5, green] task 197 took 35 mins while task 218 took less > than 2 mins. But if you look into the size of output size/records they have > almost same size. Even more strange, the size of shuffle spill for memory > and disk is 0 for task 197 and yet it is taking a long time > > Same issue for my other node (sk3, red) > > Can you please explain what is going on? > > Thanks, > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org >
