Hi Steve et al., It is possible that there's just a lot of skew in your data, in which case repartitioning is a good idea. Depending on how large your input data is and how much skew you have, you may want to repartition to a larger number of partitions. By the way you can just call rdd.repartition(1000); this is the same as rdd.coalesce(1000, forceShuffle = true). Note that repartitioning is only a good idea if your straggler task is taking a long time. Otherwise, it can be quite expensive since it requires a full shuffle.
Another possibility is that you might just have bad nodes in your cluster. To mitigate stragglers, you can try enabling speculative execution through spark.speculation to true. This attempts to re-run any task that takes a long time to complete on a different node in parallel. -Andrew 2014-12-04 11:43 GMT-08:00 akhandeshi <ami.khande...@gmail.com>: > This did not work for me. that is, rdd.coalesce(200, forceShuffle) . Does > anyone have ideas on how to distribute your data evenly and co-locate > partitions of interest? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Any-ideas-why-a-few-tasks-would-stall-tp20207p20387.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
