On Wed, Sep 10, 2014 at 2:00 PM, Jeffrey Picard <[email protected]> wrote:
> After rebuilding from the master branch this morning, I’ve started to see > these errors that I’ve never gotten before while running connected > components. Anyone seen this before? > [...] > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:65) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > I think GraphX might not handle sort-based shuffle properly, and it was made the default recently [1]. If that's the problem, a temporary workaround would be to set spark.shuffle.manager to "hash". Ankur <http://www.ankurdave.com/> [1] https://github.com/apache/spark/pull/2178
