Hi, I'm running a simple connected components code using GraphX (version 0.9.1)
My input comes from a HDFS text file partitioned to 400 parts. When I run the code on a single part or a small number of files (like 20) the code runs fine. As soon as I'm trying to read more files (more than 30) I'm getting an error and the job fails. >From looking at the logs I see the following exception java.util.NoSuchElementException: End of stream at org.apache.spark.util.NextIterator.next(NextIterator.scala:83) at org.apache.spark.InterruptibleIterator.next(InterruptibleIterator.scala:29) at org.apache.spark.graphx.impl.RoutingTable$$anonfun$1.apply(RoutingTable.scala:52) at org.apache.spark.graphx.impl.RoutingTable$$anonfun$1.apply(RoutingTable.scala:51) at org.apache.spark.rdd.RDD$$anonfun$1.apply(RDD.scala:456) >From searching the web, I see it's a known issue with GraphX Here : https://github.com/apache/spark/pull/367 And here : https://github.com/apache/spark/pull/497 Are there some stable releases that include this fix? Should I clone the git repo and build it myself? How would you advise me to deal with this issue Thanks, Alex