Dear All,
I am using connected components function of GraphX (on Spark 1.0.2) on some
graph. However for some reason the fails with StackOverflowError. The graph
is not too big; it contains 10000 vertices and 500000 edges. Can any one
help me to avoid this error? Below is the output of Spark:
14/10/28 16:08:50 INFO SparkContext: Starting job: reduce at
VertexRDD.scala:100
14/10/28 16:08:50 INFO DAGScheduler: Registering RDD 13 (mapPartitions at
VertexRDD.scala:423)
14/10/28 16:08:50 INFO DAGScheduler: Registering RDD 26 (mapPartitions at
VertexRDD.scala:318)
14/10/28 16:08:50 INFO DAGScheduler: Registering RDD 30 (mapPartitions at
VertexRDD.scala:318)
14/10/28 16:08:50 INFO DAGScheduler: Registering RDD 34 (mapPartitions at
GraphImpl.scala:184)
14/10/28 16:08:50 INFO DAGScheduler: Got job 7 (reduce at
VertexRDD.scala:100) with 1 output partitions (allowLocal=false)
14/10/28 16:08:50 INFO DAGScheduler: Final stage: Stage 12(reduce at
VertexRDD.scala:100)
14/10/28 16:08:50 INFO DAGScheduler: Parents of final stage: List(Stage 15,
Stage 13)
14/10/28 16:08:50 INFO DAGScheduler: Missing parents: List(Stage 15, Stage
13)
14/10/28 16:08:50 INFO DAGScheduler: Submitting Stage 13
(VertexRDD.createRoutingTables - vid2pid (aggregation) MapPartitionsRDD[13]
at mapPartitions at VertexRDD.scala:423), which has no missing parents
14/10/28 16:08:50 INFO DAGScheduler: Submitting 1 missing tasks from Stage
13 (VertexRDD.createRoutingTables - vid2pid (aggregation)
MapPartitionsRDD[13] at mapPartitions at VertexRDD.scala:423)
14/10/28 16:08:50 INFO TaskSchedulerImpl: Adding task set 13.0 with 1 tasks
14/10/28 16:08:50 INFO TaskSetManager: Starting task 13.0:0 as TID 8 on
executor 0: kw2260.kaust.edu.sa (PROCESS_LOCAL)
14/10/28 16:08:50 INFO TaskSetManager: Serialized task 13.0:0 as 3269 bytes
in 0 ms
14/10/28 16:08:57 WARN TaskSetManager: Lost TID 8 (task 13.0:0)
14/10/28 16:08:57 WARN TaskSetManager: Loss was due to
java.lang.StackOverflowError
java.lang.StackOverflowError
at scala.util.Sorting$.sort2$1(Sorting.scala:128)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
at scala.util.Sorting$.sort2$1(Sorting.scala:176)
Regards,
Zuhair Khayyat