GraphX: ShortestPaths does not terminate on a grid graph

[NicolasC]

Hello,

I try to execute a simple program that runs the ShortestPaths algorithm
(org.apache.spark.graphx.lib.ShortestPaths) on a small grid graph.
I use Spark 1.2.0 downloaded from spark.apache.org.

The program's code is the following :

object GraphXGridSP {

    def main(args : Array[String]) : Unit = {
        val appname : String = "GraphXGridShortestPath"
        val conf = new SparkConf().setAppName(appname)

        val sc = new SparkContext(conf)

        val gridSize : Int = 70
        val nPartitions : Int = 4

        val graph = GraphGenerators.gridGraph(sc, gridSize, gridSize).
              partitionBy( PartitionStrategy.EdgePartition2D, nPartitions)

        val landmarks : Seq[VertexId] = Seq(0)
        val graph2 : Graph[SPMap, Double]= ShortestPaths.run(graph, landmarks)
        graph2.vertices.count
    }
}

This program runs more than 2 hours when the grid size is 70x70 as above, and 
is then killed
by the resource manager of the cluster (Torque). After a 5-6 minutes of 
execution, the
Spark master UI does not even respond.

I use a cluster of 5 nodes (4 workers, 1 executor per node).

For a grid size of 30x30, the program terminates in about 20 seconds, and for a 
grid size
of 50x50 it finishes in about 80 seconds. The problem appears for a grid size 
of 70x70 and
above.

What's wrong with this program ?

Thanks for any help.

Regards.
Nicolas.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org