Hello,
I try to execute a simple program that runs the ShortestPaths algorithm
(org.apache.spark.graphx.lib.ShortestPaths) on a small grid graph.
I use Spark 1.2.0 downloaded from spark.apache.org.
The program's code is the following :
object GraphXGridSP {
def main(args : Array[String]) : Unit = {
val appname : String = "GraphXGridShortestPath"
val conf = new SparkConf().setAppName(appname)
val sc = new SparkContext(conf)
val gridSize : Int = 70
val nPartitions : Int = 4
val graph = GraphGenerators.gridGraph(sc, gridSize, gridSize).
partitionBy( PartitionStrategy.EdgePartition2D, nPartitions)
val landmarks : Seq[VertexId] = Seq(0)
val graph2 : Graph[SPMap, Double]= ShortestPaths.run(graph, landmarks)
graph2.vertices.count
}
}
This program runs more than 2 hours when the grid size is 70x70 as above, and
is then killed
by the resource manager of the cluster (Torque). After a 5-6 minutes of
execution, the
Spark master UI does not even respond.
I use a cluster of 5 nodes (4 workers, 1 executor per node).
For a grid size of 30x30, the program terminates in about 20 seconds, and for a
grid size
of 50x50 it finishes in about 80 seconds. The problem appears for a grid size
of 70x70 and
above.
What's wrong with this program ?
Thanks for any help.
Regards.
Nicolas.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
