I used the packaged Pagerank example on a 120,147,484 node,
1,675,678,758 edge network. It finished successfully after 5.6h. Is
that reasonable?
All the parameters (as well as cluster info) are below. If the 5.6
hours is suboptimal, what parameters should I consider changing?
Cluster info:
Workers: 21
Cores: 672 Total, 0 Used
Memory: 2.6 TB Total, 0.0 B Used
Applications: 0 Running, 1 Completed
Application info:
Cores: 654 (640 Granted , 14 )
Memory per Slave: 94208
Submit Date: Thu Aug 22 19:50:14 PDT 2013
State: FINISHED
//usual Spark properties
System.setProperty("spark.serializer", "spark.KryoSerializer")
System.setProperty("spark.kryo.registrator", classOf[PRKryoRegistrator].getName)
System.setProperty("spark.cores.max", "654")
System.setProperty("spark.executor.memory", "92g")
System.setProperty("spark.default.parallelism", "1080")
System.setProperty("spark.storage.blockManagerHeartBeatMs", "300000")
System.setProperty("spark.worker.timeout","30000")
System.setProperty("spark.akka.timeout","30000")
System.setProperty("spark.storage.blockManagerHeartBeatMs","30000")
System.setProperty("spark.akka.retry.wait","30000")
System.setProperty("spark.akka.frameSize","10000")
//bagel specific parameters:
val numPartitions = 54 // or 1 per CPU core? If you don't have enough
it will hang on communication.
val usePartitioner = true
val epsilon = 0.1 / numVertices
val messages = sc.parallelize(Array[(String, PRMessage)]())
val utils = new PRFigureUtils
val result =
Bagel.run(
sc, vPRs, messages, combiner = new PRCombiner(),
numPartitions = numPartitions)(
utils.computeWithCombiner(numVertices, epsilon))
The max supserstep is 10:
bash-3.2$ cat pr.log | grep -i super
13/08/22 19:53:46 INFO bagel.Bagel: Starting superstep 0.
13/08/22 19:56:37 INFO bagel.Bagel: Superstep 0 took 170 s
13/08/22 19:56:37 INFO bagel.Bagel: Starting superstep 1.
13/08/22 20:06:19 INFO bagel.Bagel: Superstep 1 took 582 s
13/08/22 20:06:19 INFO bagel.Bagel: Starting superstep 2.
13/08/22 20:17:22 INFO bagel.Bagel: Superstep 2 took 662 s
13/08/22 20:17:22 INFO bagel.Bagel: Starting superstep 3.
13/08/22 20:28:52 INFO bagel.Bagel: Superstep 3 took 689 s
13/08/22 20:28:52 INFO bagel.Bagel: Starting superstep 4.
13/08/22 20:55:15 INFO bagel.Bagel: Superstep 4 took 1582 s
13/08/22 20:55:15 INFO bagel.Bagel: Starting superstep 5.
13/08/22 21:41:05 INFO bagel.Bagel: Superstep 5 took 2749 s
13/08/22 21:41:05 INFO bagel.Bagel: Starting superstep 6.
13/08/22 22:26:45 INFO bagel.Bagel: Superstep 6 took 2739 s
13/08/22 22:26:46 INFO bagel.Bagel: Starting superstep 7.
13/08/22 23:17:09 INFO bagel.Bagel: Superstep 7 took 3023 s
13/08/22 23:17:09 INFO bagel.Bagel: Starting superstep 8.
13/08/22 23:55:49 INFO bagel.Bagel: Superstep 8 took 2320 s
13/08/22 23:55:49 INFO bagel.Bagel: Starting superstep 9.
13/08/23 00:40:28 INFO bagel.Bagel: Superstep 9 took 2678 s
13/08/23 00:40:29 INFO bagel.Bagel: Starting superstep 10.
13/08/23 01:27:38 INFO bagel.Bagel: Superstep 10 took 2829 s
